Identify channel and correlation-like dimensions in non-standard MS columns

[sjperkins]

• Closes Non-standard MS columns have an auto-generated schema which is not chunked according to logic for standard dimensions #241

• Closes Silently fails when providing incorrect schema for WEIGHT column #268

• Tests added / passed

  $ py.test -v -s daskms/tests

  If the pep8 tests fail, the quickest way to correct
  this is to run autopep8 and then flake8 and
  pycodestyle to fix the remaining issues.

  $ pip install -U autopep8 flake8 pycodestyle
  $ autopep8 -r -i daskms
  $ flake8 daskms
  $ pycodestyle daskms
  

• Fully documented, including HISTORY.rst for all changes
  and one of the docs/*-api.rst files for new API

  To build the docs locally:

  pip install -r requirements.readthedocs.txt
  cd docs
  READTHEDOCS=True make html
  

[sjperkins]

I also want to to try address #268 in this PR.

[landmanbester]

I'm gonna rerun a convert command that used to fail to confirm that the fix works but it looks pretty uncontroversial to me. I'll get back to you as soon as the convert is done

[JSKenyon]

Looks good! Am correct in saying that this doesn't store this information - it simply presumes that any non-standard column will probably be visibility shaped? This will work in 99.99% of cases, but if the sizes of chan and corr are the same it may fail/assign (row, chan, chan).

[sjperkins]

Looks good! Am correct in saying that this doesn't store this information - it simply presumes that any non-standard column will probably be visibility shaped? This will work in 99.99% of cases, but if the sizes of chan and corr are the same it may fail/assign (row, chan, chan).

Its slightly more intelligent than that, it checks if the dimension name is something like {varname}-{d} where varname is the variable name and d is the dimension number. Then it checks if it matches chan or corr.

[sjperkins]

Looks good! Am correct in saying that this doesn't store this information - it simply presumes that any non-standard column will probably be visibility shaped? This will work in 99.99% of cases, but if the sizes of chan and corr are the same it may fail/assign (row, chan, chan).

Yep, I think this should try to handle the chan == corr case by prioritising chan, and then corr.

[JSKenyon]

One extra note - this doesn't work if you only select the column with unknown dimensions:

print(xds_from_ms("msdir/C147_minimal.MS", columns=("TEST_DATA",))[0])

prints

<xarray.Dataset> Size: 3GB
Dimensions:    (row: 1374420, TEST_DATA-1: 64, TEST_DATA-2: 4)
Coordinates:
    ROWID      (row) int32 5MB dask.array<chunksize=(10000,), meta=np.ndarray>
Dimensions without coordinates: row, TEST_DATA-1, TEST_DATA-2
Data variables:
    TEST_DATA  (row, TEST_DATA-1, TEST_DATA-2) complex64 3GB dask.array<chunksize=(10000, 64, 4), meta=np.ndarray>
Attributes:
    __daskms_partition_schema__:  (('FIELD_ID', 'int32'), ('DATA_DESC_ID', 'i...
    FIELD_ID:                     0
    DATA_DESC_ID:                 0

[sjperkins]

One extra note - this doesn't work if you only select the column with unknown dimensions:

Yes. Unfortunately due to the fact that dask-ms was designed to handle arbitrary tables produced by the CTDS, it doesn't really reason about the MS structure very deeply -- so the returned datasets will only have chan or corr dimensions if standard columns are read to provide this information.

Is this a critical blocker to your current workflows or could this be left to a future PR? I'd imagine that dask-ms convert should work with this change as it bulk exports MS columns.

[JSKenyon]

Is this a critical blocker to your current workflows or could this be left to a future PR? I'd imagine that dask-ms convert should work with this change as it bulk exports MS columns.

Not really, as the schema option still exists (QC always used it for extra model/weight columns). As you say, this should be good enough on the convert. Just thought it worth mentioning as it was the first thing I stumbled over when trying out the branch.

[landmanbester]

Is this a critical blocker to your current workflows or could this be left to a future PR? I'd imagine that dask-ms convert should work with this change as it bulk exports MS columns.

This will work perfectly almost every time. Thanks @sjperkins