Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adapt to changes between astropy 1.2 and 1.3 in how table units are handled #62

Closed
dkirkby opened this issue Apr 17, 2017 · 3 comments · Fixed by #63
Closed

Adapt to changes between astropy 1.2 and 1.3 in how table units are handled #62

dkirkby opened this issue Apr 17, 2017 · 3 comments · Fixed by #63

Comments

@dkirkby
Copy link
Member

dkirkby commented Apr 17, 2017

Quoting from @sbailey in [desi-data 2788]:

astropy 1.3 made a fairly major change to how units are treated for Tables, which breaks desi code, especially specsim that uses units a lot.  In particular, consider the following code that creates a Table column with units "Angstrom" and then assigns a unit-less numpy array:

import astropy
import numpy as np
from astropy.table import Table
import astropy.units as u

t = Table()
t['x'] = np.arange(10) * u.Angstrom
unit_before = t['x'].unit
t['x'] = np.arange(10)
unit_after = t['x'].unit
print(astropy.__version__, unit_before, unit_after)

astropy 1.2.1 preserves the original units, while astropy 1.3.2 blows them away:

1.2.1 Angstrom Angstrom
1.3.2 Angstrom None

This is problematic for specsim which creates its output tables with units and then internally works without units for speed while expecting the outputs to retain their units.  e.g. specsim.simulator line 501 is now losing its units while assigning the output flux.

This broke quickgen with desiconda/20170414-1.1.2-spectro which uses astropy 1.3.2, since quickgen checks the units of specsim outputs to make sure it knows whether it is getting erg/s/cm2/A or 1e-17 erg/s/cm2/A .  specsim does test with astropy 1.3.2 but apparently doesn't have a check for output flux units.

I haven't found mention of this on astropy issues to know whether they consider this a bug that was fixed or a bug that was introduced.

Options:
1. remove units checks in quickgen.  I think the specsim output *values* are still correct, but I haven't explored this enough to be sure that the change in units behavior doesn't mess something else up more subtly.
2. throw away desiconda/20170414-1.1.2-spectro and cut a new version pinned to astropy1.2.1 until this is fixed in astropy or specsim
3. fix in specsim (while making sure that we don't degrade performance)
4. stop using astropy.units all together; they prevent some bugs while causing others (e.g. astropy #5961 too)

Background/credit: John originally reported specsim testing problems with astropy 1.3.2 a month ago but we didn't finish tracking that down.

This issue is to investigate option 3 above, and to ensure that specsim gives the same results for both versions of astropy.

@dkirkby
Copy link
Member Author

dkirkby commented Apr 17, 2017

For testing purposes, I need both versions of astropy installed:

conda create --name astropy121 astropy=1.2.1 scipy pyyaml
conda create --name astropy132 astropy=1.3.2 scipy pyyaml

Since I don't specify the python version, it uses python2.7 from my root conda install (and keeps the new environments relatively lightweight).

[Updated to add scipy & pyyaml since these are needed to test specsim.simulator]

@dkirkby
Copy link
Member Author

dkirkby commented Apr 17, 2017

I can reproduce @sbailey's inconsistent behavior between 1.2.1 and 1.3.2 but I think the fix is straightforward: use t['x'][:] = ... after the column has been initially created. Otherwise, the syntax t['x'] = ... should really overwrite the original column and 1.2.1 was broken. This pattern of using [:] for existing columns is already used for many of the Simulator columns, so I propose to use it consistently to address this problem.

@dkirkby
Copy link
Member Author

dkirkby commented Apr 30, 2017

For the record, the change in astropy 1.3 table behavior is documented at http://docs.astropy.org/en/stable/table/modify_table.html#api-change-in-replacing-columns

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant