Assertion errors in tractor.optimize

[knyland]

Hi Dustin,

I've discovered that for complex fields, the minimization routine in tractor.optimize can "blow up" (i.e., the fit begins to diverge). This produces an assertion error at line 48 of pointsources.py. This is understandable, and can be "fixed" for a given source of interest by adjusting the size of the fitting region. However, my end goal is to do photometry on a survey with a large number of sources, and it would be impractical to individually define fitting region sizes using trial and error. Since this seems like a fairly standard application of the Tractor software, do you have any advice on how I might be able to catch an instance like that where the fit doesn't converge during a tractor.optimize step, and force it to continue? I have a few ideas of how to work around it, but it seems like others must have run across this as well. Ideally, it would be useful for tractor.optimize to provide a warning and produce masked arrays of NaNs (or something clever like that) for other tractor classes/functions to manipulate.

I've attached some figures showing the multiwavelength data for one source that I'm working with to demonstrate the problem. The first figure has 12 optical/IR cutouts with radii of 5". Tractor.optimize fails (line 48 of pointsources.py) on the very first cutout, the Ks-band image on the top left (again, not surprising as there are multiple bright point sources present). The 2nd figure is identical to the first, but the cutout radius shown (and used during optimization) is 4". An example of model and chi arrays successfully generated using 4" cutout radii is shown in the 3rd figure (yes, I'm aware it has other issues; I am only concerned with getting the code to run consistently at this point).

Thanks!!! :)
~Kristina

P.S. - Exact error identified in ipython's debug mode:

/Users/knyland/tractor/tractor/pointsource.pyc in getModelPatch(self, img, minsb, modelMask)
46
47 if upatch.patch is not None:
---> 48 assert(np.all(np.isfinite(upatch.patch)))
49
50 p = upatch * counts

AssertionError:

Field_74_optimize_failure.pdf
Field_74_optimize_success.pdf

[dstndstn]

That assertion error is thrown when a point source is being rendered. It first renders a unit-flux image of the PSF, and then scales it by the number of counts. That assertion is thrown when the unit-flux image has non-finite values. It's an assertion because it indicates that something is very wrong with the PSF model. Could you please print out the PSF model where that error occurs? "print self.psf" in pdb. What kind of PSF model are you using?
thanks

[knyland]

Hi Dustin,

Thanks so much for getting back to me, I really appreciate the help!

I tried the following to get you the output of self.psf:

python -m -pdb scriptname.py
print self.psf

but got a "self is not defined" error. But I don't have much experience
with the pdb module, so maybe I forgot to do something obvious.

The psf models used are generated following the examples in the Tractor
documentation using NCircularGaussianPSF. For instance:

ks_band_image = Image(data=subim, invvar=np.ones_like(subim) /
ksskynoise**2,
psf=NCircularGaussianPSF(kspsig,kspsigwt),wcs=subwcs,photocal=NullPhotoCal(),sky=ConstantSky(0.0))

where kspsig=[1.60,3.09,10.0], kspsigwt=[0.59,0.27,0.135], and ksskynoise
is defined in the fits header (SKYNOISE).

The Gaussian sigma and weight parameters above were determined by
optical/IR experts (not me) and may need to be improved in the future.
However, I don't think the psf model can be too bad if tractor.optimize
will run for some cutout sizes but not others . . . what do you think?

I've attached Ks-band cutout fits files with radii of 4" (success) and 5"
(failure) if you want to take a closer look at them. The center of the
cutout should be (36.736793188075652, -4.3744872499777552). It seems like
the intruding point source near the bottom of the 5"-radius image could be
to blame . . .

If you have any ideas at all please let me know!

Thank you again!!!!

That assertion error is thrown when a point source is being rendered. It
first renders a unit-flux image of the PSF, and then scales it by the
number of counts. That assertion is thrown when the unit-flux image has
non-finite values. It's an assertion because it indicates that something
is very wrong with the PSF model. Could you please print out the PSF
model where that error occurs? "print self.psf" in pdb. What kind of PSF
model are you using?
thanks

---

Reply to this email directly or view it on GitHub:
#73 (comment)

Dr. Kristina Nyland
Postdoctoral Researcher
NRAO
520 Edgemont Rd.
Charlottesville, VA 22903

[dstndstn]

Hi,
In pdb, you have to actually run the code up to the point where the assertion is thrown (much like you've done in your ipython debug session, I'm guessing; I've never used that). When you start pdb, you probably have to "continue" to get it to actually run, and then wait fro the assert to trigger, and then print the value of the PSF at that point.

Are you freezing the PSF model, or are you fitting that as well? If you're trying to fit that, it could be that the PSF model is going wild. Otherwise, I don't see how an NGaussianPSF can yield non-finite values.

cheers,
--dustin

[knyland]

AHA! I guess the psf wasn't frozen. Including a freezeAllRecursive call
followed by a thawPathsTo('brightness') fixed the problem. Although I
still don't understand why the optimization worked at all with a varying
psf for certain cutout sizes but not for others . . . very weird.

I previously misunderstood that tractor.freezeParam('images') was all that
was needed to do a standard fit (it says "# Freeze all image calibration
params -- just fit source params" in the example . . . ).

Assuming I understand correctly now and the following isn't nonsense, you
might want to update the example under Optimization/Fitting in your
documentation
(http://thetractor.org/doc/intro.html#thawing-freezing-params) to include
freezeAllRecursive and thawPathsTo('brightness') steps since I think this
is probably the most common application of the tractor package anyway.

Thanks for your help!

~Kristina

Hi,
In pdb, you have to actually run the code up to the point where the
assertion is thrown (much like you've done in your ipython debug session,
I'm guessing; I've never used that). When you start pdb, you probably
have to "continue" to get it to actually run, and then wait fro the assert
to trigger, and then print the value of the PSF at that point.

Are you freezing the PSF model, or are you fitting that as well? If
you're trying to fit that, it could be that the PSF model is going wild.
Otherwise, I don't see how an NGaussianPSF can yield non-finite values.

cheers,
--dustin

---

Reply to this email directly or view it on GitHub:
#73 (comment)

Dr. Kristina Nyland
Postdoctoral Researcher
NRAO
520 Edgemont Rd.
Charlottesville, VA 22903

[dstndstn]

tractor.freezeParams('images')
should indeed freeze all of the image calibration parameters.

You can use the
tractor.printThawedParams()
function at any time to get an idea of what's going on.

cheers,
--dustin

On Thu, Dec 3, 2015 at 5:31 PM, Kristina Nyland [email protected]
wrote:

AHA! I guess the psf wasn't frozen. Including a freezeAllRecursive call
followed by a thawPathsTo('brightness') fixed the problem. Although I
still don't understand why the optimization worked at all with a varying
psf for certain cutout sizes but not for others . . . very weird.

I previously misunderstood that tractor.freezeParam('images') was all that
was needed to do a standard fit (it says "# Freeze all image calibration
params -- just fit source params" in the example . . . ).

Assuming I understand correctly now and the following isn't nonsense, you
might want to update the example under Optimization/Fitting in your
documentation
(http://thetractor.org/doc/intro.html#thawing-freezing-params) to include
freezeAllRecursive and thawPathsTo('brightness') steps since I think this
is probably the most common application of the tractor package anyway.

Thanks for your help!

~Kristina

Hi,
In pdb, you have to actually run the code up to the point where the
assertion is thrown (much like you've done in your ipython debug session,
I'm guessing; I've never used that). When you start pdb, you probably
have to "continue" to get it to actually run, and then wait fro the
assert
to trigger, and then print the value of the PSF at that point.

Are you freezing the PSF model, or are you fitting that as well? If
you're trying to fit that, it could be that the PSF model is going wild.
Otherwise, I don't see how an NGaussianPSF can yield non-finite values.

cheers,
--dustin

---

Reply to this email directly or view it on GitHub:
#73 (comment)

Dr. Kristina Nyland
Postdoctoral Researcher
NRAO
520 Edgemont Rd.
Charlottesville, VA 22903

—
Reply to this email directly or view it on GitHub
#73 (comment).

[knyland]

Ok, got it. A varying psf was not the issue, I guess the varying position
or shape parameters left by doing only freezeParams('images') before the
optimization step must have caused the issue.

Including cat.freezeAllRecursive() and then cat.thawPathsTo('brightness')
fixed the problem.

Thanks again!

tractor.freezeParams('images')
should indeed freeze all of the image calibration parameters.

You can use the
tractor.printThawedParams()
function at any time to get an idea of what's going on.

cheers,
--dustin

On Thu, Dec 3, 2015 at 5:31 PM, Kristina Nyland [email protected]
wrote:

AHA! I guess the psf wasn't frozen. Including a freezeAllRecursive call
followed by a thawPathsTo('brightness') fixed the problem. Although I
still don't understand why the optimization worked at all with a varying
psf for certain cutout sizes but not for others . . . very weird.

I previously misunderstood that tractor.freezeParam('images') was all
that
was needed to do a standard fit (it says "# Freeze all image calibration
params -- just fit source params" in the example . . . ).

Assuming I understand correctly now and the following isn't nonsense,
you
might want to update the example under Optimization/Fitting in your
documentation
(http://thetractor.org/doc/intro.html#thawing-freezing-params) to
include
freezeAllRecursive and thawPathsTo('brightness') steps since I think
this
is probably the most common application of the tractor package anyway.

Thanks for your help!

~Kristina

Hi,
In pdb, you have to actually run the code up to the point where the
assertion is thrown (much like you've done in your ipython debug
session,
I'm guessing; I've never used that). When you start pdb, you probably
have to "continue" to get it to actually run, and then wait fro the
assert
to trigger, and then print the value of the PSF at that point.

Are you freezing the PSF model, or are you fitting that as well? If
you're trying to fit that, it could be that the PSF model is going
wild.
Otherwise, I don't see how an NGaussianPSF can yield non-finite
values.

cheers,
--dustin

---

Reply to this email directly or view it on GitHub:
#73 (comment)

Dr. Kristina Nyland
Postdoctoral Researcher
NRAO
520 Edgemont Rd.
Charlottesville, VA 22903

—
Reply to this email directly or view it on GitHub
#73 (comment).

---

Reply to this email directly or view it on GitHub:
#73 (comment)

Dr. Kristina Nyland
Postdoctoral Researcher
NRAO
520 Edgemont Rd.
Charlottesville, VA 22903

[dstndstn]

I would check with tractor.printThawedParams() that it is doing what you
think it is doing. When you do "freezeAllRecursive()", it's going to
freeze all levels of the parameter space. Thawing just down to the
'brightness' might not be enough, since the brightness probably has .r, .g
and so on sub-parameters. All of those need to be thawed as well.