1.6.9.651 osx x86_64: can't calibrate files (master flat can not be scaled)

[pfile]

i just upgraded to the latest, and tried to calibrate some files from last night.

the master calibration frames i am using were prepared using astropixel's tutorial and are bayerRGB files with no CFA data, and have always worked in the past.

when it comes time for the ImageCalibration tool to compute the scaling factors for the master flat, it comes up with 0,0,0 and quits. i reverted to the previous release of 1.6.9 and was able to calibrate my CR2 files just fine with the same master calibration files. to be honest the scaling factors are quite low (0.013, 0.017,0.04), so maybe there's something wrong with my files. but the calibrated lights look very flat to me.

by the way, because of the way 651 dynamically finds it's process binaries, i could not revert to the previous version of 1.6.9 without moving PixInsight64.plist out of the way. otherwise it chokes trying to dereference the new pointer to the dylib directory...

[pfile]

hmm, well... the files from last night do not look so flat. i don't understand this though because i calibrated them identically. perhaps something else is going on.

i do know that if i debayer my master flat and divide one of my raw lights by the master flat, the resulting frame is very flat. so i'm not sure why PI wants to scale the flat so drastically.

regardless the behavior between versions of 1.6.9 does differ, but it is possible it's the prior version that is buggy and proceeding with such a drastic scaling of the master flat...?

[pfile]

okay, now i can see that the last time i did this, i used 1.6.1 to calibrate the lights. so it would appear that there's a problem with ImageCalibration in both versions of 1.6.9.

[pfile]

okay, sorry for all the noise. i narrowed this down some more. even in 1.6.1, it seems that the flat scaling goes bad when the "calibrate" checkbox is ticked in the master flat section. this must be because i calibrated the master flat when i created it.

in 1.6.1 ImageCalibration seems to respect this checkbox.

in 1.6.9 (both versions) it would seem that ImageCalibration is ignoring the checkbox and always trying to calibrate the master flat. in the original 1.6.9 the scaling factors are small but the tool proceeds and the resultant calibrated light still shows vignetting. in 1.6.9.651, the scaling factors are determined to be 0 and the tool quits.

now, i think i might be going crazy because i seem to recall there was an update to ImageCalibration for this very problem, but now i can't find any evidence of that.

[astropixel]

You're not going crazy. I've had the same problem. Linux x86_64. Other than that no problems so far.

First. I created a new master bias/dark and flat per Vicent's tutorial.

I don't want to confuse the issues, as it is very easy to do so. So I'll state outright, that the presence of a master flat during the calibration of light frames whether the calibrate check box was selected or not, causes noise evaluation to fail during image integration.

I spent hours yesterday testing this, trying every combination of master bias/dark/flat - calibrate selected and not selected.

sub note: On the basis of the above. I assumed that no calibration of the Master Bias/Dark frames had taken place during the calibration of the Flat frames (although no errors were observed). And it was necessary to check the Master Bias/Dark Calibrate check box to avoid failure of noise estimates during image integration. This is consistent with Vicent's advice in another thread.

[pfile]

You're not going crazy. I've had the same problem. Linux x86_64. Other than that no problems so far.

First. I created a new master bias/dark and flat per Vicent's tutorial.

I don't want to confuse the issues, as it is very easy to do so. So I'll state outright, that the presence of a master flat during the calibration of light frames whether the calibrate check box was selected or not, causes noise evaluation to fail during image integration.

I spent hours yesterday testing this, trying every combination of master bias/dark/flat - calibrate selected and not selected.

sub note: On the basis of the above. I assumed that no calibration of the Master Bias/Dark frames had taken place during the calibration of the Flat frames (although no errors were observed). And it was necessary to check the Master Bias/Dark Calibrate check box to avoid failure of noise estimates during image integration. This is consistent with Vicent's advice in another thread.

noise eval fails completely, or just falls back to the k-sigma estimate? i see the k-sigma fallback all the time but i never made the connection with flats.

looks like there's some issues in ImageCalibration then, hopefully Juan can get to the bottom of them.

[astropixel]

The only solution that worked for me, though not perfectly, was to convert raw flats to .fit, integrate (Vicent's tutorial settings), and check all calibrate check boxes during calibration of the light frames.

This defaulted to k-sigma during noise estimation for individual light frames, but the final MRS succeeded.

The only variation that produced an identical result, was to calibrate the master flat independently - that is, following integration of the flat frames (as above). Then check the master bias/dark calibrate boxes - but not the master flat.

Any attempt to calibrate the flat frames and then integrate them into a master and then apply that to calibration of light frames caused MRS to fail at some point during image integration - though k-sigma succeeded on some frames.

Hope that all  makes sense. I think it boils down to not calibrating the flats before integration. The only method that succeeded was to calibrate the master flat following integration - by either of the first two methods above.

edit: An improperly calibrated flat shows up during debayering - the output/process file that appears behind the console window is distorted - should appear as a black linear frame - but that doesn't mean that MRS will necessarily work during II.

[astropixel]

Without putting too finer point on it. Is this a PI problem? Or is it a user problem?

[pfile]

as far as subexposure noise evaluation failing: i think there was some discussion of this earlier. i thought juan was going to look into it at some point. it seems to be related to DSLR raw files. so i don't think that part is user error.

i think there have got to be problems in ImageCalibration, since the behavior has definitely changed between 1.6.1 and 1.6.9...

maybe we need to post these files somewhere so juan can try it out and debug it.

[astropixel]

To be fair to PI, I may revisit the creation of flat frames and see if longer exposures make a difference. I'm not sure, but following a recent exercise I found that II is very sensitive to noise levels. Just to be sure, I intend running the exercise again.

We have been down this road before with master frames, and looking back at some of the images can't help feeling that flats have been a common denominator.

[Juan Conejero]

i think there have got to be problems in ImageCalibration, since the behavior has definitely changed between 1.6.1 and 1.6.9...

That's odd, since the ImageCalibration module is exactly the same in 1.6.1 and 1.6.9. In fact, I still have to fix a bug that was detected some months ago...

maybe we need to post these files somewhere so juan can try it out and debug it.

Yes! that would be useful to understand the problems you're having. Please upload the files and describe the issues in a 'compact form', so I can reproduce the same scenario that you have. The key here is reproducibility; if I can't reproduce the problems, it is very difficult for me to understand and fix them.

[pfile]

i think there have got to be problems in ImageCalibration, since the behavior has definitely changed between 1.6.1 and 1.6.9...

That's odd, since the ImageCalibration module is exactly the same in 1.6.1 and 1.6.9. In fact, I still have to fix a bug that was detected some months ago...

maybe we need to post these files somewhere so juan can try it out and debug it.

Yes! that would be useful to understand the problems you're having. Please upload the files and describe the issues in a 'compact form', so I can reproduce the same scenario that you have. The key here is reproducibility; if I can't reproduce the problems, it is very difficult for me to understand and fix them.

i guess the problem might be in some underlying piece of code. but anyway i'll package this junk up and describe exactly what the problem is. i also noticed that even in 1.6.1 if i check the "calibrate" box for the master flat, the flat is 'calibrated' but the resultant calibrated light frames are completely wacky.

i'm willing to believe there is a problem with my master flat, though it was created per vincent's tutorial and it's pretty straightforward. that never stopped me from screwing something up in the past, however. 

[astropixel]

Compact!

The only combination that works for me, is to calibrate the master frames during calibration of the light frames - that is, 'integrate only' to create master files and then 'check all the master frame Calibrate boxes' when calibrating the light frames.

However, if the master flat is calibrated independently and then applied to light frame calibration - success. The problem seems to extend from the calibration of the individual frames that go to make up the master flat.

I have spent a lot of time narrowing this down. But if this input is not particularly useful please let me know.

[astropixel]

Without putting a finer point on it. The files involved are large, but I'm quite happy to upload them to my ftp account. I presume that 3 of the lights and the master bias/dark and flat should do it, to reproduce the problem. Some guidance here would be helpful.

[Juan Conejero]

In fact if you can reduce the data set to just one light frame with its master bias and dark frames, that's sufficient. What is important is being able to reproduce the problem consistently.