ImageCalibration problem

[pfile]

as seen near the end of this thread: http://pixinsight.com/forum/index.php?topic=1930.msg12207#msg12207



the flat frame started life as an f32 fits file, as integrated by MaximDL. i have converted it to i16 to match the raw science frame.

if i use the flat frame as-is in ImageCalibration, the result is as in the top right image in the screenshot above: the flat has pretty much destroyed the science frame.
if i use the histogram tool to compress the data in the flat all the way to the right and use that new flat in ImageIntegration, the results are satisfactory.

it seems like ImageCalibration is not scaling the flat properly? or is this user error?

here are the science and flat files in question:

http://public.me.com/ix/pfile/pixinsight_1.6_flat_problem.zip?disposition=download+1274505738440

thanks.

[NKV]

What about ImageCalibration settings?
One time I forgot to disable "Calibrate Flat" and got similar overcorrection, because MasterFlat was double calibrated.

PS MaximDL adds a Pedestal. Care about this?

[Niall Saunders]

Hi Guys,

I am struggkling a bit with Flats in IC myself at the moment - I can get my calibration data to 'behave' when I run through the individual calibration steps 'manually', using  nothing more than PixelMath and ImageContainers, but I still see all my 'prime' dust-bunnies after IC.

So, further investigation needed.

However, whilst I am waiting for your two source files to download, my first thought is how Juan has dealt with the 'divide-by-zero' scenario that can occur if the MasterFlat contains 'zero-level' ADU data. To this end, I plan to have a quick look at my MasterFlats as presented to IC, and to see if I get better results if I use PixelMath to 'normalise' the MasterFlat first.

And, in this case, 'normalise' is the WRONG term. What I will be doing is 'maximising' the ADU data, by 'adding' a fixed value (which will be [ 1.0 - max($T) ] to each pixel value, such that the max($T) value then becomes '1.0'. This 'addition' will also guarantee that there are NO 'zero' ADU values left behind in the image (thus eliminating the 'divide-by-zero' problem in the first place).

Of course, this is ENTIRELY POSSIBLE in my calibration images, because my Flats (used to create the MasterFlat) were all exposed such the the nominal 'maxADU' value was around 50% full-range (i.e. ~32,000) in the first place.

But, if you were not in control of the acquisition of the data in the first place, then you cannot be sure WHAT was actually acquired.

Cheers,

[Niall Saunders]

OK, continuing from where I left off last time . . .

Now that the 50Mb(!!) of data has arrived, and has been opened in PI, I see the attached histogram from the Flat image

Whilst, at first glance, the data seems to fit my previous suggestions (that everything should be 'maximised') this is NOT actually the case. With the image loaded, and with the Histogram in 'Track View' mode, if you click the <Auto Zero Shadows> and <Auto Zero Highlights> buttons, you will see that NOTHING changes.

This is confirmed by the second attachment - which is the Statistics window tracking the same view. Once again, you can see that Minimum = 0.00000 and Maximum = 1.00000.
[EDIT : Sorry, I forgot to attach that image - but, you can verify this for yourself anyway]

If you use a MasterFlat with these characteristics, the extremely 'low' ADU data is just going to 'blow-out' your data.

(By the way, I see NO EVIDENCE of any 'pedestal' having been ADDED to an original set of data. If this had been the case, you would NOT see the little 'humps and bumps' down at the bottom end of the Histo - the data would be 'zero' up to the ADU valuse that had been used as the 'pedestal'. It could be, however, that a 'pedestal' has been 'simulated', by MULTIPLYING the data by some fixed value. That would retain the 'rubbish' down at the bottom end, but would also 'accentuate it' at the same time.)

In any case, there is a way forward. Looking at the Histogram of the supplied MasterFlat, it seems that there is NO 'really useful' data below an ADU value of around 0.65. We can check this by a very simple PixelMath expression:

iif($T<0.65, $T, 0.65)
Note that it is ESSENTIAL to un-click the <Rescale result> tickbox[/tt]

This will set all ADU values 'above 0.65' to a fixed value OF 0.65. Apply this to the MasterFlat (create a copy image, for convenience) and you will see an image with little or no 'structural detail' - i.e. little or no 'information' (I haven't attached an image demonstrating this - try it for yourself)

So, now that we have determined THAT fact, we can 'swap' our logic, and keep all the data ABOVE an ADU of 0.65, as follows:

iif($T>=0.65, $T, 0.65)
Note that it is ESSENTIAL to un-click the <Rescale result> tickbox[/tt]

Here it is also important to note that we did not use '0.0' for the case when $T<0.65 - that is exactly what we do NOT want to happen, as this would introduce a huge area of 'divide-by-zero' problems into our MasterFlat. Once this PixelMath expression has been applied to the original MasterFlat, the new image will show a Minimum of 0.65000 in the Statistics window - which is now beginning to get us somewhere!!

I have attached the newMasterFlat image, captured after the application of the PixelMath process (second attachment). This image has had an STF applied, but - unusually - in this case the mid-tone slider has had to be raised UP (to about 0.9 in this case) because the image is 'mostly bright', unlike our 'mostly dark' normal astroimages.

Finally, we can now use this newMasterFlat image on our RAW frame, by applying the following (standard 'flat division') expression:

$T/newMasterFlat
Note that it is now ESSENTIAL to re-enable the <Rescale result> tickbox[/tt]

The final result is shown in the third attachment, and has been given a reasonably aggressive STF to let the background show through.

Interestingly though, I do not actually see any evidence of the dust-donuts in the original RAW image, and I also do not see 'over-division' of said dust-donuts in my attempts at flat calibration, so I am then immediately concerned as to just how 'raw' the supposedly 'raw' images actually are.

Anyway, it has been a fun exercise for me, and maybe it might help others.

Cheers,

[pfile]

i can't explain this yet, but i have tried what you are suggesting and i get the same result, whether i divide the flat or i use ImageCalibration. the science frame is still way blown out. i guess i still don't understand the basics here - if you divide any part of that image by 0.65, won't that part just be ridiculously bright?

are you also using pixinsight 1.6 ?

as for the raw-ness of the science frame, there are definitely donuts there, but they are hard to see. they are more obvious when i've stacked the 20 uncalibrated science frames i have and also applied curves. in the attachment i've also given a crazy STF so they can be seen. i did ABE on the image to try to correct the vignetting.

the cause of the junk at the bottom end of the histogram seems to be a stacking artifact. if you look at the bottom of the flat there are 10 rows of dark pixels. it's almost like maximDL has tried to register the flat subexposures before stacking them. in fact my darks look the same way. this would be a fatal error as far as i'm concerned. EDIT: i think this stuff is actually an artifact in the old apogee driver. i can see that my raw science frames from last year look the same.

also, NKV: all the calibration checkboxes were unticked.

[Niall Saunders]

Hi again,

OK - I wonder whether my explanation may have just confused things somewhat - I knew what I was trying to say, but perhaps I lost the plot somewhat on the way.

• Take the original MasterFlat image and apply this Pixel Math expression to it : iif($T>=0.65, $T, 0.65)
  Note that it is essential to UN-CHECK the <Rescale result> tickbox, and request that PixelMath creates a new image
  
• Apply the PixelMath process to the original MasterFlat
  
• Rename the new image as "newMasterFlat"
  
• Create a new PixelMath expression, this time it will be : $T/newMasterFlat
  Note that it now important to RE-CHECK the <Rescale result> tickbox, and you should still request that PixelMath creates a new image
  
• Apply the new PixelMath process to the original RAW frame
  

This should give you the 'properly' flat-calibrated frame that I was able to create.

Cheers,

[pfile]

no, no, i understood. that's exactly what i did. i'm left with a histogram that looks pretty much the same as the original, except no "junk" at the low end, and a big spike at the left edge of the histogram where all the black pixels were replaced with 0.65.

what i am getting at is simply this: the outer corners of the flat have values in the 0.7 range. dividing the light by 0.7 is the same as multiplying those pixels by 1.4. however, if you take a background pixel of the science frame that's near the middle and divide it by a background pixel that's near the edge, you get about 1.014. that's the factor you'd have to multiply an edge pixel by to make it as bright as a center pixel. multiplying it by 1.4 is way overcorrecting the science frame.

that's what i'm trying to understand. maybe this means the master flat is just "wrong" because it gives a greater variance between center pixels and corner pixels than i see in the science frame? still this does not explain why your version of the flat works properly.

[Juan Conejero]

Hi pfile,

I've downloaded your zip file and have made some tests. The flat frame is incorrect IMO, in the sense that it does not reproduce the illumination profile in the light frame. Along with that, there are a few artifact rows at the bottom of the flat that are incorrect, but ImageCalibration protects itself from these artifacts so they are not causing problems.

You can use ATrousWaveletTransform in layer preview mode to check that the illumination profiles don't match. Set 5 or 6 layers, 5x5 B3 Spline scaling function and select the residual layer R. Select Layer Preview = All Changes and apply to both images with no STF active. In this way you can isolate large-scale structures. You'll see how illumination is clearly excessive on the central region of the flat frame. For this reason this flat frame it is overcorrecting your light frame. Along with that, the flat does not reproduce two bright regions at the left and right borders of the image. It looks like this flat does not correspond to your light frame.

You can easily apply the flat correction with PixelMath. As Niall has pointed out, first you must remove those bad rows at the bottom of the flat:

Max($T,0.65)

with Rescale result disabled.

Then you can apply a standard flat frame correction to your light frame:

$T*Avg(flat)/flat

with Rescale result enabled. This will give you exactly the same result as ImageCalibration: the central part of the image is severely overcorrected.

I have also tried with a synthetic background model (DBE) and the result is excellent (and the model is completely different from your flat frame!).

[pfile]

Juan,

thanks very much for looking at this. i've been busy with work and home stuff the last few days and have not been able to spend any time on this.

i suspected that the flat frame was just wrong after thinking about it for a while. although it is possible to manipulate this flat until it divides properly, it seems like a proper calibration routine should never attempt to do this.

while i'm not terribly sure why the apogee driver change would cause such a drastic difference between the flat illumination and the science frame's illumination, it is a fact that this flat was created from subs from the older version of the driver, which was producing 4098x4098 frames. there is a glimmer of hope - lightbuckets has completed generating the new calibration master frames with the new camera driver. maybe tonight i'll be able to download those and see what happens in ImageCalibration.

edit: whoops, i am confused. it looks like all my flats were generated after the driver update. so in theory they should be "correct" both in the sense that the came from the same driver and also that they were taken within a couple of days of the science frame aquisition. at this point i really don't know what to say, or what to do next. when i massaged the flats they did seem to properly correct the dust donuts. i did not look carefully enough to see if they had messed up the overall illumination.