ImageCalibration Bug

[NKV]

Hi, just small bug.

Impossible to left blank Output directory.
(tested on Win7 64)

Best regards,
Nikolay.

[moscow]

Hi, just small bag.

Impossible to left blank Output directory.
(tested on Win7 64)

Best regards,
Nikolay.

... same on XP 32

[vicent_peris]

No problem in Linux 64bit

[Juan Conejero]

This is a confirmed bug, thanks for reporting it so quickly 

Fixing it is extremely easy for those who can compile a PixInsight module. There are excellent PI developers here, so ... 

In ImageCalibration.cpp, line 1455, the following two lines:

Code: [Select]

      if ( !File::DirectoryExists( outputDirectory ) )
         throw( "The specified output directory does not exist: " + outputDirectory );must be changed to:

Code: [Select]

      if ( !outputDirectory.IsEmpty() && !File::DirectoryExists( outputDirectory ) )
         throw( "The specified output directory does not exist: " + outputDirectory );
That's all. I am taking a brief rest in an extremely nice rural location where I can't compile code --you know, green, Spring, trees, cows, dark skies and that kind of things

I'll release a new ImageCalibration module next week with all bugs fixed.

[NKV]

Fixing it is extremely easy for those who can compile a PixInsight module. There are excellent PI developers here, so ... 

Juan, thank you. I done it. 8)

[NKV]

Next two bugs.
1) ImageCalibrationInstace not preserve Overscan Source&Target parameters.

2) Negative value during overscan calibration.
Example: I want to prepare master Bias, but before integration I need to apply overscan correction to individual biases files (using current Imagecalibration). After overscan correction I see too many zero pixels.
I mean the overscan correction don't work properly without additional pedestal. 

Workaround: I add 100ADU to final image:
ImageCalibrationInstance.cpp line 221

Code: [Select]

      for ( int c = 0; c < target.NumberOfChannels(); ++c )
//         if ( 1 + v[c] != 1 )
         {
            target.SelectChannel( c );
            target -=  v[c] - 100.0 / 65535.0;
         }
   }
Juan, please add control of pedestal to ImageCalibration module.

[Juan Conejero]

Hi Nikolay,

1) ImageCalibrationInstace not preserve Overscan Source&Target parameters.

This is a known bug that is already fixed in the next version. Thanks for pointing out anyway

2) Negative value during overscan calibration.

Yes, negative values are a problem in some cases. A pedestal option will be added to the next version. However, instead of implementing it in the overscan correction routine, I think it would be more appropriately done as a final step. How about adding the pedestal in the WriteCalibratedImage() routine (line 1078)? In this way we'd be correcting negative values that could arise at any stage during the whole calibration process. What do you think?

[NKV]

Hi Juan,

Yes, negative values are a problem in some cases. A pedestal option will be added to the next version.

Thank you.

However, instead of implementing it in the overscan correction routine, I think it would be more appropriately done as a final step. How about adding the pedestal in the WriteCalibratedImage() routine (line 1078)? In this way we'd be correcting negative values that could arise at any stage during the whole calibration process. What do you think?

You are right. When we must add pedestal ? Only if enabled overscan correction ? I think dark calibration can produce negative value. So you are right, better to add pedestal as a final step (n the WriteCalibratedImage() routine)
But how many times we must add pedestal? I think one time enough. But I run Image Calibration process twice ( first for overscanCalibration of all images, then integrate masters files and second to apply masters to light frames )

So, thinking about history: I want to see the value of pedestal in FITS header and I think ImageCalibration have to read header, if the pedestal value exist in header deduct the value from image, do calibration, and at final step add new pedestal value to image,
and set the value in FITS header, like in other software :

Code: [Select]

PEDESTAL = -100 /Correction to add for zero-based ADU
What do you think?

[NKV]

One more bug:
My initial frame size is 4157x4142 (it's KAF-16803 + overscan area)
After overscan correction + integration MasterDark/MasterBias frame size is 4096x4096.
OK, now I am ready to apply ImageCalibration to Light frames ( which size is 4157x4142 )

But I get error: "Incorrect image geometry", because my master files already overscan cropped during preparation.

I mean: in which situation need to apply overscan correction to master files?

IMHO preparation of master files should finished before and don't need to apply overscan to masters during Light calibration.
In other words: overscan should be applied to Target Frames only. Or (I don't know why) need to populate checker for overscan enable in Bias/Dark/Flat section.

Best regards,
Nikolay.

[Juan Conejero]

Hi Nikolay,

That shouldn't happen. The "Calibrate" checkboxes should be *unchecked* for your master dark and bias if they are already calibrated. Are they?

Please confirm as this is a 1.6.1 stopper bug if confirmed.

[NKV]

Hi Nikolay,

That shouldn't happen. The "Calibrate" checkboxes should be *unchecked* for your master dark and bias if they are already calibrated. Are they?

Please confirm as this is a 1.6.1 stopper bug if confirmed.

Juan,
Impossible to completely calibrate Master Dark in advance.
 
First problem:
To prepare Mater files from 4157x4142 frame I doing this:
MasterBias: Calibrate (overscan correction , +100 pedestal) > Integrate. I got frame size is 4096x4096 with median 122, Min 114, StdDev 1.24 ADU.
MasterDark: Calibrate (overscan correction , +100 pedestal) > Integrate. I got frame size is 4096x4096 with median 179, Min123, StdDev 53.6 ADU.
I can't simple deduct MasterBias from MasterDark, because its generate negative values in many pixels. So I can't create masterDark which consist only thermal noises. So I can't to use darkFrame Optimisation. It's first problem.

Second problem:
My light frame size 4157x4142. I hope it's good idea to apply overscan correction and dark subtraction in one round. Right?
OK. What I want from ImageCalibration for my Light frame?
1) Apply overscan correction to Light + crop light frame to normal size 4096x4096. (need to enable overscan checker)
2) Subtract Bias from Light. (need to enable Master Bias checker) note added pedestal during preparation MasterBias must disappear.
3) Subtract Bias from Dark. (need to enable Master Dark checker + Calibrate checker). note added pedestal during preparation Master files must disappear and I hope internally negative value not problem for PixInsight. So we got "clear thermal noises data".
4) Optimize "clear thermal noises data" to Light.
5) Subtract thermal noises from Light.
6) Flat calibration.

Problem is the point 3 can't be finished, because "Master Dark Calibrate checker" initiate overscan correction for masterDark (which already overscan corrected).
One question: Juan, why you apply overscan to master's files? I think overscan correction can be applied only to individual files, no reason to apply it to integrated images.   

As solution for all the problem, I like to see in next version:
a) FITS header with PEDESTAL value, which will automatically added upon save and deduct upon load files to ImageCalibration module.
b) Independent checker for "enable overscan" in MasterDark section. But I think better don't apply overscan to MasterFiles.

Juan, I think its very very easy for you. It is not stopper, it is very short pit stop, where you can relax from hard work.

[Juan Conejero]

Hi Nikolay,

The pedestal feature is already available. See this post where I published the source code of ImageCalibration a few days ago. Could you compile it for Windows and see if this works for you?

Let me think about the overscan problem a bit more. No, I cannot relax even a little bit because I am working against the clock. This must be published immediately for several reasons. However I am doing my best to finish everything, and this is no exception. Don't worry, I work very well under high pressure (I am accustomed) 

[Juan Conejero]

OK, after talking with Vicent, I think we have this more or less clear now.

* An optional pedestal addition is already implemented. ImageCalibration will write the added value to a PEDESTAL FITS header keyword.

* I will implement an optional pedestal subtraction option. The pedestal value can be specified in two ways:
- As a user-specified literal value in ADU.
- As the value of a user-specified FITS header keyword (PEDESTAL by default), when it is present.

* You'll have no problem with your point 3:

3) Subtract Bias from Dark. (need to enable Master Dark checker + Calibrate checker). note added pedestal during preparation Master files must disappear and I hope internally negative value not problem for PixInsight. So we got "clear thermal noises data".

Bias subtraction, along with overscan correction, must be applied to your individual dark frames when you calibrate them. You first calibrate your bias frames applying overscan correction, then you integrate them to get a master bias. Then you can calibrate your dark frames with overscan+bias, and integrate them to get a master dark. Since the pedestal subtraction will be implemented, you'll have no problems during these phases.

Note that to perform a truly correct calibration work, you must also calibrate your individual flat frames with overscan and master bias subtraction, and integrate them into a master flat frame. That is, all master calibration frames must be completely pre-calibrated when you calibrate your science frames.

When you calibrate your science frames, you'll have to uncheck the "Calibrate" check boxes for the three calibration master frames (bias, dark and flat). You'll have to specify the appropriate pedestal subtraction because in your case the three masters will have a pedestal added, due to your negative values problem. Everything should work fine this way

[NKV]

The pedestal feature is already available. See this post where I published the source code of ImageCalibration a few days ago.

Oh... I lost the post completely... A am sorry and many thanks.

Everything should work fine this way

Yes, with new ImageCalibration should work fine. I will check it soon, and report to you. Thank you!

[NKV]

Hi, ImageCalibration version 1.6.1 working fine.
IMHO Dark & Flat scaling perfectly calculated.
Juan, thank you.

Code: [Select]

* Subtracting pedestal keyword 'PEDESTAL': 100.0000 DN

Computing master flat scaling factors ...
s0 = 0.332834

Calibrating target frame 1 of 1
Reading FITS: 16-bit integers, 1 channel(s), 4157x4142 pixels: 100%

Dark scaling factors:
k0 = 0.979
Writing FITS image: 64-bit floating point, 1 channel(s), 4096x4096 pixels: 100%