How can I resolve flat calibration artificats?

[knro]

I'm quite new to PixInsight and image processing in general, so I'm not sure how to identify this artificat. I took HaRGB of NGC7000 last night, and then used the Batch Preprocessing script to process the images. I noted that all master lights are OK, except for the blue component. All images shown below are stretched via STF AutoStretch.

Here is the (awful) calibrated flat field for the blue component:



And here is the raw light blue image:



And here is the calibrated blue image:



You can see clearly see "rings" at the bottom. Any idea what is causing this?

[knro]

Well, the problem was with PixInsight preprocessing script settings. When I calibrated the images using the ImageCalibration tool, all artifacts are gone, including the noise and rings! Turning off "Optimize" in the master dark frame settings did the trick.

[vicent_peris]

Hi,

Are you using separated master bias and dark frames? I would know some additional data:

- Exposure time of your light image.
- Number of integrated bias and dark frames.
- Image sensor.


Regards,
Vicent.

[knro]

Hi Vincent,

I actually take back what I said about the problem as "fixed". I was looking at the wrong window when I thought the artifacts are gone, but they are still there!

Here are links to the files:

+ Master Bias: http://dl.dropbox.com/u/52658751/bias-BINNING_2.fit
+ Master Dark: http://dl.dropbox.com/u/52658751/dark-BINNING_2-EXPTIME_180.fit
+ Master Flat (Blue Filter): http://dl.dropbox.com/u/52658751/flat-FILTER_Blue-BINNING_2.fit

+ Sample Light Frame: http://dl.dropbox.com/u/52658751/NGC7000_Light_Blue_01.fits

- Exposure Time: 180 seconds
- 15 Bias Frames, 15 Flat Frames (for each filter), 5 dark frames, and 5 light frames. I didn't take "dark flats" since according to PixInsight, it's not necessary.
- Sensor: KAF 8300

[knro]

Well, I didn't get any reply to my previous post for a few months, and the problem REPEATED itself yesterday while taking images for M31. The _exact_ same issues and I am no where close to figuring out the problem.

Here is the master light frame, you can see all those horrendous ring artifacts. It seems the calibration process makes them more pronounced than in any individual frame! The following images were stretched using Pixinsight's Auto screen transfer function:



And here is the master flat frame in the Lum filter:



Here are the raw FITS files for reference:

+ Master Light Frame: https://dl.dropboxusercontent.com/u/52658751/light-FILTER_L-BINNING_1.fit
+ Master Flat Frame (Lum filter): https://dl.dropboxusercontent.com/u/52658751/flat-FILTER_L-BINNING_1.fit
+ Master Bias Frame: https://dl.dropboxusercontent.com/u/52658751/bias-BINNING_1.fit
+ Master Dark Frame: https://dl.dropboxusercontent.com/u/52658751/dark-BINNING_1-EXPTIME_30.fit

+ Sample Light Frame: https://dl.dropboxusercontent.com/u/52658751/M31_Light_L_13_2013-11-04T22%3A10%3A26.fits

I took the flat field frame using my OTA (Orion Sirius APO 120 mm) in front on a laptop screen with white background. There was a distance of 3-4 cm from the laptop screen to the OTA. The flat frame exposure in the luminosity filter was 0.5 seconds, resulting in 22,000 ADU which is within the linear range on my QSI 583wsg CCD. Took 30 flats for each filter using the same binning, temperature and focus as the rest of the images. I read suggestions for ADU as low as 15,000 while other say go for 50k since it is still within the linear range of the KAF-8300! I'm not sure which way, I might try taking flats again today at 45-50k range and see if it makes the dust rings go away. But anyone has any ideas on what possible be wrong?

[pfile]

unfortunately this looks to me like a classic case of "flats don't match the lights."

there are so many things that could have gone wrong... but the simplest and most onerous problem is that the dust motes could simply have shifted. assuming they are on your filter, either the dust motes moved as the wheel was turned, or they moved of their own accord as time has gone by.

another issue could be that the filter wheel is not putting the filter back in exactly the same place from move to move.

that embossed look is what you get when you take any image, shift it by a few pixels, and add it back to itself. the process of stacking the images is what makes it more pronounced; those shadowed edges are very subtle in the subexposures, but they are there too.

hopefully it's just a matter of taking new flats.

rob

[jkmorse]

Knro,

I have an idea since it was a problem I had when I first started out trying to do flat frames.  One of the critical factors is that everything has to stay exactly the same as when you took your lights.  If anything moves, even a little, the flats just won't work and in fact will make things much worse becasue now instead of subtracting the dust circles, you are simply adding new ones in different places.  You need to make sure that you are taking your flats right after you take your lights and before any part of your image train changes.  That means not refocusing, not removing the CCD from the telescope, not moving the telescope from its mount, etc., etc.  Also, you may need to consider a different method for shooting flats. 

I currently use a flat panel from Alnitak for my CDK12.5, but they also have smaller sizes and I used one of their smaller versions when I was shooting with my TeleVue 101.  Before that I built a light box using the instructions in Richard Berry's Handbook of Astronomical Imaging.  Those methods will give you a cleaner light and allow you to control the timing of the flat by dimming or brightening the light.  If you decide to stick with using your computer screen, you might want to consider putting a difuser on the front of the OTA.  Something as simple as stretching a tee shirt over the front will help. 

As to shooting the flats, I used to shoot with an 8300 as well and an ADU count in the 20,000 to 30,000 range is where you want to be.  You want a nice high count (so a floor of 20,000) but not too high so you don't bump into the non-linear count when the anti-blooming chip kicks in (thus the ceiling of 30,000).  I try to stay around 25,000 but anything in the range with work fine.  Also, try to shoot your flats for a little longer.  Everything I have read says your flats should be between 2 and 10 seconds.  I shoot 3 seconds for RGBs though narrowband filters can take up to 20 seconds, even when I set my light panel to its highest setting.

I hope at least some of that helps,

Best of luck,

Jim 

As to the timing of the flats, I was using

[Carlos Milovic]

Yes. Long exposures in the flat are used to minimize the effects of the shutter, ussually slower than in dslrs.

[knro]

Ok, thanks a lot for explaining whats going on, now it finally makes sense. Now, since it is hard to get things EXACTLY as they are. Is it possible to correct for the offset in the flat using any PixInsight tool? I examined the flat vs. light under Krita and there is a vertical shift of about 11 pixels, so maybe correcting for that offset can make the images align and those artifacts will go away?

[pfile]

you can play around with pixelmath expressions of the form:

Code: [Select]

Pixel($T,xpos()+10,ypos()+10)

for instance, that shifts the image data up and to the left by 10 pixels. tell pixelmath to make a new image and then drag the triangle onto your flat. the result should have the same pixel dimensions as the flat.

you can fix the black areas like so:

Code: [Select]

iif($T==0,med(original_flat,$T)

that replaces the black areas with the median of the original frame. hopefully that keeps the statistics of the flat the same so when it's scaled ImageCalibration won't get confused.

in fact it could probably be done all in one go:

Code: [Select]

iif(Pixel($T,xpos()+10,ypos()+10)==0,med($T),Pixel($T,xpos()+10,ypos()+10))

rob