Scratches on my pictures?

[mads0100]

Hey guys,

   I have a tough one that I can't seem to figure out what I'm doing wrong.  I've tried just about every conglomeration of creating darks and I always end up with a similar defect on my images.  The lights themselves do not have the issue, only when I create the master dark and perform dark subtraction do I have this problem.  Essentially it looks like there are 'cold' pixels throughout my image.  I know this isn't the case since the light doesn't have this pattern of 'dots' on it.  What I can't seem to figure out is what is going on? So, without further ado, here is the images.

Third image, shows what I can get done with Cosmetic Correction.

Second image, shows what it looks like in a stacked image just doing a normal calibration and integration.

First Image, shows a screenshot of one frame uncalibrated.

I do my master calibration just like the tutorial says. http://pixinsight.com/tutorials/master-frames/en.html
 I've tried just about every combination and I always end up with these darn dots on everything!   Any help would be greatly appreciated.  So far, the guys at CN and on Astrobin Questions are at a loss.

[mads0100]

Guess I should add details as well.  I'm using 60 bias frames and 10 dark frames at the specific integration.

[Nocturnal]

Hi,

it looks like your darks don't quite match your lights. Due to differential flexure you now have little squigly lines. Check a few hot pixels and see how the light and dark values compare. Also make sure you are not subtracting both bias and dark. Dark includes your bias.

[mads0100]

Sander,

   Thanks you for your reply.  I am not subtracting my darks and biases.  I only stated it in case it came up in a discussion about dark optimization. However, I am utilizing the 'dark' created specifically for this integration. 

So you're telling me that my moonlight focuser is sagging when I take my darks versus when I'm taking my lights and this is causing the pixels to line up incorrectly? Can you explain the steps for checking the pixels?


Edit:
OK, measured the pixel value.  it decreases from the original uncalibrated image to the processed calibrated image from .0043 to match the value in both the dark and calibrated image of .0026. 

Thank you,
Chris

PS if it makes a difference, if I use the defectmap tool, this problem is non-existent. 

[sreilly]

Have you tried calibration without using the bias?

-Steve

[kolec]

Sander,
When I met with this problem I tried

1. Range selecion  ( You must see only  "black bad pixels" - big zoom in)
2. Aplly like mask  (You see only with black bad pixel)
3. Morhphology - erosion min
 You must try with every picture.
 

Kolec

[chris.bailey]

Chris

And I thought my Atik 383L had a lot of warm pixels!

Though dark subtraction gets rid of most of them I too find a few that slip through. I run cosmetic correction on the lights after calibration with Auto detect and sigma values of 2.8 warm and 3 cold. Using the Master Dark in CC only works if those pixels are warm in the master dark so I find Autodetect to be a better option. Then proceed to register and combine.

Chris

[mads0100]

Chris,

    Yea, no kidding? I guess I didn't really realize how many till you mentioned it!  Maybe I should up the cooling a tad?  These were all shot at -20C.  I have some room to grow. 

So, I can expect that dark calibration is not going to ever take care of this?  I already have a routine that works (defect map) that cleans up the images perfectly without the scratches.  The real question I can't figure out is what I'm doing to my darks to create the scratches in the first place. The other side of it is CC doesn't detect the cold pixels as such... auto detect fails to see those spots. 

If Sanders is right above; I'd love to know what I can do to fix this.  I just don't understand how flexure would play into it.

Oh, and I never specified equipment.  This is an St-10XME @ -20C.  Exposures are from 1200" darks/exposures.  Probably explains the warm pixels a bit.

Thanks again guys,

Chris

[Carlos Milovic]

Unfortunately, there are some hot pixels that do not follow a linear behaviour. So, the darks scalation routine fails to address them, and are either overcorrected, or undercorrected. This is probably the case. If you have your temperature well controled, and darks match light, try disabling the dark scalation and see what happens.
If not, DefectMap and CosmeticCorrection should make almost the same thing. The difference, from the user point of view, is that for DM you have to manually create the map of defective pixels. Also, I think that it is more flexible... Anyway, both should do the job done. DM can be batch applied to your calibrated frames with an ImageContainer, so it is not too troublesome.

[pfile]

hey chris, sander was only saying that you have under/overcorrected pixels and they appear to be traveling in a straight line in the integrated result because DF has caused your lights to shift just a bit on each exposure. he didn't mean that the lights somehow don't match the darks from a pixel x/y standpoint. it's impossible for that to happen as you always calibrate un-registered frames.

[kolec]

Hi

Maybe it´s sounds like  fiction, but :
I  have had the some problem. I had this scratches on my flats.  I try another software (CCD stack) for prepare flat and flat is clear

Maybe it will help, maybe

kolec

[kolec]

Hi
It´s not fiction. Now I try prepare flat,bias,dark with Batch preprocesing - on flat are scratches
First flat I prepare with ImageRegistr. and ImageCalibr





kolec

[mads0100]

hey chris, sander was only saying that you have under/overcorrected pixels and they appear to be traveling in a straight line in the integrated result because DF has caused your lights to shift just a bit on each exposure. he didn't mean that the lights somehow don't match the darks from a pixel x/y standpoint. it's impossible for that to happen as you always calibrate un-registered frames.

pfile, thanks for the response.  So what you're saying is that my camera has is warm pixels that are non-linear so that my darks are inadequate?  Can you explain why it a defect map works so well?  I guess I don't understand what is different between the two other than I'm manually adjusting the defect map (trial and error). 

Carlos,

    I tried changing dark scaling.  It had no effect on the development of the dots. I've tried just about every variation of creating darks that I could think of and nothing removed the over-application of the darks during calibration.  Can you explain what you mean about creating a process window?  I don't know how to do that?

Kolec,

   That is EXACTLY what is happening to mine as well .  I don't know what I'm doing wrong.  What kind of camera are you using?  What temperature are pictures at?  Any details you can provide me might provide a correlation.  My next goal is to try a colder CCD.  Hopefully that will make it easier to apply the darks automatically.  BTW Kolec, I am able to make my images work using a defect map. I posted in a thread on how I'm doing it if you want to try it out.  Use Median and Circular, it gives the best results.

[Geoff]

pfile, thanks for the response.  So what you're saying is that my camera has is warm pixels that are non-linear so that my darks are inadequate?  Can you explain why it a defect map works so well?  I guess I don't understand what is different between the two other than I'm manually adjusting the defect map (trial and error). 

The defect map and dark frame subtraction are fundamentally different.  Suppose a pixel in your light frame has x ADU of dark current and y ADU of signal for a total value of (x + y)ADU. In an ideal world, dark subtraction will remove the x ADU of dark current and you will be left with  y ADU of signal. On the other hand, the defect map simply removes all signal from the pixel and replaces it with some sort of average of the surrounding pixels. So dark subtraction should give you real signal, the defect map will give you guesstimated signal.
Geoff

[Carlos Milovic]

Can you explain what you mean about creating a process window?  I don't know how to do that?

I guess that you are talking about the ImageContainers I menctioned.
They are special objects in PI, sort of like the ProcessContainers. Basically, you make a group of views, previews or files, and you may apply a process to everyone of them, just by dropping the process instance to the ImageContainer icon with your defined set.
So, what you have to do is:
- Right-click on the background and select ImageContainer
- Create the group with all the frames you want to correct. Just press the "Add files" buttom, and select your files.
- Specify the output folder and the template name.
- Drag and drop the blue triangle on the bottom left corner to the background to create a new icon.
- Open the DefectMap process, and set its parameters (use the Mean operator, since there is a bug on the other ones).
- Drag and drop the triangle to the ProcessContainer icon.

There you go.

PD: Also remember, that DefectMap uses maps where the wrong pixels are black, and good pixels are white. You may use the HistogramTransform, Binarize, CurvesTransform and Invert for those purposes, usually working over the master flat, or the lights/flats, whatever suits you better.