Author Topic: Scratches on my pictures?  (Read 13575 times)

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #15 on: 2013 March 18 07:57:11 »
Geoff,

      So what can I do to mitigate this problem?  As I've seen quoted throughout as a PI mantra, if your data is bad, get better data and don't rely on software to fix it.  I'd much rather have good dark subtraction than deal with a defect map that is giving me a 'guesstimate'.  My only solution that I can think of, and it's a guess, is that I need to use more cooling to get rid of the warm pixels. 

Carlos,

    I apologize for the incorrect term.  That is exactly what I wanted to know, thank you for the explanation on ImageContainers.  I did not realize that I can adjust the binarized defect map with a histogram transformation.  I'll keep that in mind!

So, am I stuck with defect maps?  Is there any way to 'tone' down my darks so that they don't over-subtract?  It'd be nice to get the 'real' subtraction and just use the defect map to clean up the remaining hot pixels. 


Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #16 on: 2013 March 18 08:23:11 »
If you are 100% sure that the darks and lights have the same exposure time, and the same temperature, and you disabled the automatic dark scalation (and, you are using lots of bias to properly calibrate them), then I'm afraid that there is not much to do. Either DM or CosmeticCorrections are the answer. Also, try to use large dithering amount when you capture.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #17 on: 2013 March 18 08:42:59 »
Carlos,

   Yes, I'm sure the temperatures were correct.  In that case, I'll try dithering and see if that improves it at all. 

One more question, biases?  I don't understand?  Everyone has stated that I don't use biases on my darks since they are included in the data already?  Can you explain?

Thank you,

Chris

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #18 on: 2013 March 18 09:17:28 »
Biases are needed for the dark scalation routine.

Bias is a fixed pedestal, that accounts for the electronic signal (or "noise"). Since it is very close to zero signal, it is dominated by readout noise. So, to get the true underlying signal, we need a large amount of biases. It is true that they are included in darks, but you'll end with a much better electronic characterization using several dozens of bias, which are cheap to do. Having dozens of darks is impractical in most cases.
Darks, on the other hand, measures the thermal current, the signal that is generated only by thermical effects. It also suffers from noise, but it is more farther than the readout noise. Darks are not fixed. The dark current depends on the exposure time (mostly linear) and from the temperature (T^4). Also, there are some weird pixels that do not follow the linearity rule, and have either too large darks current, or too low; these are the hot and cold pixels. In many cases, there are defective pixels, where no data is recoverable.
If the dark current do not precisely match the lights, one may rescale them. But, to do so, we need to subtract the fixed pattern measured by the bias.
Also, bias are needed to apply the flat frames. They also include the bias factor, and since flats are multiplicative effects, we need to apply that calibration to get proper master flats.
To sumarize, the main equation for calibration is:  (lights - bias - k*(darks-bias))/(flats-bias)

Do not try to take shortcuts. We need the three calibration frames to get the most of our light frames.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #19 on: 2013 March 18 09:37:11 »
Carlos,

   Question then, do you think if I applied a master bias during calibration that it would possibly clear up the problem if I enabled dark scalation?

Just an idea,
Chris



Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #20 on: 2013 March 18 09:51:17 »
I would try, but I'm not optimistic. It seems to be a far worse problem that just the added pedestal. Anyway, results should have slightly higher SNR.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #21 on: 2013 March 18 11:26:59 »
Thanks Carlos, I appreciate all the assistance.  I'll give dithering a try and see if it helps.

Chris

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #22 on: 2013 April 07 16:46:21 »
Well guys,

     I'm back.  I tried a new program that allowed me to figure out dithering (Sequence Generator Pro).  Great program BTW; amazing features for a fairly low cost.  Now, back to my issues with calibration in PI.  Same problem as before.  Dithering didn't solve anything!  I was pretty disappointed of course; but I decided to try one more thing before I just gave up and went with defect maps forever.  I downloaded a new copy of DeepSkyStacker 3.3.2!  Sure enough, I did the exact same 'techniques' as I had done in PI.  Stacked my darks, biases, flats and lights exactly (as far as the software clicks are concerned)the same methods as I had been using in PI.

And, it worked.  No scratches, no hot/cold pixels.  Mind you, I did turn on automatic cold/hot pixel removal.  I also used automatic deblooming. But, I have integrated images that look fantastic when I bring them into PI. 

So, what is wrong with PI?  I know it's not my data now.

Chris

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #23 on: 2013 April 07 17:36:57 »
What happens if you integrate in PI your subs calibrated with DSS?
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #24 on: 2013 April 07 17:58:02 »
Good question :).  I'll check it out. 

Integration works fine.  The issue was always during calibration.  PI is over-applying my darks to a point that it's damaging the data around it.  For whatever reason, DSS is not doing that.  Attached is a screenshot of M97 that I registered in DSS and aligned/integrated in PI.  Of course, DSS has a horrible star detection routine so I couldn't even attempt to integrate it in DSS (too few stars found).  If you'd like to see the data I have so you can try and figure this out, I'd be more than willing to get it hosted on Astrobin (in FITS of course).

The other benefit of using DSS is that they have a viable deblooming routine.  I know, nothing you guys want anything to do with.

I'd really like to figure this out; I hate having to use other software when the whole point of getting PI was to use it for everything.

Chris


« Last Edit: 2013 April 07 18:43:34 by mads0100 »

Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #25 on: 2013 April 07 18:59:19 »
The only thing I can think of, is just disabling the dark scalation. Also, check that you are using the fit files in the correct orientation. It could be that darks do not match the lights, because they are upside down.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #26 on: 2013 April 07 19:11:19 »
Yea, dark scalation has always been off.  I'll see if the images are reversed, though I don't know how that would happen and why it wouldn't be reproducible across all the different packages.


Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #27 on: 2013 April 07 21:38:16 »
No, it is just how PI manages the FITS format... I change the behaviour of it to Top-left corner (see the FormatExplorer), and then, check the top-bottom option in the batch-preprocessing script.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com

Offline mads0100

  • PixInsight Addict
  • ***
  • Posts: 116
Re: Scratches on my pictures?
« Reply #28 on: 2013 April 08 05:53:42 »
OK, I'll give it a try tonight.  I'm not always using the batch pre-processing script.


Offline Carlos Milovic

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2172
  • Join the dark side... we have cookies
    • http://www.astrophoto.cl
Re: Scratches on my pictures?
« Reply #29 on: 2013 April 08 06:40:31 »
Also you may experiment with the integration settings. Change the rejection width, making it narrower, specially in the shadows.
Regards,

Carlos Milovic F.
--------------------------------
PixInsight Project Developer
http://www.pixinsight.com