Author Topic: Flat doubt while calibrating with PI  (Read 6624 times)

Offline Astrocava

  • PixInsight Addict
  • ***
  • Posts: 180
    • Astrocava.com
Flat doubt while calibrating with PI
« on: 2010 November 09 01:41:48 »
Hi all!

Attached is the integrated image of IC405 fully calibrated with PI and DSS. I used Vicent's tutorial.

18 light frames
37 flat frames
70 BIAS frames (I have 99 but PI ran out of memory integrating them)
42 dark frames

It seems PI overcompensated the image using the flats. Check out the "hair" on the left of the image and the global glow around the center.

Any thoughts?

Sergio
Moonfish ED80 over a Meade LX200GPS 8"

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Flat doubt while calibrating with PI
« Reply #1 on: 2010 November 09 02:20:05 »
This is less extreme but similar to what I saw in my experiments, see http://pixinsight.com/forum/index.php?topic=2371.0.  See http://pixinsight.com/forum/index.php?topic=2371.msg15857#msg15857 for what helped in my case.

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Astrocava

  • PixInsight Addict
  • ***
  • Posts: 180
    • Astrocava.com
Re: Flat doubt while calibrating with PI
« Reply #2 on: 2010 November 09 04:29:28 »
Worked for me too  ;)

Thanks, Georg.
« Last Edit: 2010 November 09 07:32:10 by cavamen »
Moonfish ED80 over a Meade LX200GPS 8"

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Flat doubt while calibrating with PI
« Reply #3 on: 2010 November 09 07:57:58 »
Hola Sergio

So, problem solved, then? Could you please post the same comparison with the image calibrated with PI?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Astrocava

  • PixInsight Addict
  • ***
  • Posts: 180
    • Astrocava.com
Re: Flat doubt while calibrating with PI
« Reply #4 on: 2010 November 09 09:10:07 »
Here is the comparison. The results are very similar, but check the color of "Canon's mark" in both cases.

I don't know if for DSLR cameras calibrating with PI instead DSS worth it. I spent almost 10-12h to make the calibration, registration and integration. Perhaps was longer because was my first time. DSS is a single button and autonomous process.

Have fun!

Sergio
Moonfish ED80 over a Meade LX200GPS 8"

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Flat doubt while calibrating with PI
« Reply #5 on: 2010 November 09 10:46:49 »
...
I don't know if for DSLR cameras calibrating with PI instead DSS worth it. I spent almost 10-12h to make the calibration, registration and integration. Perhaps was longer because was my first time. DSS is a single button and autonomous process.
...
I am also not yet convinced, and would wish for a "fewer buttons" solution in PI. The results I have obtained with PI so far are also not better than those with DSS.

Juan, what would happen to your optimization procedures if image noise in Canon DSLR images was not proportional to exposure time? Would your optimization routines fail? Also we already have a couple of reports that the wavelet based noise estimation sometimes fails for DSLRs. Would this kill optimization? Just a gut feeling that this may be the cause for the non-improvement compared to DSS despite of PIs sophistated procedures. I have not yet had the time to investigate...

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Flat doubt while calibrating with PI
« Reply #6 on: 2010 November 09 12:33:37 »

Quote
Juan, what would happen to your optimization procedures if image noise in Canon DSLR images was not proportional to exposure time? Would your optimization routines fail?

Not at all. The optimization algorithm is absolute; it does not depend on any physical property of the sensor, neither on any acquisition conditions. It will always find the optimum scaling factor that minimizes small-scale noise induced by dark subtraction in the calibrated image

With our current dark scaling routine, nonlinearity of the sensor for very bright pixels causes undercorrection of hot pixels for some cameras. This happens because the optimization algorithm finds the best dark scaling factor for noise minimization, while optimal hot pixel removal involves subtracting darks with higher scaling factors close to one. We have designed a variant of the optimization algorithm (multipoint dark scaling) that should fix this problem very well. Hopefully It will be released with the next version of PixInsight.

Quote
Also we already have a couple of reports that the wavelet based noise estimation sometimes fails for DSLRs.

Those reports refer to a few odd issues with the MRS noise evaluation algorithm implemented in the ImageIntegration tool (multiresolution support noise evaluation). MRS noise evaluation is extremely accurate but has the drawback that it can fail in some cases. The evaluation algorithm used in the ImageCalibration tool (k-sigma iterative noise thresholding on the first wavelet layer) is much simpler and 100% robust. It cannot fail unless the image has absolutely no noise, which obviously cannot happen when we are subtracting a dark frame. Even if you try to fool the algorithm providing noise-free images, the optimization will converge "graciously" toward a zero dark scaling factor :)

Quote
Just a gut feeling that this may be the cause for the non-improvement compared to DSS despite of PIs sophistated procedures. I have not yet had the time to investigate...

Well, one possibility can be that DSS is carrying out a very nice job :) On one hand, we are speaking of "exquisite" algorithms and features, whose actual repercussions on final images are difficult to evaluate. On the other hand, with the exception of automatic dark scaling there is not too much room for improvement in the image calibration stage, since the procedures are rather simple and calibrating is just subtracting and multiplying things after all.

Where PixInsight really shines --in my opinion-- is in image registration and integration, where we provide superior algorithms and implementations. Both tools will receive important improvements in the short term, which will make them even more powerful and versatile. Whether these routines can have a clearly measurable result depends basically on the characteristics and quality of the raw data, in my opinion.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Astrocava

  • PixInsight Addict
  • ***
  • Posts: 180
    • Astrocava.com
Re: Flat doubt while calibrating with PI
« Reply #7 on: 2010 November 09 12:38:09 »

Where PixInsight really shines --in my opinion-- is in image registration and integration, where we provide superior algorithms and implementations. Both tools will receive important improvements in the short term, which will make them even more powerful and versatile. Whether these routines can have a clearly measurable result depends basically on the characteristics and quality of the raw data, in my opinion.


I was thinking to tell DSS to do the calibrated frames and then register and integrate them in PI. We'll have the best of each one.
Moonfish ED80 over a Meade LX200GPS 8"

Offline RBA

  • PixInsight Guru
  • ****
  • Posts: 511
    • DeepSkyColors
Re: Flat doubt while calibrating with PI
« Reply #8 on: 2010 November 09 20:45:10 »
IMHO if people new to doing this process constantly run into the same - or very similar - problem, and not only that, it often takes them an "unacceptable" amount of hours to get it to work, there is a problem.

Yes, calibration is an area where many people could have a "better education", and people with "better education" probably won't "suffer" this learning process. And one of the things PI has been good at is at getting people to learn things that otherwise would be plug-and-play without the need for them to do any further research. But when everyone hits the same stone and spends 10 hours to resolve the very same problem seems to me it's a red light that should be changed, at least, to amber.

I'm not trying to be negative, quite the opposite.



Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
Re: Flat doubt while calibrating with PI
« Reply #9 on: 2010 November 09 21:28:00 »
I have been a strong advocate of making the cal set up simpler to use.
 I generally do some steps in other programs but I love the integration module.

Max

Offline Harry page

  • PTeam Member
  • PixInsight Jedi Knight
  • *****
  • Posts: 1458
    • http://www.harrysastroshed.com
Re: Flat doubt while calibrating with PI
« Reply #10 on: 2010 November 09 23:07:01 »
Hi


Still with you on this ,  >:D


Harry
Harry Page

Offline zerro1

  • Newcomer
  • Posts: 20
Re: Flat doubt while calibrating with PI
« Reply #11 on: 2010 November 10 02:16:41 »
I used Vincents tutorials in my quest to learn how to apply PixInsight to my DSLR imaging. I did however, use flat darks, and bias frames for the flats. The reason for this was because I used ISO 100 flats, while my light frames were ISO1600. I can also say without any doubt, my results are vastly different with PixInsight over DSS.

Here is the result I got.
« Last Edit: 2010 November 10 02:25:36 by zerro1 »

Offline mmirot

  • PixInsight Padawan
  • ****
  • Posts: 881
Re: Flat doubt while calibrating with PI
« Reply #12 on: 2010 November 10 09:25:48 »
I kind of doubt the difference is based on calibration alone. 
I don't use DSS so I can not say too much.
My results from MaxIM DL are very similar.

In theory they should be identical if are you not scaling darks.  Perhaps you could find difference on how flats are normalized

Offline zerro1

  • Newcomer
  • Posts: 20
Re: Flat doubt while calibrating with PI
« Reply #13 on: 2010 November 10 11:41:38 »
In reality....Probably is not due to calibration alone. The results speak for themselves. I just simply got tired of having little or no control over what DSS spit out. PixInsight gives me the latitude to make adjustments that improve what comes out in the final stack. DSS gives you a vary narrow set of options to control how it handles what you give to it.

I can make an "honest Master_flat frame" with an ISO setting at the opposite end of the spectrum in PixInsight. Can't do that in DSS. DSS does not give you that control. Sure it would be wonderful to have the magic button; drop everything in the asket and let the program sort it out. But I've already been there and done that with DSS.

I started looking at my images at 100% view, asking "what can I do to improve my results?" PixInsight has shown me what "I" can do to improve my results.

Offline Jack Harvey

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 975
    • PegasusAstronomy.com & Starshadows.com
Re: Flat doubt while calibrating with PI
« Reply #14 on: 2010 November 10 15:26:56 »
As I said previously.......
   
Master Flat Creation
« on: 2010 September 20 19:07:18 »

This process, although very exacting, is tedious enough to discourage the creation of Master Flats in PixInsight.  Master flats must be created for each image so this is a process used frequently, unlike creation of Master Bias and Darks for a library.  The use of two separate tools to perform this process is extremely cumbersome.  I like the Flat Calibration routine and if that would flow into the production and saving of the integrated and labeled Master Flat all would be well.  But to go back and find the calibrated flats and run through a integration and then label and save the integrated master in the appropriate folder is too much.  This process, should with a single click, deposit a integrated and named master flat into the same file from which the raw flats were obtained IMO.

I am working on a couple of images today and by the time I have created the Master Flats for 5 filters for two images I am to burnt to continue processing the raw image data:-(

Having said all that, I have done a side by side visual comparison of the calibrated light frames and the ones calibrated via PI are certainly preferrable to my eye over the same frames calibrated using another program. Lets just make it more user friendly<G>.
Jack Harvey, PTeam Member
Team Leader, SSRO/PROMPT Imaging Team, CTIO