PixInsight Forum

PixInsight => General => Topic started by: Cleon_Wells on 2010 October 01 12:04:38

Title: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 01 12:04:38
       If I Calibrate a single Flat.fit sub that was used in the integration process to make the Master Flat.fit file; should this new single Flat_c.fit  file have any signal level above the set output pedestal level in the CR2 to fit conversion?
Could I expect similar results if I Calibrated a single Bias or Dark .fit sub?
I use a Canon T1i/Hap mod on a 10” RC.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 12 11:40:06
I calibrated the DSLR Bias, Dark and the Flat fit bayer files, all three Master fit files had Calibrate checked. I then Debayered all the _c files, DeBayer Bias_c file has a signal level between 0.0002 and 0.0005 on all pixels, the Debayer Dark_c file  has all pixels set to 0.000, the Debayer Flat_c file has all pixels set to R=0.0101-G=0.0159, B=0.0149 so there's no image and note one of the hot pixels @ 2522x1573= R=0.0101, G=0.000, and B=0.0149. I unchecked the Master Flat Calibrate box and Calibrated the Flat again, the Debayered uncalibrated Flat looks like a flat with dust donuts and has R=0.0100, G=0.0152, B=0.0153, the hot pixel @ 2522x1573 has been moved to 2524x1573 and has R=0.0102, G=0.0414, and B=0.0116.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: georg.viehoever on 2010 October 12 13:15:10
Cleon,

       If I Calibrate a single Flat.fit sub that was used in the integration process to make the Master Flat.fit file; should this new single Flat_c.fit  file have any signal level above the set output pedestal level in the CR2 to fit conversion?

This is an interesting test. I would expect the calibrated single flat to have only a random noise signal (thermal+shot noise) with poisson distribution, no large scale gradients or patterns. You may be able to verify a poisson distribution of values by looking at the HistogramTransform GUI.

Georg
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 12 19:32:13
Georg, also when I calibrate the image subs, the dust donuts are removed if I have the Flat Cal box check, but the hot pixels are not removed. If I uncheck the Flat Cal box and Calibrate the image subs the hot pixels are removed but the dust donuts are not, this is similar to just using the Dark only. I am definitely chasing my tail here.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: georg.viehoever on 2010 October 13 01:21:55
Hi Cleon,
Georg, also when I calibrate the image subs, the dust donuts are removed if I have the Flat Cal box check, but the hot pixels are not removed. ...

I am not sure if hot pixels can be removed by the calibration process. In my case, they usually disappear when I integrate the registered lights, because in this case the hot pixels will not always be on the same location due to my less than perfect tracking (I have no autoguider), and the outlier rejection methods of ImageIntegration remove them.

Recently I have integrated a couple of pictures close to Polaris, and in this case hotpixels are at the same location for a number of frames. As a consequence, I see them in the integrated version, nicely forming arcs around Polaris  :'(  I am now experimenting with the DefectMap process to get rid of them. Another option is to use the method employed by DSS http://deepskystacker.free.fr/english/technical.htm#hotpixelsremoval .

 Georg
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 13 05:00:39
Georg, _c
Thanks for the link on calibration. Before Pix I only used CS3 for processing, I would remove  the hot pixels  from each sub with a Dark difference layer and then align each sub_c  layer comparing  each sub_c  against a reference sub layer by switching each sub_c layer from screen to difference. I have a lot to learn when it comes to image processing.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 27 05:07:06
I've been working with Calibration and stacking of 17-90seo_iso800 subs of M78 and can't figure out how to use my CR2 image files with the .fit Cal files, the CR2_c image files have no image data.

Also  I notice some structure change in my M78 image compared to images I see on the web, see attachment; could this be from the DSLR subs?

Also it's unfortunate that Randall had so much trouble with the Pix basics because he had the wrong BatchFormatConversion Version..... I had a similar problem and was able to have Vicent look over my shoulder at the ADLER workshop and point out the version problem.

Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: georg.viehoever on 2010 October 27 08:35:55
I also don't see the structure on Google Sky, e.g. http://www.google.de/sky/#latitude=-0.09613032599286975&longitude=-93.42979431152344&zoom=11&Spitzer=0.00&ChandraXO=0.00&Galex=0.00&IRAS=0.00&WMAP=0.00&Cassini=0.00&slide=1&mI=-1&oI=-1

But it appears to be here, McNeil's Nebula: http://www.oneilladvertising.com/Resources/M78rev5.jpg

Anyway, I think it is unlikely to come from calibration. Could be a reflection, or its really McNeil's...

Georg
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 27 08:51:14
Georg, yes it is McNeil's Neb, it's interesting that the other M78 web images I saw only faintly show some structure in this area of M78.
Thanks for the info.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: georg.viehoever on 2010 October 27 10:57:16
Georg, yes it is McNeil's Neb, it's interesting that the other M78 web images I saw only faintly show some structure in this area of M78.
Thanks for the info.
Cleon

Well, you are only 6 years too late for the discovery of this interesting object.  :'( http://en.wikipedia.org/wiki/McNeil's_Nebula

Georg
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 27 12:11:30
Georg, oh well, maybe I can still discover how to Calibrate CR2 14bit files with B_D_F .fit files. LOL
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 October 27 14:24:25
I've just begun this PixInsight journey, and recently succeeded with calibrating my modified Canon 1000D cr2 files, with the .fit master farmes.  One thing I discovered along the way; If by chance your system or PixInsight locks up, and then you get out and restart... The settings for DSLR in format explorer reset to default. This will cause you many headaches if you were not aware of this little tid bit. I've experienced the same issue that you describe "no data in the RAW files", and I believe it was caused by this.
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 27 19:29:54
Zerro1, Let me frist correct my statement (Also it's unfortunate that Randall had so much trouble with the Pix basics because he had the wrong BatchFormatConversion Version..... I had a similar problem and was able to have Vicent look over my shoulder at the ADLER workshop and point out the version problem.)  I meant to say "the wrong BatchDebayer Version..."
Now lets go back into the ImageCalibration woods , so to speak. I also have just begun the Pix journey and can't get the camera CR2 files to calibrate. My process starts by BatchFormatConversion of my Bias, Darks and Flat CR2 files to .fit with the DSLR_Raw preference set with only two boxes checked, "create Raw Bayer image()"  and "No black point correction". Then I use Imageintegration on the B, D, F .fit files to create my master Cal files. Then I add my CR2 camera files into the ImageCalibration Target Frame list and cal, this does not work; what am I doing wrong?
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 October 28 00:07:59
are you calibrating the flat files with the master bias and master dark before you integrate for the master flat?

also do take note that when integrating the calibrated flat's into the master flat, the settings are different for the flat integration than the settings for the dark and bias frame integration
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 28 10:23:19
Zerro1, I've tried Cal_ing the flat subs with both B and D Masters and also just B with no luck. I used Vicent's setting to integrate the Master Flat. I tried Cal_ing the CR2 files using the Master Bias only I tried changing  the BatchFormatConvert Raw Format preference and made new Master Bias files made with bias subs that where BatchFormatConverted with the Brightness set to 1.0 ( I normally use a Brightness setting of 4.0 for my 14bit CR2 files), I tried the Raw Bayer CFA conversion settings, still no luck. I'm definitely lost in the  CR2 file Calibration woods.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 October 28 22:50:55
Quote
Zerro1, I've tried Cal_ing the flat subs with both B and D Masters and also just B with no luck.
You're saying here that the flat calibration did not complete successfully? or it appears that it didn't based on the failure at the calibration of your light frames?

as far as the DSLR RAW pref settings in "format explorer" I have only used it set at 1.000000 for all three options.

Did you try just calibrating the lights (image subs) with only the master Dark?

You're making sure that the "calibrate" under Master Flat is un-checked"

Now here is a question I have for anyone who might know the answer; why, in Image Integration, would you have the default setting to add calibration to the master flat "checked" instead of "un-checked"?

I'm just sayin..
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: georg.viehoever on 2010 October 29 01:27:53
...
You're making sure that the "calibrate" under Master Flat is un-checked"

Now here is a question I have for anyone who might know the answer; why, in Image Integration, would you have the default setting to add calibration to the master flat "checked" instead of "un-checked"?
...

See http://pixinsight.com/forum/index.php?topic=2371.msg16345#msg16345
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 October 29 08:44:09
Cleon
I didn't see earlier in the thread where you stated that you did do a dark only cal.

(qualifier) Ok I know that the "accepted method" of capturing flats is the camera can't be moved , defocussed Blah Blah Blah...!

But lets try to get this sorted out so you can make this work.  I have a T1i/500D, somewhere I have some subs from the T1i that I'll dig up and also give this a go...

put your camera back on your scope as close to the position imaged at as you possibly can! I want you to re-shoot 10 flats, 10 Dark_flats, and then get 25 bias frames.. ALL at ISO 100! I know, it will not be the optimum Master Flat, but 99% of the time the dust that you see is on your sensor, and unless you're really disturbing it, it's not moving much.

Take these files and just focus on creating a master_flat with them. I know that PixInsight has dark scaling but let's eliminate that for now and just try to get past this roadblock.  

This image of the Rosette was just completed this morning using exactly that method but was captured with a modified 1000D . Darks, Bias and lights were all ISO1600 and the Master Flat was created using ISO 100 flats, bias and darks.

EDIT: 

So I completed a couple calibration and integration procedures with Data from my T1i. after a couple of errors on "My Part" I got through it fine.  It is do-able with the tutorial guides.
You might try a couple variations with the windsorized, sigma, or averaged clipping in the final integration if your result comes out a little undesirable.

Robert


________________________________________________________________________________________

Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 October 30 07:32:04
Zerro1, and Georg, I'm down in Nashville Tn. this weekend, visiting the grandchildren, will be back on Monday and hopefully figure this out.
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 November 02 20:19:34
In my last test, trying to Cal 14bit CR2 files from my T1i Canon, I used one CR2 Bias sub, one CR2 Dark sub and one CR2 Flat sub for the Master Calibration files. With all the files being  CR2 files I get a _c image file with an image. If I convert the Bias,Dark an Flat CR2 files to  fit files with all three levels set to 1.000 my _c image file has all the pixels set to 0.000: what am I missing in the raw CR2 file Calibration procedure?
Cleon
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 November 03 00:40:19

Cleon

when you set the parameters in the format explorer, leave them alone. what ever is causing this is not from those settings. I have a T1i, I tried the whole process from start to finish. It does work.

please read through it first.

Lets take this one step at a time,  Until you successfully complete the creation of the calibration frames, nothing is going to work.  you will need 10 flats, 10 dark flats, and 20 bias frames, all with the same ISO speed. (I chose ISO 100 because I'm trying to keep the noise low).

we are only going to calibrate and create you Master_Flat_Frame  with these files. To do this, you will need to not only follow the directions in the tutorial, you also need to make absolutely certain that all check boxes and menu selections are identical to the pictures.
 

when I ran BatchFormatConversion for each set "flats", "Dark_flats", "Bias",  to make sure I didn't have any conflicts, I saved them all as 32-bit IEEE 754 floating point.

 BatchFormatConvert all calibration files
 integrate Flat_Darks and Flat_Bias
   a:settings: Integration
          Combination = average
               Normalization = No Normalization
              Weights = Don't care (all weights =1)
              Generate Integrated image = checked
          Buffer = 16
          Use file cache = checked
   b: settings: Pixel Rejection 1
          Rejection algorithm = Windsorized Sigma Clipping
          Normalization = No Normalization
          Generate rejection maps = checked
          Clip low pixels = checked
          Clip high pixels = checked
          Clip low range = checked
          Clip high range = checked
    c: settings: Pixel Rejection 2
          Sigma low = 4.000
          Sigma high = 3.000

          Range low = 0.000000
          Range high = 0.980000
 First integrate the darks using the above settings. then repeat the process to integrate the    bias files.

at the end of each pass through the integration process, you will be presented with three files.
You want to only save the one that is designated "integration". save the one from the dark frames as Flat_Dark. after you integrate the bias frames, save that "integration" frame as  Flat_Bias.

I'm going to provide the walkthrough for Calibration and Integration of the Master_Flat in a second response to isolate it.
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 November 03 01:47:53


After completing this part you will stiill need to create a MASTER_DARK frame for the calibration of your Light Frames The Master Dark subs should be the same ISO as your "Image subs" and should be equal to or  longer than your image subs to allow for "Dark Scaling".  Master Bias must be created using more subs than are used in the creation of the Master_Dark....

I'm going to bed now
_____________________________________________________________________________________


Creating the Master_Flat is a two process task.
First you must calibrate the Flat subs that were converted to .fit format. To Do This:

open the ImageCalibration process.

add the 10 converted flat subs.

you must select an Output Directory

the default settings should be

Output extension =.fit
prefix = cal_
sample format = 32-bit floating point
Overwrite existing files = un-checked
On error = continue

Overscan = un-checked

master bias = checked
this is where you need to select that "Bias_flat" file that you created

calibrate = checked

Master Dark = checked

this is where you need to select the Dark_Flat file that you created

Calibrate = checked

Optimize = checked

Optimization window (px) = 1024

CFA pattern detection = Detect CFA

Master Flat = YOU MUST Un-Check this box

Now click the round button and calibrate these flats.




PART 2

open the Image Integration Process

Input Images..

add the Calibrated Flat frames

Under "Image Integration"

Combination = Average

Normalization = Multiplicative

Weights  = Don't care (all weights =1)
weight keyword "blank"

Generate Integrated Image = checked

Generate a 64 bit result = "un-checked"

Evaluate noise = "un-checked"

close previous image = "un-checked"

Buffer size (MB)  = 16

Use File Cache = checked

PIXEL REJECTION (1)


 Rejection algorithm = Percentile Clipping

          Normalization = Equalize Fluxes

          Generate rejection maps = checked
          Clip low pixels = checked
          Clip high pixels = checked
          Clip low range = checked
          Clip high range = checked



PIXEL REJECTION (2)


         Percentile low = 0.010
         Percentile High = 0.010



          Range low = 0.000000
          Range high = 0.980000


Click the round button.

The "integration" File that you get from this process will be your Master_Flat.
Save it with that name.



Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: Cleon_Wells on 2010 November 03 19:29:40
Zerro1, I kept all the Bias,Dark and Flat subs at the same iso and for the first time I could Calibrate my CR2 science subs. Last night I took more subs of M33 at iso-1600_90-sec exposure, I used ios-1600 B,D,F sub files to create the Master Cal files for Calibration of these iso-1600 CR2 subs.
       You mentioned you were using iso-100 B,D,F subs to create low noise Master Calibration files; so the Master Calibration files don't have to be the same iso as the science subs?
Cleon
PS. thanks for getting me  past this basic CR2 Cal problem, I hope the CR2 Cal subs will give me better results than my converted .fit subs produce.
Title: Re: Enigma= Master Bais_Dark_Flat files and my DSLR files
Post by: zerro1 on 2010 November 03 19:43:09
That low ISO usage was ONLY to create the "Master_Flat Frame" not the Master_bias and Master_dark.

I completely divorced the Master_flat creation from the creation of the Master_Dark and Master_bias. It makes for extra work but, but we do what we can for every improvement.. :)

I'm glad I was able to help !