Author Topic: DSLR_RAW work flow - overview  (Read 21198 times)

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1
« Reply #15 on: 2010 December 02 19:43:40 »
Nice work pfile :) Would you like me to provide links from the main thread?

pfile

  • PixInsight Guru
  • ****
  • Posts: 990
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1
« Reply #16 on: 2010 December 02 22:38:24 »
Nice work pfile :) Would you like me to provide links from the main thread?

sure, yeah. BTW i didnt upload to youtube because i thought maybe they would be too small to see anything, even with youtube's HD support... it might be worth a try, though.

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #17 on: 2010 December 25 22:06:24 »
My website is defunct, for the time being. The server has crashed and the hard drives are unrecoverable, according to an email from the Webmaster. That means that I've had to remove the links to the Video demo's from the main post. Fortunately, pfile's video links are working. I won't be replacing the links. Nice Christmas present.

pfile

  • PixInsight Guru
  • ****
  • Posts: 990
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #18 on: 2010 December 26 22:24:33 »
sorry to hear that, it's terrible. seems amazing in this day and age that a webhosting company would not have backups...

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #19 on: 2010 December 27 04:58:46 »
I found it hard to believe and was about to point out that one of the attractions to the hosting company is regular back ups. This morning I received an email saying that they were looking for ways to recover the data, this was followed by another stating that they were using back ups ???

I think it's case of state the worse and things can only get better from there. Don't know when it'll be up and running.

Enzo De Bernardini

  • PixInsight Addict
  • ***
  • Posts: 213
  • Resistance is futile.
    • View Profile
    • Astronomía Sur
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #20 on: 2011 February 01 21:55:16 »
Very good summary, be very useful. Thanks!

On DSLR, is it equally valid calibrate the master dark checking 'calibrate' on dark frames section when flat and light frames will be calibrated? (like Vicent tutorial flats calibration). What's the best procedure?

Greetings,

Enzo.

(for spanish readers, I have similar tutorial here)
Orion SkyviewPro 8" F/5
Orion Shortube 80 mm F/5
Canon EOS 300D
http://astrosurf.com/astronosur

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #21 on: 2011 February 02 00:25:39 »
It depends on whether the flat frames will be dark calibrated, which depends on the flat exposure time. I believe this is consistent with Vicent's advice above. I have a master flat that must be dark calibrated when calibrating lights - I guess because I didn't dark calibrate the flats to begin with.

zvrastil

  • PixInsight Addict
  • ***
  • Posts: 185
    • View Profile
    • Astrophotography
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #22 on: 2011 February 02 08:54:41 »
Hello,

thanks for the great summary. Still, I would like to clarify one thing. From all the threads, it seems like conversion from RAW to FITS is an optional step. However, ImageIntegration does not accept anything but FITS. Am I correct that it is necessary to convert RAW to FITS before creating master bias? For other steps, ImageIntegration is always preceded with ImageCalibration, which accepts RAWs and generates FITS, so it's not a problem.

thanks, Zbynek

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #23 on: 2011 February 02 09:48:49 »
 :)Yes, bias frames are converted to fit. Vicent indicates, in the process icon comments, summarised above, that converting all RAW bias, dark and flat frames to fit is the first step.

I have been meaning to ask the question, whether it is OK to calibrate RAW dark and flat frames, or whether it is essential to convert to fit in the first place. I would think not - maybe it's worth asking the question.

martin_magnan

  • Member
  • *
  • Posts: 67
    • View Profile
    • Messager Céleste
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #24 on: 2011 February 02 14:34:32 »
Hi

I'm not converting .cr2 files to .fits because I calibrate my dark and flat with a master_bias. Since the CI routine accept raw files, I skip the conversion. Does it makes a difference ????? I dont know. The conversion is necessary only for Bias frames but I use the same Master_Bias for many months so the conversion is done only once.

My Master_Dark and Master_Flat are already calibrated so I uncheck the calibrate box (for dark and flat ) when I calibrate my light frames.

Martin
AT8RC / Takahashi EM-200 Temma II / QSI583wsg - Astrodon LRGB 2 filters / QHY5 Guider /Canon 20mm F2.8; 50mm F1.4; 135mm F2.0L / Celestron 17X70 / SPC900NC

http://www.messagerceleste.com

Niall Saunders

  • PixInsight Jedi
  • *****
  • Posts: 1056
  • We have cookies? Where ?
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #25 on: 2011 February 02 14:57:08 »
The other important thing to bear in mind is that the steps so eloquently described in this process are also equally as valid for those using 'One-Shot'Colour' CCD imagers (except for any steps involving conversion from RAW to FITS, obviously)

However, I have been struggling with a SIGNIFICANT issue when I have been procsessing my images using this workflow - and the issue is specifically realted to Flats.

Bearing in mind that Flat Calibration involves 'dividing' your individual Lights by your Master(Calibrated)Flat, you have to consider the process in somewhat greater detail when working (as we are at this stage) with images that have not yet been deBayered. In a 'standard' (monochrome) process, we set about acquiring Flats with a typical maximum ADU level of around 32,000, and then we calibrate these with a MasterFlatDark image.

Importantly here is the fact that the MINIMUM ADU value for the Master(Calibrated)Flat should NOT be (anywhere near) 'zero' - otherwise the 'division process' is going to start creating all sorts of horrible 'infinity' values (which, fortunately can be 'trapped' by PI in any case - but still remain undesirable in the first place). And this situation is undesirable for either OSC or Mono imagers, and for DSLR users as well.

However, if you have managed to avoid the 'zero ADU' problem, and have a Master(Calibrated)Flat (for a mono CCD, for the moment) with values now ranging (say) from 1000 to 32000 - all will be well. PI will perform the division quite happily, and will even ReScale the output ADU values back into the [0.0, 1.0] internal working range - all automagically. Which is great.

Now we come to the 'OSC problem' (also the 'DSLR problem') - as I see it

Unless you have the luxury of illuminating your OTA objective with a light source that causes EVERY SINGLE PIXEL on the CCD - irrespective of its CFA filter colour - to respond 'linearly' to the incoming light spectrum, then you ARE going to have quite a strange resultant Master(Calibrated)Flat to work with. Basically, instead of the 'single peak' visible on the Histo panel (when examining a 'mono' Flat frame), you will (most likely) have FOUR esily distinguishable 'peaks' visible on the HT graph when you look at a Flat from an OSC camera.

And I am struggling to understand how, or even 'if', this has a knock-on effect when this type of Flat is used to calibrate an OSC 'Light' frame. I could see that, if each of the four 'arrays' of pixels could be 'flat-calibrated' individually, then the 're-scaling' functionality of PI would effectively eliminate the problem. But, because this obviously does NOT happen (as I understand), then the resultant 're-scale' operations affect each of the four pixel sets (one per colour in the CFA) DIFFERENTLY - leaving the final calibrated Light with quite horrendous colour bias.

Well, this colour bias problem has been what I have experienced anyway.

I have created a 'workaround', and it isn't THAT difficult to implement (in that it only has to be applied to the initial MasterFlat), but it does require around ten PixelMath applications to 'normalise' the four pixel-sets of the CFA (in fact, the process is easily applied by re-using a ProcessContainer that I devised, and now store on disk - it doesn't even require a PJSR - the only 'downside' being that eight 'interim' images that are created need to subsequently be deleted, manually).

What my process does is to 'split' the original MasterFlat into four images - each containing the pixel data from ONE of the colour filters of the CFA. Importantly, these images remain the SAME SIZE as the original MasterFlat, only they contain '0' for all pixel locations NOT assopiated with this colour of the CFA. I then apply (either) a SCALING or an OFFSET process to this 'extracted' image, such that the 'brightest' ADU value is moved up the luminance scale (to '1.0') - either by 'addition' for the OFFSET process, or by 'multiplication' for the SCALED process. Once the four 'extracted and normalised' images have been created, these are then simply 'added' back together (easily done, because all 'non-associated' pixel values were set to '0' during the CFA 'extraction' stage) to recreate a 'normalisedMasterFlat'.

So, in the normalisedMasterFlat, the 'brightest' pixel in each CFA sub-set is only ever '1', and the 'dimmest' will always be 'non-zero'. And, importantly, the four CFA pixel-sets are each 'equally as bright' as each other - irrespective of the spectral content of the light source used to create the flat in the first place.

What I now ask is that 'minds immeasurably superior' to mine have a think about the problem, and about the solution that I am currently using (for better or worse), and decide whether the problem really does exist in the first place, and whther my solution is effective at addreesing the problem, WITHOUT introducing subsequent additional problems of it's own making !
« Last Edit: 2011 February 02 21:11:47 by Niall Saunders »
Cheers,

Niall Saunders
Clinterty Observatories
Aberdeen, SCOTLAND

LX90 Classic 8"SCT - Homebrew Wedge and Pier
Moonfish ED80 APO Piggybacked - Homebrew 3D Balance
QHY10 CCD + TS-OAG
Meade DSI IC AllSkyCam, IIC, IIPro+Outback TEC
Atik Filterwheel + Meade RGB Filters & Astronomik 13nm Ha

Niall Saunders

  • PixInsight Jedi
  • *****
  • Posts: 1056
  • We have cookies? Where ?
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #26 on: 2011 February 03 15:21:20 »
I spent several pleasant hours last night trying to collate my thoughts on this, and processed an identical set of Lights using three different methods of achieving a MasterFlat.

There were HUGE and SIGNIFICANT differences - with the best method being to use a 'normalised' Flat (by (one of) the method(s) I described earlier).

It looks as if I will have to put together a more detailed reponse (probably as a new thread), with pictures and screenshots to help explain.

Hopefully someone will rise to the challenge of reviewing my findings!
Cheers,

Niall Saunders
Clinterty Observatories
Aberdeen, SCOTLAND

LX90 Classic 8"SCT - Homebrew Wedge and Pier
Moonfish ED80 APO Piggybacked - Homebrew 3D Balance
QHY10 CCD + TS-OAG
Meade DSI IC AllSkyCam, IIC, IIPro+Outback TEC
Atik Filterwheel + Meade RGB Filters & Astronomik 13nm Ha

pfile

  • PixInsight Guru
  • ****
  • Posts: 990
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #27 on: 2011 April 19 17:33:20 »
i can upload those mp4s to youtube, that should fix everything... i think mac users might have problems with the .avi files.

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9
« Reply #28 on: 2011 April 30 06:50:18 »
Thanks, that would be good.

astropixel

  • PixInsight Old Hand
  • ****
  • Posts: 462
    • View Profile
Re: DSLR_RAW Workflow PI 1.6.1 - 1.6.9 - 1.7
« Reply #29 on: 2011 June 03 00:02:34 »
Had a look over 1.7. It's very nice. Unless other users can provided feedback to the contrary, the DSLR_RAW workflow guide applies to 1.7 also.