When to debayer?

[Diane Miller]

I'm finding some conflicting information.  I'm using a DSLR and have PI set to Pure Raw.  After calibrating lights, is it best to debayer before of after cosmetic correction?   

Thanks!

[Warhen]

Debayer after CosmeticCorrection. If using BPP, the script will of course follow this sequence after calibration and before alignment.

[Diane Miller]

Thanks!  I should probably go to the batch script , but for starters I'm doing things the hard way to try to get things figured out.

When I open CosmeticCorrection and try to load the calibrated lights, all the files are grayed out (as are all .xisf files in any folder).  I tried doing the debayer right after calibrating the lights and those files will load in CosmeticCorrection -- they are not grayed out.

[Diane Miller]

I'm becoming more confused.  If I run the BPP script I have no information to put in the cosmetic correction section, but it's going to debayer after calibration of lights and then register them and integrate to make a single master.  When and how do I do cosmetic correction in that case?

And what about the warning that integration of lights won't be optimal?

From what I can understand of Harry's video, he first uses BPP with calibrate only checked, then exits to do something else and comes back....

All is all, it seems easier to me to do things in separate steps, following a careful cribsheet.

But I'm back to questioning the right sequence.  Here's what I'm doing and where the hitch is:

using a DSLR, setting PI to Pure RAW
not using darks for now, due to concerns I have seen raised
create a master bias (to be used for several months, as long as I'm shooting with the same camera at the same ISO)
create a superbias -- but its use is being questioned for now
calibrate flats with master bias (or superbias)
integrate calibrated flats to a master
calibrate lights with master flat
run cosmetic correction on calibrated lights -- BUT all .xisf files are grayed out in the cc dialog
debayer cc lights should be done here
select good lights (cosmetically corrected and debayered) with SubframeSelector
register selected lights
integrate to a single image
crop
correct gradients and vignettes
background neutralization
color calibration
then a lot more I haven't gotten to yet

Any advice on the path through the woods will be greatly appreciated!  Thanks!

[dzso.bacsi]

Hello Diane,

If you follow step by step this http://www.lightvortexastronomy.com/2015/07/tutorial-pixinsight-pre-processing.html tutorial, I think it will clear the confusion a lot. It is better than video, I actually followed it step by step, sentence by sentence with my own lights, darks, flats, biases and it made the whole process much clearer. It helped to "understand" instead of following something which was done in a video. You can do it in your on pace, step back, do it again and so on. Actually I do not use the BPP since I learned pre-processing from Kayron.

Hope it will help
Cheers 

[Diane Miller]

Thanks!   I've had it cited in my notes and will look at it again.

A big problem for newbies is that there are several sources I've been given that look good, and some seem to differ in details, and some (the more official-looking ones) get too complicated very fast.  It would be so wonderful if there were a section here on the forum for newbies, using just one-shot cameras with a basic workflow, with mention of how to know what might need some parameter-tweaking.

I'm pretty good at digging out and organizing information, but first I need to know what is applicable to me and what isn't.  I feel like I'm digging a hole to China right now.

I'll be posting more questions -- stay tuned.

And thanks!!

[pfile]

i guess a good rule of thumb is that anything that has to be done on the original pixel matrix from the OSC should be done before debayering. this includes calibration and cosmetic correction. most debayering methods employ some kind of pixel interpolation, and some methods even look at one channel in order to reconstruct another (though i think PI does not offer a method that actually does that.) anyway once you have started interpolating pixel values you can't reliably fix hot/cold pixels (CC) or calibrate, because those pixels have been 'smeared' into adjacent pixels.

rob

[Diane Miller]

But in the link dzso cited two panes above, the debayering step is done before CC.  He says (halfway down the page), "Since the light frames I am working with are monochrome, I can continue on to cosmetic correction. However, if your light frames are colour images from a one shot colour camera, you will first need to Debayer them. This process converts a raw image with colour matrix information into an actual colour image and must be done if your light frames are colour images."

And I have to debayer first, or nothing will load in CosmeticCorrection.

So -- that quandary settled.

Part of the difficulty is that there seem to be different workflows for different cameras.

[pfile]

it does not make sense that they won't read in CC. a CFA file from an OSC is mono, just like an image from a mono camera. CC can clearly load mono images since i do that all the time. and a debayered file is a 3-plane color image, which is what a raw RGB bayer image would be (the other choice for how to represent raw data from an OSC), which is the same representation as your debayered file.

i think we're looking at some kind of bug here rather than the wrong flow, because CC after debayering is wrong from a technical standpoint.

an experiment here would be to write the calibrated files out as fits and see if CC recognizes them OK. this could have something to do with the new xisf format.

rob

[pfile]

well... no idea what's going on - i opened a CR2 as bayer CFA and also bayer RGB, then saved it as 16-bit fits, 16-bit xisf, 32b float fits and 32b float xisf and cosmeticcorrection was able to open all of them.

i am using version 1190 on OSX, so perhaps the behavior is different on earlier versions of PI.

[pfile]

ah - ok, on the prior release of PI (1123), CosmeticCorrection does not know how to open .xisf files. that's the problem you're having. of course the debayered files are probably also xisf, so not sure what's different about them... unless your debayering step outputted .fits files.

rob

[Diane Miller]

OK -- so I'm considering myself stuck until I can get this working. 

I'm on a Mac (switched some years ago when my XP workstation died just after Vista came out, and now I hate them both).  My PI version is 1123 -- you have pinpointed the problem/bug.  All files have been output as .xisf.  Guess I could try going back to .fits for the whole workflow for now....

Or just hang it all up.  That's very tempting right now.

[Diane Miller]

But I'm curious (or confused) why "lightvortex" says what he did about the order of the steps:

"Since the light frames I am working with are monochrome, I can continue on to cosmetic correction. However, if your light frames are colour images from a one shot colour camera, you will first need to Debayer them. This process converts a raw image with colour matrix information into an actual colour image and must be done if your light frames are colour images."

But wait -- he was using .fits for all files.  Maybe I should wait on .xisf.  Can I mix the two in different steps?

[pfile]

i know the feeling!! i kinda took a year off of AP last year due to work and i think that was beneficial.

in theory, you should be able to take your calibrated .xisf files and run them thru BatchFormatConversion to convert to .fits and then continue from there rather than starting all over. but your mileage may vary - i am very surprised to see this XISF problem as of course software is always based on layers of abstraction. if the file I/O subsystem supports xisf then i would expect every process to also support xisf, but clearly not. anyway that's a long way of saying that BatchFormatConversion may also have problems with XISF.

at least this problem's been fixed in 1190!

rob
 

[pfile]

But I'm curious (or confused) why "lightvortex" says what he did about the order of the steps:

"Since the light frames I am working with are monochrome, I can continue on to cosmetic correction. However, if your light frames are colour images from a one shot colour camera, you will first need to Debayer them. This process converts a raw image with colour matrix information into an actual colour image and must be done if your light frames are colour images."

But wait -- he was using .fits for all files.  Maybe I should wait on .xisf.  Can I mix the two in different steps?

i think that quote is just incorrect, based on what i was saying about interpolation. it's not *impossible* to run CC on a debayered file, it's just that mathematically the results don't make sense (even if the resultant image looks OK).

you should be able to mix xisf and fits...

xisf is a new file format that juan invented to basically be a sane super-set of fits. fits has a lot of problems that xisf fixes up. but i guess for whatever reason this problem with XISF vs CC just slipped thru the cracks.

rob