IC (master dark, bias, flat) and Sony ARW files

[zillion42]

Hello,

I'm new here, and I'm new to astrophotography. I have one ok picture so far (attached), which I stacked with deep sky stacker and processed with Photoshop. Since I own a full license of PI, I'm eager to learn more about my specific worklflow using DSLR Raw Files
My Camera is a Sony Nex-5 and the Sony Raw files are called .arw.

I'm trying to make a  master dark, bias and flat following this tutorial
http://pixinsight.com/tutorials/master-frames/en.html

First for the technical part (please correct):

• My Dark frames are taken on the same day and apporximately with the same ambient temperature, exposure and ISO as the light frames ?
• My Bias frames are taken with room temperature and the shortest possible exposure (1/4000s) but same ISO ?
• My flat frame is made with the camera attached to my telescope and focal reducer with a white t-shirt in front of the lens looking at a bright wall ?

For the ARW part:
I read somewhere...
- Open the Format Explorer window
- Double click on the DSLR_RAW item
- On the RAW Format Preferences dialog, select:
- Uncheck both white balance options
- Check the Create RAW Bayer image option
- Check the No black point correction option
- Click OK and you're done.

Is that necessary ?

and yes, I understand those are newbie questions. I'm currently reading the DSLR_RAW work flow tools PI 1.6.x - 1.7.x, would those Batch scripts be applicable for PI 1.8 aswell ?

Thx a lot in advance
Tobi

[Philip de Louraille]

Under the Scripts menu, access the Batch Preprocessing Script. It is more recent than the tutorial you are reading. Steps are combined.
A tutorial can be found here:

http://troypiggo.blogspot.com.au/2012/06/pixinsight-tutorial-batchpreprocessing.html

Cheers,

[Geoff]

Also see here http://www.harrysastroshed.com/preprocessing.html

[zillion42]

Thank you !

ok, I'm reading. I would like to stress the technical aspect of creating dark, bias and flat frames again, please correct me. I found a pretty good description about aquisition is the one posted in the deep sky stacker FAQ:

Light Frames
The Light Frames are the images that contains the real information: images of galaxies, nebula...
This is what you want to stack.

Dark Frames and Dark Flat Frames
The Dark Frames are used to remove the dark signal from the light frames (or the flat frames for the Dark Flat frames).
With DSLRs and CCD Camera, the CMOS or CCD is generating a dark signal depending of the exposure time, temperature and ISO speed (DSLR only).
To remove the dark signal from the light frames you use a dark frame that contains only the dark signal.

The best way to create the dark frames is to shoot pictures in the dark (hence the name) by covering the lens.
The dark frames must be created with the exposure time, temperature and ISO speed of the light frames (resp. flat frames).
Since the temperature is important try to shoot dark frames at the end or during your imaging session.

Take a few of them (between 10 and 20 is usually enough). DeepSkyStacker will combine them automatically to create and use a clean master dark or master dark flat.

Bias Frames (aka Offset Frames)
The Bias/Offset Frames are used to remove the CCD or CMOS chip readout signal from the light frames.
Each CCD or CMOS chip is generating a readout signal which is a signal created by the electronic just by reading the content of the chip.

It's very easy to create bias/offset frames: just take the shortest possible exposure (it may be 1/4000s or 1/8000s depending on your camera) in the dark by covering the lens.
The bias frames must be create with the ISO speed of the light frames. The temperature is not important.

Take a few of them (between 10 and 20 is usually enough). DeepSkyStacker will combine them automatically to create and use a clean master bias/offset frame

Flat Frames
The Flat Frames are used to correct the vignetting and uneven field illumination created by dust or smudges in your optical train.

To create good flat frames it is very important to not remove your camera from your telescope before taking them (including not changing the focus).
You can use a lot of different methods (including using a flatbox) but I found that the simplest way is to put a white T shirt in front of your telescope and  smooth out the folds. Then shoot something luminous (a flash, a bright white light, the sky at dawn...) and let the camera decide of the exposure time (Av mode),

The flat frames should be created with the ISO speed of the light frames. The temperature is not important.

Take a few of them (between 10 and 20 is usually enough). DeepSkyStacker will combine them automatically to create and use a clean master flat frame.

Is it possible to use colored flat frames?
The short answer is yes.
The overall tint of your flat frames is not really important because DeepSkyStacker is processing each channel separately and is applying the flat frames to each channel accordingly.
Of course if your flat frames are fully red only the red channel of your light frames will be properly calibrated but otherwise as long as as the peak in each channel is between 1/3 and 2/3 of the maximum you should have no problem using slightly colored flat frames.

Would those instructions be satisfactory for PI aswell ?

Its probably obvious in the picture I posted above, I have so far not created any flat frames (see vignetting). So considering that I found some detailed instructions here:

FLAT FIELD FRAMES

Flat fields are an important part of the image calibration procedure. Well prepared flat fields correct for varying sensitivity of the CCD chip and uneven illumination of the CCD chip due to the optics and dust and other objects on the optical elements.

The current wisdom in collecting flat field frames is to expose them long enough to collect sufficient light to expose the chip to 35% to 50% of its full saturation level. The saturation level is determined by dividing the gain of the camera into the chip's full well capacity. For example, my ST-10XME has a full well capacity of 77,000 electrons. To find the saturation level, I divide the gain, 1.3 electrons/ADU, into the full well capacity. The result is 59,213. Thus, pixel saturation point of the ST-10 is 59,213 counts. The best range for the flat field counts using this camera is 20,000 to 29,000 counts.

I always calibrate my individual flat field frames with a master dark frame of the same exposure time as the flat field frames. Although the dark current contribution in a flat field image of a few seconds is negligible, the pre-charge applied to activate the CCD can be significant [about 1,000 counts in the case of my ST-2000XM with a KAI-2020 chip]. Thus, I make it a practice to also subtract dark frames [which also takes care of the bias] from my flat fields.

Flats are a good application for a scalable dark frame. My exposure times to reach optimal chip saturation varies with the filter. Thus, I can take one set of longer exposure time scalable darks and a set of bias frames which can then be later used to create a master scalable dark frame for application to the flat frames.

A single flat field will have random noise. Thus, a median combination of at least 16-20 flat fields will create a high signal-to-noise flat field which will not add noise to your image when applied.

The image below is an example of a master flat field frame through the Takahashi Epsilon 160. the flat below contains examples of vignetting, dust on the optical element and uneven illumination of the chip.

Especially this sentence makes me wonder:

The current wisdom in collecting flat field frames is to expose them long enough to collect sufficient light to expose the chip to 35% to 50%

Should the center of my flat frame really be 35%-50% gray or should it be white for proper results in PI ?

Thx a lot again.

Tobi

P.S. I find it very true considering documentation:

I know of quite a few people who have bought the astrophotography image processing software PixInsight (PI), but only use it for one or two specific features, such as the gradient removal tools, or merging mosaics. The common reasons/excuses given for not using it for their full processing workflow seem to be "there isn't sufficient documentation", "I tried it, but calibrating and stacking images was so tedious, I just want to click a few buttons and have my images stacked", and "I don't want to have to learn a whole new platform."

[Philip de Louraille]

You do not want your flats to be white (fully exposed.) You want them to be gray. Hence go for less than 50%. It is the variations of the grays that will be used for correcting (flattening) the frames.

[zillion42]

Thank you again !

getting my car, thus getting my scope and then getting a white t-shirt momentarily.

[Josh Lake]

Regarding this quote:

I know of quite a few people who have bought the astrophotography image processing software PixInsight (PI), but only use it for one or two specific features, such as the gradient removal tools, or merging mosaics. The common reasons/excuses given for not using it for their full processing workflow seem to be "there isn't sufficient documentation", "I tried it, but calibrating and stacking images was so tedious, I just want to click a few buttons and have my images stacked", and "I don't want to have to learn a whole new platform."

Most people are a bit taken aback by the 'documentation issue' in PI, but from the other side, having learned PI, it's really not a problem. Easy for me to say, I know... but the methods for learning the platform are: online tutorials, both typed and screencasts; the tooltips within PI, as you hover over an image; seeing others do it and mirroring their steps with your own variations. When Juan and the others do documentation, they do it very, very well: http://pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html

As for the 'click a few buttons and have my images be stacked,' that's where Batch Preprocessing comes in, that complaint is no longer valid. But *good* processing takes a bit of care, and you'll find your results to be noticeably superior. Unless, of course, you spend the time to truly and deeply learn another whole new platform.

I've been on that steep learning curve hill, but from the peak (or a fairly high ledge!), I have to say that it's worth it.

[Geoff]

Most people are a bit taken aback by the 'documentation issue' in PI, but from the other side, having learned PI, it's really not a problem. Easy for me to say, I know... but the methods for learning the platform are: online tutorials, both typed and screencasts; the tooltips within PI, as you hover over an image; seeing others do it and mirroring their steps with your own variations. When Juan and the others do documentation, they do it very, very well: http://pixinsight.com/doc/tools/ImageIntegration/ImageIntegration.html

As for the 'click a few buttons and have my images be stacked,' that's where Batch Preprocessing comes in, that complaint is no longer valid. But *good* processing takes a bit of care, and you'll find your results to be noticeably superior. Unless, of course, you spend the time to truly and deeply learn another whole new platform.

I've been on that steep learning curve hill, but from the peak (or a fairly high ledge!), I have to say that it's worth it.

All true, but the ultimate success of PI rests in recruiting as many users as possible and I'm sure the lack of an easy intro to PI has deterred a lot of prospective users.  I would never have gone the PI route if it hadn't been for the advice of someone who is an absolute expert on image processing.
Geoff

[zillion42]

Hmm,

I'd like to jump in right at that point again, because I really need to get this off my chest.

"you can't expect your customers to know how to milk the cow you just sold them if you are unable to explain to them what an udder is."

Seriously, I can understand that most of the PI customers so far are observatory operators and doctorates who use mighty equipment and image in  LRGB+H-?, do exposures of 30 minutes or longer for each of their subs and generally produce the best images out there, using the best software out there. That is all fine and good, you have established workflows and pipelines for these people and a mighty respect for that achievement.

But then again Pleiades Astrophoto, needs to realize at some point or another, that there is a whole new market segment, which until now they are not really able to satisfy. Even worse the before mentioned market segment is the biggest out there. Your average customer is not sitting in an observatory and is not imaging LRGB+H-? with a fancy expensive cooled CCD camera and Baader Planetarium filters, but rather owns a small refractor/reflector and images with a DSLR.

Marketing wise, like I said earlier I can understand why the emphasis lies on the professionals in your customer pool, but you should really consider spending much more time and effort in the amateur segment in case you feel like building a whole community and as a side effect earn more money.

Just my two cents.
Tobi

[pfile]

had a huge repsonse typed up... but forget it. really just tired of this meme.

[zillion42]

had a huge repsonse typed up... but forget it. really just tired of this meme.

I can really understand how that topic gets on peoples nerves, its probably not the first time it came up. I really didn't want to go down that road when I signed up here, and it will be the last time anyone hears anything concerning that topic from me.

Altogether, I just really needed to get it out of my system yesterday night, so steem is gone. Going to search the forum now, in search for answers to my other Thread.