Proper Steps to Convert Nikon nef files to fits [sanity check on file size]

[jerryyyyy]

Hi,

I have gotten some good flats for my D800 and a lot of good lights and trying to get the workflow right. 

These are all collected in nef.  There is a very long confusing post about procedures:

http://pixinsight.com/forum/index.php?topic=5708.0

This discussed BatchPreprocessing and BatchDeBayering.  Seems like there should be an optimal simple 1,2,3 set of procedures.  Is it just BDB for both Flats and Lights -> then BPP? 

When I do this, the typical D800 nef file starts as 43MB and ends up as 435MB as debayered fits [yes 435MB apiece -> sanity check here please].

Also, the flat(s) do not seem to be well applied.  If I do BPP before Debayering flats seem better applied. 

BTW  If an Administrator could help, a recent thread on the D800 seems to me to have the third page corrupted. 

http://pixinsight.com/forum/index.php?topic=7669.0

[pfile]

calibration of OSC images should always be done before debayering! you should just be able to throw everything into BPP. just make sure that "CFA images" is checked, or else BPP will not perform the debayering step before registration (the proper flow for OSC images is calibrate, debayer, align, integrate, which is what BPP does).

rob

[jerryyyyy]

Thanks.  Took > 1h to run....  I am actually amazed I can get this image from where we live.  The flats need more work but that is really very hard:

[andreasmax]

Hi jerryyyyy,

I also am using an unmodified D800 and am pretty surprized about the quality a typical DSLR can do. Ofcourse a b/w cooled astrocam is better.
But I am using the cam on an Astrotrac and will keep the setting light for travelling.
I usually are setting up my equipment in about 15 minutes.
All my pictures I store into Lightroom for organizing them, do an export as original RAW files to dedicated folders (Lights, Flats, Darks), I have stored the MasterBias files onto my server.
For Flats I am using my iPad with a white picture, a white tshirt (folded 2x) covering the hood and tested some settings on the cam. Best results were using Mode A, exposure correction set to +0.7.
All of them I put into BatchPreprocessing.
Yes, the final stacked images are about 450MB. Due to the huge chip data the D800 is creating.
This last picture I did took about 3 hours to stack on my Apple MacPro 2008 with 2xQuadcore 2.8, 16GB, SSDs. And I always have to use small previews for the AutoUpdateLive pictures.

http://www.astrobin.com/137766/

And yes, the thrid page of the forum entry you mentioned seems to be broken.
Cheers from Salzburg, Austria

Andreas

[pfile]

for your flats, here's what i would do. completely overexpose a flat and load it into PI as a raw CFA file, then debayer it. note what the value is using the readout cursor (likely to be 0.25). all 3 channels should be the same since it's completely overexposed.

then experiment with different flat exposures and load these flats into PI in the same way. using the histogram tool or cursor, find the one that has the best histogram - where the peak is somewhere in the vicinity of 1/2 the value you measured in the first step. if you want to get fancy, you can plot the pixel values at the center of the frame vs. the exposure time. the graph should form a straight line for a while, but at a certain exposure the graph will diverge from a straight line. that's the point where the sensor is no longer linear. whenever you make flats, you want the brightest part of the flat to have ADU values that are still within the linear range of the sensor, but no brighter.

having said all that, for canon cameras, Tv metered flats + 2EV works pretty well. not sure if that translates to nikon. if you are using some kind of light source for flats you need to worry about whether or not that light source is flickering and make sure that your exposure is long enough to capture multiple cycles of flicker, just so you don't end up with uneven flats. if you have to lower the ISO to accomplish this, that's ok, but you should probably calibrate the flats with matching ISO biases/darks.

sometimes if you are using a filter of some kind, one of the channels will be very bright and the others too dim. this is bad because the SNR in those other channels is too low. the only way to compensate for this is to tune your light source such that it undoes the color cast of the filter. for instance i used to use an astronomik CLS filter which has a very blue cast, so i used a slightly pink T-shirt to make flats.

rob

[jerryyyyy]

Well guys, I appreciate all the input.  I have been warned about the flats using a Electroluminescent screen by Wei... flicker... although I saw a 12"x12" one on Alibaba for $32. 

I spent some quality times at twilight last night with my D800 and must have collected 200 flat images.  I followed Pfiles suggestions on processing and got the following images.  The grey scale are the BPP CFA processing of the Pleiades and the flat.  I have to say that if you look at these images they are pretty nice and I got my old SSAG working (as good at my SBIG if not better) so this is a guided image of 120sx19. 

But these have the Bayer patterning in them and when I debayer, I get the abstract impressionist art.  I thought I should post this on Astrobin and would have a good chance for the image of the day. 

Any suggestions?  Seems that the flats give a nice total grey result, but the color parts may be independent?  Wonder if creating separate flats for RGB and applying them to separate RGB light images?  A lot of work, but the data are there somewhere... this took 1h+ to process but the images are huge. 

[pfile]

if the integrated light still has the bayer pattern, then something has gone wrong… the steps are calibrate, debayer, align, integrate.

if the integrated light was made from bayered images that have been thru star alignment, then the result of debayering the integrated light is pretty meaningless. debayering always has to happen on an image which has not been rotated, scaled or translated (although there are some translations that would still be legal, there's no reason to do it)

did you check "CFA images" in the BPP main control panel?

rob

[jerryyyyy]

I redid it from scratch and it worked entering all the raw files... took 1.5h.  Must have screwed up before.  There is a small issue at the bottom (Lack of biases?) but I am wildly enthusiastic.  These are unprocessed just with STF. 

[jerryyyyy]

Image posted.  Still need to work on noise reduction and I think there is dirt on the sensor and I have no biases. 

[pfile]

that looks a lot better!

the huge file size is kind of a mystery. 43MB -> 435MB is 10x in size. for the sake of argument if the sensor is 1000x1000 x 16bits, that's a 2MB file. when debayered to a 3-plane, 1000x1000 file x 32bits, that's 12MB. so there's a 6x increase, not 10x.

the line along the bottom is most likely the shadow of the mirror. the 180ED has a very wide light cone owing to its fast f/ ratio. the light cone is much wider than any nikon lens would ever produce, so it sort of falls outside of the design spec as far as nikon is concerned - you'd probably never see the shadow of the mirror when using any nikon lens. it's possible that because the flats were not bias-calibrated, the scaling is wrong and the shadow of the mirror is not fully eliminated by the flat.

rob

[Don]

When I do this, the typical D800 nef file starts as 43MB and ends up as 435MB as debayered fits [yes 435MB apiece -> sanity check here please].

The file sizes are about right.  The D800 acquires 36 megapixel images at either 12 or 14 bits per pixel (user choice), and stores the data in the NEF file either uncompressed or with lossless compression (also the user's choice).  This would result in either 54 MB or 63 MB raw files without compression, so it seems your 43 megabyte NEF files are stored with lossless compression.  The lossless compression is not a problem, but you need to make sure you are acquiring 14 bits per pixel instead of 12 bits per pixel.

In either case, the uncompressed, debayered 36 megapixel images at 32 bits per pixel would be around 430 MB in size:

36 megapixels x (32 bits per pixel / 8 bits per byte) x 3 channels = 36 x 4 x 3, or 432 megabytes

Regards,

Don

[jerryyyyy]

When I do this, the typical D800 nef file starts as 43MB and ends up as 435MB as debayered fits [yes 435MB apiece -> sanity check here please].

The file sizes are about right.  The D800 acquires 36 megapixel images at either 12 or 14 bits per pixel (user choice), and stores the data in the NEF file either uncompressed or with lossless compression (also the user's choice).  This would result in either 54 MB or 63 MB raw files without compression, so it seems your 43 megabyte NEF files are stored with lossless compression.  The lossless compression is not a problem, but you need to make sure you are acquiring 14 bits per pixel instead of 12 bits per pixel.

In either case, the uncompressed, debayered 36 megapixel images at 32 bits per pixel would be around 430 MB in size:

36 megapixels x (32 bits per pixel / 8 bits per byte) x 3 channels = 36 x 4 x 3, or 432 megabytes

Regards,

Don

Thanks for the math help!  Pretty sure the images were 14-bit at the default settings of CameraRC.  In general the images have a good deal of noise compared to CCD images but of course they are also huge and oversampled so there may be room for processing.  The enclosed is unprocess M42.  Working on that before the turkey today.

Happy Holidays!