[newbie] Issue to integrate RAW bias from nikon d800

[zangetsu74]

Hello everybody,
I have been trying PI this summer, and I was delighted with the results (I was shooting at that time with a nikon D300s and a Nikon D700)
I followed a french tutorial (arnaudom.fr) that helped me in getting very good results (or at least results I was satisfied of )

Therefore I decided to buy PI this week, as the evaluation period was over and as I went out sunday nigh to shoot M31 (I have now changed my main camera which is the Nikon D800)

My issue is that I am stuck at the first step of the tutorial : bias integration.

I first have a warning message :** Warning: No convergence in MRS noise evaluation routine - using K-sigma noise estimate.done
(WHICH I also have when I used to integrate D700 or D300s bias)
BUT afterwards it gives me an error message (that I did not have before) :
*** Error: C:/Users/.../Documents/Scripts/PixInsight/RAW/offsets/SFP_141027_5394.NEF: Zero or insignificant signal detected (empty image?)
<* failed *>

And the image integration process stops.

I have tried a couple of things to make it work :
- use different bias (my first lot is 1/8000s and I have done another lot with 1/4000s) : FAIL
- reduce buffer size / stack size : FAIL
- raise buffer size / stack size : FAIL
- change the sigma low/high level in pixel rejection (2) : FAIL
The only way to get the process going on is to untick the "evaluate noise", but I am not sure it is the correct way to go.

Do you have any idea of what I am doing wrong ?

(I have already tried to search the forum, but did not find any helpful tip/hint for my case)

Thank you for your time & help

[jerryyyyy]

Just let me send a note to let you know people are reading.  I have a D800 but use my SBIG for astro since I never figured out how to do this.  I'll look at the tutorial. 

I would like to ask what software are you using to capture the images and biases.  Are the FITs images?  Or Nikon Raw?  I have used Maxim, which is unreliable.   I found a lot of noise in the D800 at normal temperatures. 

If it is a technical issue in how you are using the software, one of the gurus here will respond. 

There is one other person I know who uses the D800 for astrophotography, Wei Wang, on astrobin:

http://www.astrobin.com/users/whwang/

[zangetsu74]

Hello !
Thank you for your message.

Well, I do not use any software to capture.
I only have photo gear and recently bought (last year) an equatorial mount to get longer shutter times.
Therefore my D800 "works alone" and then I copy the raw files (.nef) to the pc using Lightroom (I shall try this morning to directly copy-paste from the SD card to see if it changes something. Nevertheless, it used to work with LR and d300s/d700 raw files before, that is why I dare to ask here)
I have also tried yesterday evening to convert the raw files to fit before integrating them, but I had the same issue.

It might be important to note that my issue is when I do not de-bayer the files ("CFA" and no black point correction) ; if I debayer them it works, but it does not seem the correct way to go.

By the way : I have tried on Windows 7 64bits (both home desktop and work laptop), it is the same.

EDIT : I have just tried with "copy-paste" from sd card (no lightroom in between), I still have the same issue.
EDIT 2 : just tried with the 1.8.3 update (thought the newer version of dcraw might do the trick, but no : FAIL again )

Feel free to ask whatever log or anything that might be helpful to understand what is going on !
(I do not know exactly what to send "by default" :/  )

[georg.viehoever]

Did you check what data is actually in the failing file SFP_141027_5394.NEF? Is this a light, bias, dark or flat? Have a look at it in PI, and check the statistics. Have you switched off all internal processing within your camera (such as noise reduction etc.)? Did you try using the BatchPreprocessing script?

Georg

[zangetsu74]

Hello,

Yes I did check the nature of the pic : the failing file (and the others) are bias. That's the first step of my tutorial, therefore I did not get farther.
Noise reduction is always switched off in all my cameras (and all algorithms likewise : my pc is more powerful than the reflex).

As for the statistics, what I am supposed to do exactly to do that (sorry, I am a REAL noob with PI...) ? I open the file and copy-paste the log here afterwards ?

I did try BPP, but I am not sure of the "DSLR_RAW" config at that time. I shall try again this morning and let you know about that

Many thanks for your time, and sorry for the inconvenience.

[georg.viehoever]

...
As for the statistics, what I am supposed to do exactly to do that (sorry, I am a REAL noob with PI...) ? I open the file and copy-paste the log here afterwards ?
..

You can use the Statistics process. I would also recommend that you open the offending bias files and check what you see there. Does it look like a real bias file? If the signal in your bias files is really this low, I think you can actually switch of the noise evaluation. I don think it is used anywhere in the processing of bias files.

Georg

[zangetsu74]

Great, I know that when I uncheck "noise evaluation" it does work perfectly (for the bias integration), I shall try this way and continue the process.

Here are the statistics of the file we are talking about (I did not check "unclipped" box) :

SFP_141027_5394
               K
count (%)      16.61913
count (px)     6037609
mean           2.1495e-005
modulus        1.2978e+002
norm           1.2978e+002
sumOfSquares   4.0618e-003
meanOfSquares  6.7275e-010
median         1.5259e-005
variance       2.1071e-010
stdDev         1.4516e-005
avgDev         7.8155e-006
MAD            0.0000e+000
sqrt(BWMV)     0.0000e+000
sqrt(PBMV)     1.0805e-005
Sn             0.0000e+000
Qn             0.0000e+000
minimum        1.5259e-005
maximum        2.6093e-003
minimumPos         14,     0

you are going to "kill me", but I am not sure of what a real bias is supposed to like in PI after STF, therefore here is a screenshot so you can check...


EDIT : I add the exifs :

Timestamp: Mon Oct 27 22:48:58 2014
Camera: Nikon D800
Owner: (c)Sebastien Fanget                 
ISO speed: 800
Shutter: 1/8000.0 sec
Aperture: f/5.6
Focal length: 300.0 mm
Embedded ICC profile: no
Number of raw images: 1
Thumb size:  7360 x 4912
Full size:   7424 x 4924
Image size:  7378 x 4924
Output size: 7378 x 4924
Raw colors: 3
Filter pattern: RGGBRGGBRGGBRGGB
Daylight multipliers: 2.099064 0.928145 1.096731
Camera multipliers: 1.925781 1.000000 1.410156 0.000000

[jerryyyyy]

Georg,

Thanks you for the response.  I'll copy this to my friend who also has a D800. 

PS  It would be good to develop a step-by-step procedure for importing D800 files into PI.  A lot of people who like to use this camera if it were a bit more user friendly.  In my collimation tests with my Tak 180 ED that chip in the D800 was flatter (more normal to the light input) than was the chip on my SBIG CCD. 

JY

[whwang]

Hi,

I am probably the one jerryyyyy referred to

I had limited experience using PI to process D800 raw images, and I don't
remember seeing this.

What I do know is that in D800's bias files, there are many zeros, because of
the notorious black-clipping of Nikon cameras.  (There is a Nikon firmware
hack that can solve this.  You may want to check the nikon hacker forum.)
This definitely makes statistical analyses on D800's files give unreliable
results, but it kind of surprises me that it leads to a failure.

In your original post, you said that this problem goes away if you uncheck noise
evaluation.  I would say if the results of the final image looks OK, then
it is not too big a problem to uncheck this.  (In another recent thread, I asked
how to disable any statistical analyses during the integration.  I am not a big
fan of such analyses, although I do believe it could be useful for other people.)

An alternative is to not provide bias files at all.  (I am not sure if this is
allowed by PI.)  As mentioned, because of the black-clipping of Nikon DSLRs,
D800's bias file mostly contains zeros.  Not using bias files will only have minor
impact to the final results, especially when you have good dark files. 

These are what I might do to bypass the problem.  Really solving the problem
may require more investigation on your bias file and on PI.  Unfortunately I am
traveling, otherwise I can simply take some bias images with my D800 and see
if I can reproduce the failure you saw.  Sorry for being not too helpful.

Cheers,
Wei-Hao

[sil]

I use a D800E too. I've seen that error before from corrupt raw files from running out of drive space. I used to run them through a Photoshop action to prepare lights or subtract darks and save as 16bit tiffs to take into PI. The BatchPreProcessing will chew up drive space but PI works just fine with NEF files. I also keep my master flat, bias and Dark frames to use later if i need them (Igo back andreprocess sets later on as I learn more plus I've spent 5 months in hospital fighting paralysis and learning to walk again after a stroke, as I am currently limited in what equipment I can use I've spent a lot of time recently re-processing.

[zangetsu74]

Hello everybody,
Just to let you know that I have eventually succeeded in processing this M31 with my D800 raw files...
without any dark bias or flat.
When I tried to use only dark & flat files, it gave me black light files.
Therefore I tried without.
Here is the result (I know this is not perfect, but I guess I do not miss those dark & flat files too much) :
http://www.astrobin.com/full/134057/0/
(it's a ~50% crop from the final file)

Next time, I'll get out with my old D700, as D800 raw files are really to big and slow the whole process

EDIT : I have also tried with "noise evaluation" unchecked for bias. Then I had a problem with dark processing. So I unchecked it also for darks. But it gave me also "black" light files (even with STF). Therefore I tried without dark also.

[zangetsu74]

Hello,

A quick update to let you know that I had eventually made it with bias/darks&flats.
I got my master files & calibrated lights from the BPP, then I followed my french tutorial.
Here is the final result :
http://www.astrobin.com/full/134057/C/

Which seems not too bad to me for a first try.

Thank you all for your time.

Note : when I have some freetime, I shall check what settings BPP uses for getting those masterfiles (or not, since it is an automated process, it saves me some time in front of my screen ;-) )

Have a great day :-)

[jerryyyyy]

Looks good.  I have some problems with my CCD camera so maybe I'll put the D800 back up on the mount and see what happens.  You seem to have getting the control files down right.  I forgot, but do you have much light pollution where you are?  My sky is really clobbered. 

[zangetsu74]

thank you :-)

I live by Geneva Switzerland, which is quite a polluted place (both lights & particles).
There are worse places in France (Paris, Lille area, and so on), and I can get up a little (Mont Salève - around 1250m high) in half an hour from home.
So I try to shoot only when the objects are really up in the sky (which was the case for M31 on October 26th evening in my place).

Don't know if it helps you though, let me know.

[jerryyyyy]

Seems to be in a similar situation to me.  But, I cannot take my mount and Tak anywhere but my backyard. 

Can you tell me what exposure settings, you used and how many images you collected.  I'll try to reproduce the image here with the Tak.  Will be a good exercise for data collection and processing.  If I understand correctly you are using the Nikon Raws, but what software do you use to collect them?  Or do you just use a bulb release?