Conversion NEF to RAW RGB after update ?

[mikeoday]

Hi

I just updated Pixinsight ( last updated December ) and it seems I now need to divide my NEF files by 4 in order to use them in my old processing workflow.  Is this correct?

To be precise, NIKON adds a bias of around 600 to its images.  My Bias frame has a mean around 600.  But when I now load NEF files they have a floor around 2400 and if I plug these into my processing chain I get a mess.

Have I lost the plot or has something changed.

Cheers
Mike

[bulrichl]

Are you sure that in PI's 'Raw Format Preferences' the setting regarding black point correction was the same before and after the update?

I have no NEF files so I cannot check this. My Canon CR2 files did not behave differently after having updated PI. For my EOS 600D (14-bit ADC) the bias offset is set to 2048 ADU. For different models this value is also different, Canon uses e.g. a lower value of bias offset for cameras with 12-bit ADC. A further possibility is that - depending on the camera settings (e.g. ISO setting) - a different bias offset is set in the camera. So there is some possibility that the issue is connected with the camera or camera settings:
Are you comparing images of the same Nikon camera model?
Are you comparing images that were captured with the same camera settings (e.g. ISO settings)?

If the observed behavior is indeed not caused by one of these points, I could imagine that it might result from a different LibRaw version. However, the history of LibRaw versions doesn't mention any changes in this regard, so this possibility seems quite unlikely.

PI 1.8.6 build 1447 was released on 16.12.18, build 1448 on 19.12.18, both with the RAW module based von LibRaw 0.19.1. 26.12.19 Juan released an update of the RAW module based on LibRaw 0.19.2, see
https://pixinsight.com/forum/index.php?topic=13049.0
With the following PI updates (builds 1457 on 23.01.19 and 1473 on 14.05.19), no further change of the LibRaw version was mentioned.

Bernd

P.S.: Out of curiosity: why do you use the 'Create raw RGB image' setting instead of 'Create raw CFA image'?

[mikeoday]

Thank you very much for the repley.

Are you sure that in PI's 'Raw Format Preferences' the setting regarding black point correction was the same before and after the update?

Sorry, not really, I set up the raw conversion months/years ago.  It took me ages to get it to work with a raw workflow.  I am sure I wrote down my settings but I can't find them.  I assumed that the update would not change any of my settings - I should have saved them just in case.  What I am using now is Raw RGB, no black point correction and no white balance, all else off.


I have no NEF files so I cannot check this. My Canon CR2 files did not behave differently after having updated PI. For my EOS 600D (14-bit ADC) the bias offset is set to 2048 ADU. For different models this value is also different, Canon uses e.g. a lower value of bias offset for cameras with 12-bit ADC. A further possibility is that - depending on the camera settings (e.g. ISO setting) - a different bias offset is set in the camera. So there is some possibility that the issue is connected with the camera or camera settings:
Are you comparing images of the same Nikon camera model?

Are you comparing images that were captured with the same camera settings (e.g. ISO settings)?[/i]

I am using 14bit NEF files.  No change to camera or camera settings.  Worked fine before update to PI.  Only change is PI update.

From memory, with the Nikon, whether I used 12bit or 14bit modes, the offset was always 600.  Furthermore, if I had a 14bit image, I needed to multiply it 4 in order to occupy the top 2 bits of the 16bit histogram.  Or to say that another way, if I looked at the statistics ( 16bit scale ) of the newly converted 14bit image the maximum ( burnt out ) pixels had a value of 16383.  I needed to multiply them by 4 to get the maximum to 65535.  Now it seems that the multiplication is happening during the conversion so that the top 2 bits are now occupied.

P.S.: Out of curiosity: why do you use the 'Create raw RGB image' setting instead of 'Create raw CFA image'?


Ah, long story ... and yes I know there are benefits in using raw CFA images ...

It took me ages to get my raw workflow working.  It was a couple of years ago and for some reason I could not get it working with raw CFA but Raw RGB worked so I stuck with it.  Ever since I have been reluctant to mess with a workflow that worked.  The problem now is that I have forgotten how or why I came up with the workflow so changing it is an even bigger worry.  If all have to do to work around the issue is add an initial step where I first divide the newly converted images by 4 then I will do that.  Next step I guess is to look at changing my bias, darks and flats - maybe all I need to do is multiply these by 4 and use the converted files as is (?). 

Thanks again for your reply.

[Juan Conejero]

Hi Mike,

It seems to me that you are complicating things unnecessarily. Everything is essentially automatic in current versions of PixInsight until you reach the image integration step. Use the BatchPreprocessing script to calibrate, demosaic and register your frames, with generation of drizzle files enabled. The script will take care of all settings necessary to load and process pure raw data. If you prefer to preprocess your raw frames manually—I don't see any practical reason to do so, unless your data set has some very special issues—, make sure you specify the 'raw cfa' input hints in the ImageCalibration tool.

Once you have preprocessed your data, use ImageIntegration to refine pixel rejection parameters. Then use DrizzleIntegration to integrate your data for optimal results, with the CFA drizzle option enabled. Use drizzle x1 to perform a regular CFA drizzle integration. Use drizzle x2 if your data are undersampled and you have a very large set of well-dithered frames.

[mikeoday]

Thank you Juan, I'm sure you are right and I will have another look at it ...

One of the reasons I went the way I did is that I found that when using the automatic scripts I had trouble with colour balance and by breaking the steps down I managed to fix that. 

Thanks again I will try the automatic process again.

To use it though, will have to multiply my old bias, darks and flats by 4 to take into account the fact that the converted 14bit NEF files seem to now occupy a 16bit rather than 14bit range?

.............

Edit: I just remembered one reason why the automatic script may not always suit.  I noticed that I was getting colour spread across my outer stars.  I found that if I process the images in separate raw channels, align all channel images against a single green reference using distortion correction, drizzle integrate each channel set separately ( one set each for R, G, B ) and then recombine, I end up with a final "debayered" image with the same number of pixels as the original but with significantly reduced colour spread in the outer stars.  I'm not sure but I think I also might get better noise reduction - at least that is what I remember seeing when I was experimenting with the process and comparing results with the standard process.

[bulrichl]

Mike,

in order to further investigate this issue I would need some files. Can you please upload
1) one of the current light frames in NEF format,
2) one of the old dark frames in NEF format and
3) an old MasterDark in XISF format
to a file hoster and share the link?

Please disclose your current settings of the 'Raw Format Preferences' as well.

Bernd

[mikeoday]

Thank you very much, yes I will.  Off to work now, I will dig them out tonight and upload them.

[mikeoday]

Hi Bernd

Before I post the very large files I thought I would find a way to show the issue.

I have an old version of PI on my laptop and the new one on my desktop.  All I did was open the same file in both and copy the statistics ( no other processing).  Below are the results. ( Sorry, I can't attach more than 4 files and I don't seem to be able to get the image insert to work so most are just links instead )

Old PI version info:   www.dropbox.com/s/j0nywijnqevokg2/old%20version.png?dl=0

New PI version info: https://www.dropbox.com/s/c4nrc2mkoxfym7f/new%20version.png?dl=0

Old PI Raw Preferences:  https://www.dropbox.com/s/vvdda8jnsmyfwse/Raw%20preferences%20-%20old%20pixinsight.png?dl=0

New PI Raw Preferences:  https://www.dropbox.com/s/dvuww0fxgqbc86k/Raw%20preferences%20-%20new%20pixinsight.png?dl=0

Old PI console log for file open:  https://www.dropbox.com/s/6xih70twej7kybm/Raw%20conversion%20log%20-%20old%20pixinisight.png?dl=0

New PI console log for file open:  https://www.dropbox.com/s/4mgjwrco8gcll08/Raw%20conversion%20log%20-%20new%20pixinisight.png?dl=0

Old PI image statistics:  https://www.dropbox.com/s/e801th9whc0s0vm/Statistics%20old%20pixinsight.png?dl=0

New PI image statistics:  https://www.dropbox.com/s/k32we9sewvq59zt/Statistics%20new%20pixinsight.png?dl=0

As you can see from the statistics, the maximum pixel values are different using the different versions of PI:

Old:  max: 16383, 16383, 16383
New: max: 65434,  65515, 65354

So, basically they are 4 times as large. 

Also, I see that the new PI maximum values are not all the same ( which is "wrong" because the brightest pixels were actually clipped for all channels).

So do I still need to send the darks, etc. as the issue is at the very first step of opening/converting the raw files?

Thanks again

Mike

[bulrichl]

Hi Mike,

I'm pretty sure that the decisive difference between your PI Raw Preferences is the 'No highlights clipping'. Please try to enable the 'No highlights clipping' for the NEW PI version (build 1475), load the same NEF file and compare the statistics with the result obtained with the old PI Version (build 1448).

When this option is not enabled, the RAW module will clip highlights and rescale the result. This must not be done since you still want to calibrate the data.

Please try and report, maybe you need not upload the big files.

Bernd

[mikeoday]

Thnak you, yes I will try that.

[mikeoday]

Hi Mike,

I'm pretty sure that the decisive difference between your PI Raw Preferences is the 'No highlights clipping'. Please try to enable the 'No highlights clipping' for the NEW PI version (build 1475), load the same NEF file and compare the statistics with the result obtained with the old PI Version (build 1448).

Hi Bernd,  well spotted and thanks!

Please see image attached - the statistics are now as they should be after ticking the no clipping box.

I guess I was confused because when I looked at the old version most of the "extra boxes" ( black point, highlights, etc. ) were not ticked but greyed out.  I assumed ( silly me ) that they should all remain un-ticked in the new version even though they were now not greyed out.  The only one I had ticked was the "no black point" because that one was obvious.  I should have experimented with the others, so sorry to bother everyone.

Thanks again for the great help.

Mike