Newbie question on batch integration of calibration files

[Phil Leigh]

This does seem to be a file format issue - the light frames are a different bit-depth to the calibration frames and when the bias or dark is subtracted there is almost no data left...
The calibration files should be 16 bit integer, same as the lights
It is best to let PixInsight create the calibration masters.

Just load all of the raw bias/dark/flat/light frames straight from the camera into the BPP script and let it run.

[Buzz]

Hi Harry - is that it? I would have thought it would be mentioned somewhere as I imagine Maxim is a popular acquisition program.

So to understand - calibration is all done in 16-bit and then do the subsequent processing in 32 bit float - or do I stay clear of 32-bit entirely in the workflow?

By the way, do you have to remove the helmet to look through the eyepiece

[Harry page]

Hi

Its has been mentioned before 

Its just the old Fits file is not as universal as it should be , all files coming into pixinsight 16 bit and then on to 32 bit , this even means the calibration files
created in pixinsight will be 32 bit 

As mentioned before I encourage you to create all masters etc in Pixinsight to avoid any compatibility problems

Regards

Harry Page  ,, close relation of the stig   

[Phil Leigh]

The raw bias/dark/flat/light files from  Maxim should be 16-bit because that is what the camera is. Once the 16-bit files have been fed  into the BPP script, everything that Pi creates/saves (including the master bias/dark/flat) will be in 32-bit.

[pfile]

right - the process of averaging a bunch of integers can (usually does) lead to floating point numbers as the result, so representing the output of ImageIntegration as some kind of high-precision float makes sense.

since your masters are in floating point format (for the same reason) then the calibrated lights need to be in that format as well.

rob

[Buzz]

I think this discussion is drifting unintentionally.

To clarify - I am not mixing Maxim master files and PI.  I am taking original exposures captured by Maxim and processing them entirely in PI. All the master files I create within PI are 32-bit float. The calibrated lights are essentially black. These are the files I have linked to.

To check the original image captures were OK,  I took these same original exposures and created masters, calibrated and stacked entirely with Maxim and get an acceptable image.

If I read the last few posts correctly, if I am taking 16-bit Maxim captures and then do all my processing in PI, using 32-bit file formats, then that is not a root cause of my problem?

[Phil Leigh]

Yes but I actually downloaded your files and tried putting them into the script. The light frames integrate fine on their own but if you try adding in (say) the Master Bias it all fails spectacularly.

So, there is something very wrong with the format/content of the master calibration files.

There's only one way to sort this out. If you upload a handful of raw flat, bias and darks (not the masters!) then we can run it all though the script...

[pfile]

I think this discussion is drifting unintentionally.

...

If I read the last few posts correctly, if I am taking 16-bit Maxim captures and then do all my processing in PI, using 32-bit file formats, then that is not a root cause of my problem?

correct, that is not a root cause of your problem. something went wrong with the creation of the master bias/darks.

rob

[Harry page]

Hi

There is nothing wrong with your masters ( other than being noisey)

It is just that you lights are 32 bit ,   at a risk of sounding like a moaney old git import them into pixinsight in 16 bit

I took your lights back into maxim resaved them in 16 bit and then used these files in pixinsight and all worked just fine

Intergration image is on right

Harry

[Buzz]

I have loaded 10 raw bias, dark and flats in zip files.

http://www.toinfinityandbeyond.me.uk/astronomy/Files.html

I appreciate the help. I will look into the noise and measure it using the Craig Stark method- it is a new Trius camera and should have low read noise. I am using 50 bias files and maybe I should up it to 100.

thanks
Chris

[pfile]

well the problem is still that the lights are in 32-bit float format. if you use 16-bit lights i'm assuming it will all work fine per what harry did.

rob

[Phil Leigh]

yep - the bias/dark/flat files are fine and create normal Pi masters - the problem lies with the light frames which are (incorrectly) 32-bit and not 16-bit like the others.

[Harry page]

I have loaded 10 raw bias, dark and flats in zip files.

http://www.toinfinityandbeyond.me.uk/astronomy/Files.html

I appreciate the help. I will look into the noise and measure it using the Craig Stark method- it is a new Trius camera and should have low read noise. I am using 50 bias files and maybe I should up it to 100.

thanks
Chris

I use about 100 bias frames   

There does seem to be lot of vertical banding on your bias frames and darks that I can not see in your lights

The bias introduced these onto the lights 

perhaps this a property of the chip  , I will ask Terry at SX if this is normal

Regards

Harry

[Buzz]

Thanks for your support guys. It worked if I did it all in 16-bit integer. The doh moment was I had forgotten that my image files were IEEE float - I had saved them individually from Maxim rather than doing a sequence. A sequence would normally save 16-bit files. The batch pre-processor in PI only worked if I got Maxim to convert the light files to 16-bit. I tried converting the light file's format from 32-bit float to 16-bit integer using PI but the calibration did not work.

For others who may be following this thread, it has jumped around a bit, so in summary:

• Maxim can handle mixed format 16-bit and 32-bit files during its calibration and stacking routines.
  
• PI will not calibrate a 32-bit Maxim light file, only 16-bit but it can use 32-bit master files with 16-bit light frames in the preprocessing script

[jkmorse]

This whole thread sums up why I got fed up and am abandoning MaximDL.  There is just something wonky going on between PI and MaximDL (and MaximDL hasn't done an update for what seems like years now).  I get my MaximDL captures to work most of the time, then bang, all the same settings but something goes amiss, mostly with NB images.  I am sure its just a glitch in my system somewhere, but I am tired of the fight and have switched over to Sequence Generator Pro as my capture software (based on recommendations from Harry and others to a query I made a few weeks ago) and couldn't be happier.  Shot all new darks and bias frames in SGPro and PI handled them all without the slightest hitch.  Will test for the first time in the field this weekend, but expect the same much smoother connection between PI and SGPro than I ever had with PI and MaximDL. 

Sorry to rant, but MaximDL has been driving me crazy until I just couldn't take it anymore.

Thanks for letting me get that off my chest.

Jim