Subframes calibrating poorly

[brew]

I am having a weird problem with calibrating my images, and am hoping some expert out there can help me out.

The problem: Calibration of shorter exposures (say, 5 minutes or less) with bias, darks, and flats results in images with lots of zero values in the background areas. These images cannot be aligned by StarAlignment since it fails to find enough stars. As a result, the images are unusable.

I have been imaging for about 12 years, so I am not a total newbie . I do expect I am doing something subtly wrong, hence my hope that some clever individual can see the problem or suggest additional tests to determine the issue.

Clues:

• This happens with Narrowband images (Baader HAlpha, OIII, and SII), and occasionally with a Red filter (Astrodon Gen II?). Sometimes, in a sequence of 20 Red subs the first 5 or 6 calibrate fine, then the rest fail.
• I have two cameras in use. The SBig STF8300M has the problem. The older ST2000XM has never shown the problem.
• All images are binned 2x2 with the camera at -10 degC. The exception is the flats. I am using Sky flats, and here in Arizona it is often too hot at dusk for the camera to get down to -10. Flat subs might be at -5 to -7 degC, still trying to reach -10.
• The problem appears when I calibrate either in PixInsight or in Maxim/DL.
• Problematic Red subs show zero pixels interspersed with “regular” background pixels. The NB images tend to be completely zero in the background areas.
• 10 minute or longer subs do not exhibit the problem.
• Problem occurs with either a regular Bias master, or with a Superbias master.

Sample Files
I loaded sample files on Endor in the folder BrewCalibZero in the Forum Shared files.

Masters:
•   Superbias.xisf  - Made from 50 bias subs (bin 2x2).
•   flat-FILTER_HAlpha-BINNING_2.xisf, flat-FILTER_Oiii-BINNING_2.xisf, flat-FILTER_Red-BINNING_2.xisf - Made from 25 dusk sky flat subs.
•   dark-BINNING_2-EXPTIME_120.xisf - Made from 30 2 minute dark subs.

Sample Master Subs

I didn’t upload all of the subs, just two examples of each subframe type.

Sample lights

Sample subs to be calibrated are in folder UncalibLights. The corresponding calibrated sub is in the folder CalibLights.
•   RAW-NGC 6543-HAlpha-120-S001-R001-C030-001909 – single HAlpha sub which fails calibration
•   RAW-NGC 6543-Oiii-120-S001-R001-C001-004950 – single OIII sub which fails calibration
•   RAW-NGC 6543-Red-120-S001-R001-C010-023930 – single Red sub which fails calibration
•   RAW-NGC 6543-Red-120-S001-R001-C001-002153 – single Red sub that calibrates successfully.

Logs

•   20170406064332Bias.log – Running BPP to create the master bias file. This step was run separately in order to create the superbias from this master.
•   20170406064950DarksFlats.log – Using the superbias, created the Dark and Flat masters.
•   20170406070025Calib.log – Using the three masters, calibrate the HAlpha, OIII, and Red subframes.

Setup

• The imaging train is a Celestron Edge 11”, an Optec TCF-S focuser, then the STF8300. The STF8300 has the OAG and a filter wheel built in. I am in an 8 foot dome, mount is a Paramount MX.
  
  
• Using PixInsight 1.8.4.1195 64 bit under Windows 10 64bit.
  
• mages are collected in Maxim 6.06. I am an ACP user, using ACP’s Sky Flat routine to generate flats.

[pfile]

short narrowband images have almost 0 sky signal in the background. so what you've got there is just dark signal.

the pixel-by-pixel variation in the dark signal between your darks and the lights can be enough such that subtracting the dark results in a lot of negative pixel values, which get clamped to 0.

the solution is to add an output pedestal when calibrating the lights.

why such short narrowband exposures?

rob

[brew]

The why:

Occasionally I have a target where my typical 15-30 minute exposure is too bright - lots of the target is burned out.

For example, I am trying to do the Cats Eye planetary. The long exposures nicely show dim features around the nebula, but the nebula itself is burned out. A 2 minute exposure puts the nebula in a good range of values, but doesn't show any of the dim features. I will then do the HDRComposition to combine the two images.

So - why do I now encounter the zeros being cut off? It only happens on the 8300 - somehow the ST2000 never had an issue.


I am playing around with how to add a pedestal. I used PixelMath to add an offset. I am adding 0.01 to the raw image as my pedestal. The modified file seems to work, although I need to check what happens if the offset goes over 1.0. I am not renormalizing, or adding a PEDESTAL keyword.

I can manually run PM to add 0.01, then manually save the file for use in BPP.

The question is, how do I set up a process container to do this? I have a couple of hundred images in an Image container. I made a Process container and added the PM to change the file, but how do I get the Process container to save the file? I don't want to manually save hundreds of images to a new directory.

I see that ImageContainer has an output directory option, but that didn't seem to do anything. Certainly did not put the modified image in there, it just pops up on the screen. I tried typing &filename in the PM target file name, no luck. Somehow I want PM to save the new target file (or updated in place file)  in a new folder after modification.

This seems like a common thing to want to do, hopefully there is a simple option to make it happen:)

[brew]

Ah, got it to work. In ImageContainer, I need to specify &filename;_p.xsif as the Output Template. I also edit the PixelMath "source code" to turn P.showNewImage to false. This prevents PM from popping up a couple of hundred new images in their views.

Cool.

[pfile]

yes in ImageCalibration there is an option to add a pedestal. check under the "output files" section. i think that will prevent the clamping at calibration time rather than having to pre-pedestal the lights.

this has to do with the character of the dark signal in the sensor... i guess that accounts for why the ST2000 did not have a problem. it's likely that the older camera has much higher dark current and/or the dark current shot noise is simply lower.

rob