Blotchy background (with ASI1600MC-C Pro) camera

[iwoodward]

I'm getting a blotchy background in my images. This image of Helix was taken with my ASI1600MC-C Pro, set at gain 75 and offset 12 and cooled to minus 16. It was stacked in Pixinsight from 256 x 1 minute subs, each calibrated with a dark master and flat master.

The attached blue image is the original stack, and the attached blotchy image is after the DBE.  Also attached is a single sub.

The subs were debayered in PI using the GRGB pattern. From reading this forum and the Zwo forum I think this is correct and the colours of the Helix seem to confirm it but I'm wondering if the blotchiness is nevertheless something to do with the debayering process.

I've tried to work out what is happening with previous blotchy images, and I've found that the blotchiness was much worse when I was unthinkingly using a high sampling density in DBE.  This time I've only used a handful of samples across the image, so I don't think it's a DBE artifact (but I could be wrong).

I can't see obvious signs of blotchiness in individual debayered subs (a copy of one is attached) but perhaps it's there and only becomes obvious from the stacking.

I'd be grateful if anyone could suggest a cause and solution.

[iwoodward]

To explore what might be happening, I've attached the stacked image as it first comes out of PixInsight's drizzle integration. This has a bright blue colour cast across the whole image.  I then applied PI's SCNR process to remove blue and got the other attached image, which now shows no sign of blotchiness (although obviously the colour of the Helix nebula is now wrong because of the removal of blue).  The disappearance of the blotchiness suggests that it may be entirely due to blue noise.

Could there be something about my camera, or some setting, that could be causing blue noise? Or could a wrong debayer or integration setting in PI be adding blue noise?

[John Massey]

Did you perform a back ground neutralization and color calibration before DBE?

[sharkmelley]

The blue background is very odd.  What does a single debayered frame look like without any processing?  Does it have the same blue background?

Mark

[iwoodward]

I didn't apply background neutralisation or colour calibration before DBE but I've now tried that and although BN removes the blue the blotchiness is still there afterwards.

The blue is there in individual subs, although not as intense.  A bluey sub is attached and also the same sub after blue removal with SCNR.

[iwoodward]

Further investigations - I've been wondering whether the problem might go right back to my calibration files, so I've been having a look at my master flat.  I don't know if this is a legitimate thing to do or not but I debayered it to have a look at its histogram.  I've attached the result.

The debayered master flat is very yellow, and the blue histogram sits well to the left of the red and green (see attached).  If I'm using this master to calibrate my lights, this would mean less blue is subtracted than red and green, which would leave the excess blue that I'm seeing in my calibrated lights.

If this is a legitimate argument, it means I have to somehow balance my histogram in my master flat. 

Ideally, I would create a balanced flat from the outset - maybe my calibration and integration settings when generating the master flat are wrong?  I followed the PI guide (http://pixinsight.com/tutorials/master-frames/index.html) but perhaps the settings have to be different for my Zwo ASI1600MC-C Pro camera?

Less ideally, perhaps I could debayer my master flat, manually balance the histograms, and then somehow undebayer the balanced image to take it back to grey.  Is this possible?

[MikeOates]

Less ideally, perhaps I could debayer my master flat, manually balance the histograms, and then somehow undebayer the balanced image to take it back to grey.  Is this possible?

Yes and the tools for this are SplitCFA and MergeCFA. After the Split, use LinearFIT to balance all 4 (yes 4) files, then merge them again.

This is something I have just started to do, like yesterday.

Note, you may get an error message if you are starting with an odd number of rows in your image, I get this with my DSLR so I have to use Crop to remove the bottom row before using SplitCFA.

Mike

[bulrichl]

You are right, the strong blue color cast in the calibrated light frames is caused by the yellow light source that you are using when taking the flat frames. However, in the flat field correction the MasterFlat is not subtracted from the light frames, but the light frames are divided by the MasterFlat. Thus it is not a good idea to change the black point of the red and green channels - the flat field correction for these channels would not work appropriately any more.

Conclusion: the color balancing has to be done after ImageCalibration is finished.

Bernd

[iwoodward]

Thanks Mike and Bernd.  I think I'll remake my flats, this time using the T-shirt method, so that I have a natural light balance from the start.  I now realise that the LED light panel I've been using in my light box is the culprit - it looks white to my eyes but yellow to my camera.

Ian

[pfile]

it's not super important that an OSC flat is balanced. the main thing is just making sure that one (or more) of the channels is not underexposed because the brightest channel limited your exposure. and of course the underexposure is only a problem if you don't make enough flat subs - probably even the most underexposed flat will swamp the read noise of the sensor, so it's really just down to having enough subs that the SNR of the weakest channel of the flat is acceptable.

as Bernd says, you need to color calibrate after calibration anyway, so whatever color casts you have in your calibrated lights are not so important.

think of it this way: if you were using a mono camera with filters, would you stress out about the master flat ADU values, trying to get them to a level that leaves the calibrated images balanced properly when combined into an RGB image? you wouldn't. this is the same thing but you're just doing 3 flats at once...

rob

[iwoodward]

Thanks Rob - I appreciate your comments about it not being critical to have a balanced flat.  Before I read your post, however, I'd gone off and manually balanced my master flat anyway and I'm reporting here on the results.  I also think that I've solved the blotchiness problem in my stacked image, and report back here on that too in case it helps others.

Firstly, I’ve got rid of the blue cast that I was getting with the initial stacked image.  I worked out that my LED light pad that I was using in my light box for flats must be a “warm white” rather than a “daylight” spectrum, ie. it’s quite yellow to the camera.  So, blue was under-represented in my master flat and when that flat was used to calibrate my lights, less blue was removed than red and green, hence the blue cast in the stacked image.

To fix this, I debayered the greyscale master flat and then manually balanced the RGB histograms and then I converted that back to greyscale for use as a master.  I've attached screenshots showing the before and after colours and histograms of the master flat (both were before conversion back to greyscale).

When I restacked my lights using this balanced flat, the stacked image had no blue cast.  I've attached images of the original blue cast stack and the new stack resulting from using the balanced master flat.

Finally, (and this is why I started all this investigation!) I think I’ve now solved the blotchy background problem.  I’ve experimented with the scale settings in the Local Normalisation step used during integration.  By raising the scale from the default 128 to 512 I seem to have removed the blotchiness.  (I don’t actually understand what the scale does BTW!)

I think the blotchiness was chromatic noise inherent in OSC cameras like mine (a Zwo ASI1600MC-C Pro) and it just happened that the default scale setting was clumping and exaggerating that noise rather than reducing it.  The 512 scale seems to be best to reduce it.

The super-stretched just-stacked images attached show the comparisons between what I was getting with the default 128 scale and what I now get from the 512 scale.

The new image gives me something much better to work with.

[pfile]

how did you balance the master flat? if thru any other means than multiplication it's a bad idea.

normalization is the process of making frames 'statistically compatible' with one another (meaning to have similar signal and background) so that during integration they can be compared against one another to reject outlying pixels. ordinarily this is done globally like the linear fit process does - every pixel in the image is multiplied by / added to by the exact same amount. if you have LP gradients, etc, then those are preserved and of course those gradients can throw off the statistics of the image.

local normalization is a normalization process that attempts to do the normalization in sub-segments of the target image. the scale value describes the pixel radius that it looks at (actually it's probably a wavelet scale like everything else in PI...) if the scale is set too low, it can be fooled by structures in the images like star halos and actual signal.

here is where i'll lament that someone out there put LN into a tutorial and everyone is just now doing it blindly without understanding that it's somewhat of an advanced tool and requires care when using it. it's not really appropriate for new people following tutorials step-by-step using default parameters because it usually requires fine-tuning as you found.

rob

[iwoodward]

Thanks again Rob - I naively balanced the master flat simply by using Histogram Transformation and manually moving the blue histogram to the right to sit with the red and green.  I take your advice that this is a bad idea.

I was doing it as part of trying to get rid of the background blotchiness but that was solved by changing the scale factor in Local Normalisation so nothing to do with the imbalance of the flat.  I'll now just live with the imbalance and remove the blue cast with background extraction unless/until I make a new one with a balanced light source (eg. T-shirt method).

Ian

[pfile]

right, the HT tool applies a non-linear transformation to the data so it's not a good idea - some parts of your flat will be brightened more than others and will no longer accurately represent the optical path of your system.

rob