Flats working in reverse in Batch preprocessing....

[cmarcus]

Not sure what is happening here, but on running batch preprocessing my flats seem to be adding errors to the lights and not subtracting them, so the calibrated files are worse than the original ones...Hopefully this attached screenshot of the two side by side gives the idea - and yes the round light areas correspond to dark areas on the flat files.
Not had this prob before, so presumably I've got some incredibly simple setting wrong...
TIA for any input.

[Niall Saunders]

Hi,

Are you using Biases, Darks and Flats (and perhaps FlatDarks) in your BatchPreprocess script?

Or, are you creating MasterBias, MasterDark, MasterFlat (and MasterFlatDark) frames prior to running your BatchPreprocess script?

What are the settings that you are using within the BatchPreprocess script?

Have you tried a simple (!!) PixelMath expression to 'calibrate' a single Light frame? (The expression should be of the following form (assuming that you open a random Bias, Dark, Flat, FlatDark and Light frame within a PixInsight workspace):

Code: [Select]

( Light - (Dark + Bias) ) / ( Flat - (FlatDark + Bias) )
Let us know a little bit more about your issues- someone might then spot the problem quite easily.

[cmarcus]

Thanks for your reply.

I am using Biases, Darks and Flats. No flatdarks.
I'm not creating Masters
My settings are as per attached screenshot
Applying this Pixel math: ( Light - (dark + Bias) ) / ( Flat - Bias ) (not having Flat dark, so I'm not sure if I'm getting the formula right) I still get the bad result.

This is all a bit strange, because I have not had this issue before....

[msmythers]

You might want to look at your flats and maybe your other calibration files with the Blink tool if you haven't already. If you have a single or multiple bad cal files you will see them with blink.



Mike

[cmarcus]

Thanks msmythers,
Just checked them all, nothing looking strange there - it would have surprised me if that had been the case (but good practice to check) because I have used these calib files before...

[Niall Saunders]

If your Flats are fairly short-exposure frames then you may not have that much'thermal' noise, and so you might not need a FlatDark.

You can check this by creating a set of FlatDarks (which can be done at any time) and then subtracting a MasterBias from them (by way of a calibration process or PixelMath). What you would be left with is a set of calibrated FlatDarks that should all have very similar Image Statistics (Max, Min, Mean, StDev, etc.). You caould even then create a MasterFlatDark (using ImageIntegration), and use this to calibrate all of your Flats. Finally, you could then ImageIntegrate your calibrated Flats to give you a single MasterFlat - which would then be the denominator in the PixelMath expression.

You need to break down the process into smaller sections (this is the only disadvantage of the BatchPreprocess script - you have to be 'clever' if things have 'gone wrong')

[Nocturnal]

Hi,

You need to subtract the pedestal (bias) from your flats using flatdarks or bias if your dark current is negligible for the exposure length. Similarly you need to subtract dark from your lights before flattening. Leave out either step and flattening will result in what you get.

In a few words:

(1) pixel = (light  - dark) / norm(flat - flatdark)

For my QHY8 there is no appreciable dark current so it translated into:

(2) pixel = (light - bias) / norm(flat - bias)

Your situation may be different but formula (1) needs to be satisfied.

[cmarcus]

Thanks Niall and Nocturnal
I am now, however, led to believe that it might just be that I need some new flats... This is a remote set up and I did not do new flats after some work was done on the equipment and the person on site says I need new flats.
All that said, I am aware that I need to be a bit more sophisticated than just using a basic prebatch processing....
I'll keep you informed on any progress.
many thanks once again.

[Niall Saunders]

One piece of advice regarding Flats - they have a very short lifespan. I (nearly) always take a set of Flats at the end of every imaging session.

Remember, dust is not 'static'. Particles come and go, and you need to try and record the most accurate 'dust particle contamination' data that will successfully calibrate your captured Lights. The same can also be said for minor changes in vignetting that may occur if OTA focus has to be adjusted.

I realise that your observatory is 'remote' - but, can you think of a way of capturing Flats at the end (or start) of an imaging session? It might help.

[Nocturnal]

While it is true that dust can move we should not confuse matters right now. First get the flats working properly, then worry about details like the longevity of flats. I have taken dozens of images with the same set of flats without ill effects. I was careful to keep my sensor and filters clean. I've also read of people who feel flats need to be reshot because you use a different focus position each session. You only use a different focus position if you don't focus correctly. As long as you use the same imaging train including all the bits and bobs that go between scope and camera there is only 1 correct focus position for astro work. If your optics are clean you can re-use your flats provided you have matching dark-flats (flat darks whatever).

[Niall Saunders]

I have definite issues with dust particles changing position, even suddenly appearing or disappearing between sessions. However, I attribute that to the fact that I am using an 'open-tube' OTA (as opposed to the likes of a Schmitt-Cassegrain or Maksutov-Cassegrain).

So, I feel that the extra effort to acquire Flats at the end of every imaging session is worthwhile. But, as always, different folk will encounter different issues.

I can agree with you about focus positions and Flats though - all other things being equal, then an optical train returned to the same focal position should require the same Flats for calibration purposes. But, if you are doing things like rotating filters in a filter-wheel, I would not be surprised to see dust particles move on the filters during that process. To me, that would require renewed Flats.

[Nocturnal]

My point was that a discussion on such details detracts from the problem we are trying to solve here  for the OP

While on the topic though it is my aim to get all the dust off my filters and keep them that way by leaving them mounted for weeks or even months. I can see how with your scope that could be problematic. In my case it is feasible