Noise problem most apparent with O3 - don't understand the cause (fully).

[ssa5]

Back in 2022 I had imaged IC1396 successfully for the first time (2nd year with a 2600MM). The finished image I did like, but I wanted to do more this past summer. This 2nd time I shot more subs, then used subframe selector w/filter equation I found here to get the best images (this after also doing Blink). I also dealt with a problem later on in 2022 where my focuser was causing differential flexure, so in 2023 I dropped my max exposure to 180s from 300s. So the totals are as follows:
2023: 93x180s=279 minutes
2022: 40x300s=200 minutes

The images below are linear with only NoiseXterminator applied at 0.40 for Denoise as this is first a process I always use, and second it highlights my problem. I want to say that my problem was simply underexposed, but then again I know and understand so little that I could very well be wrong. This problem also appears the most with my O3 filter, though have a couple targets where S2 showed this. HA & LRGB do not show this problem.

Weirder still, this past fall I imaged M42, and it went perfectly. I also did NGC 2238 on the same evening, and while I did not get the same amount of subs, it still seemed it should have been good enough. For M42 I had 48x180s for O3, and it turned out good. NGC2238 has 49x180s and O3 turned out just like the IC1396 image below for 2023. Problem really becomes apparent if I ignore NoiseXterminator, stretch, Pixelmath SHO or something and the image just looks awful due to the noise, but its more than that because the blending of colors makes the HA dominant regions look bad as well.

So my hopes is what is seen in the photo and what I describe can be easily diagnosed, kind of expecting a "take longer subs". But I need to ask nevertheless. It is disheartening to take a total of 153x3 minute subs, extracting the "best", stacking to find the image is garbage.

 

[cloudbait]

Back in 2022 I had imaged IC1396 successfully for the first time (2nd year with a 2600MM). The finished image I did like, but I wanted to do more this past summer. This 2nd time I shot more subs, then used subframe selector w/filter equation I found here to get the best images (this after also doing Blink). I also dealt with a problem later on in 2022 where my focuser was causing differential flexure, so in 2023 I dropped my max exposure to 180s from 300s. So the totals are as follows:
2023: 93x180s=279 minutes
2022: 40x300s=200 minutes

The images below are linear with only NoiseXterminator applied at 0.40 for Denoise as this is first a process I always use, and second it highlights my problem. I want to say that my problem was simply underexposed, but then again I know and understand so little that I could very well be wrong. This problem also appears the most with my O3 filter, though have a couple targets where S2 showed this. HA & LRGB do not show this problem.

Weirder still, this past fall I imaged M42, and it went perfectly. I also did NGC 2238 on the same evening, and while I did not get the same amount of subs, it still seemed it should have been good enough. For M42 I had 48x180s for O3, and it turned out good. NGC2238 has 49x180s and O3 turned out just like the IC1396 image below for 2023. Problem really becomes apparent if I ignore NoiseXterminator, stretch, Pixelmath SHO or something and the image just looks awful due to the noise, but its more than that because the blending of colors makes the HA dominant regions look bad as well.

So my hopes is what is seen in the photo and what I describe can be easily diagnosed, kind of expecting a "take longer subs". But I need to ask nevertheless. It is disheartening to take a total of 153x3 minute subs, extracting the "best", stacking to find the image is garbage.

View attachment 23021
View attachment 23022

My advice would be to not filter first with SFS and/or blink, and MOST DEFINITELY do not use NoiseX before you've done other processing. What is your full workflow to get to this stage?

 

[pfile]

the 2023 image looks weirdly posterized. what happens if you turn on 24-bit STF LUTs for that image?

 

[ssa5]

My advice would be to not filter first with SFS and/or blink, and MOST DEFINITELY do not use NoiseX before you've done other processing. What is your full workflow to get to this stage?

This is the expression I am using

Approval: Weight>80
Weighting: (20*(1-(FWHM-FWHMMin)/(FWHMMax-FWHMMin))) + (20*(1-(Eccentricity-EccentricityMin)/(EccentricityMax-EccentricityMin))) + (20*(PSFFlux-PSFFluxMin)/(PSFFluxMax-PSFFluxMin))+50

The image in the screenshots are not processed, except NoiseX was run to highlight the problem. However if I was to do normal workflow I would begin with Dynamic Background Extraction and then NoiseX (as I had read to run NoiseX as early as possible). No matter what I do the problem persists. Just looking at the raw image it is noise, unbearably bad.

I don't understand why would I want to not use blink to get rid of bad images? I usually always ran SFS to at least help me select my registration file for registration reference.

the 2023 image looks weirdly posterized. what happens if you turn on 24-bit STF LUTs for that image?

No change that I can see (assuming I am doing this correct, never used the 24-bit STF).

 

[cloudbait]

This is the expression I am using


The image in the screenshots are not processed, except NoiseX was run to highlight the problem. However if I was to do normal workflow I would begin with Dynamic Background Extraction and then NoiseX (as I had read to run NoiseX as early as possible). No matter what I do the problem persists. Just looking at the raw image it is noise, unbearably bad.

I don't understand why would I want to not use blink to get rid of bad images? I usually always ran SFS to at least help me select my registration file for registration reference.

No change that I can see (assuming I am doing this correct, never used the 24-bit STF).

Because "bad" images usually improve the final result. If I try to remove what I think are lower quality images, I end up with a higher noise result. A much better strategy is to weigh the frames and scale them during integration, which is what WBPP is all about. It will normally only reject completely unusable frames. If you have reasonable equipment and imaging sessions, you will see very few rejected frames.

For workflow, I meant from the beginning... so your calibration, registration, integration, etc.

 

[ssa5]

Because "bad" images usually improve the final result. If I try to remove what I think are lower quality images, I end up with a higher noise result. A much better strategy is to weigh the frames and scale them during integration, which is what WBPP is all about. It will normally only reject completely unusable frames. If you have reasonable equipment and imaging sessions, you will see very few rejected frames.

For workflow, I meant from the beginning... so your calibration, registration, integration, etc.

What you might call workflow would be this, visually inspect images (Blink) for clouds or obvious guiding errors, etc, what I would say are unusable. Then SFS to use the expression to copy into a new folder all files that are "approved". Then those files are added to WBP with Bias, Flats, and Lights (no Darks). Out of that I had 93 lights stacked using WBP to create the Master file.

NGC 2238 is another target where I had this problem. I had total of 50 subs, of which 49 got used to stack. M42 had total of 53 subs of which 49 were used with WBP.

With the comparison between 2022 & 2023s subs, the only difference there is I used shorter subs but more of them, but the same workflow process was used. This is a problem that like the example of M42 & NGC 2238 does not always appear, it is just some targets when stacked show up as what I would describe as pixelated. Why without knowing enough I want to say I don't have enough subs. But then I consider total integration time and 2023 has more than 2022, which is what has me for a loss.

 

[nico1038]

Can you share your 2023 (unprocessed) stacked file?
I agree with @pfile: it looks very much like posterization. And if clicking on the 24-bit STF button doesn't change anything then I'd say it is a calibration problem.

 

[cloudbait]

What you might call workflow would be this, visually inspect images (Blink) for clouds or obvious guiding errors, etc, what I would say are unusable. Then SFS to use the expression to copy into a new folder all files that are "approved". Then those files are added to WBP with Bias, Flats, and Lights (no Darks). Out of that I had 93 lights stacked using WBP to create the Master file.

NGC 2238 is another target where I had this problem. I had total of 50 subs, of which 49 got used to stack. M42 had total of 53 subs of which 49 were used with WBP.

With the comparison between 2022 & 2023s subs, the only difference there is I used shorter subs but more of them, but the same workflow process was used. This is a problem that like the example of M42 & NGC 2238 does not always appear, it is just some targets when stacked show up as what I would describe as pixelated. Why without knowing enough I want to say I don't have enough subs. But then I consider total integration time and 2023 has more than 2022, which is what has me for a loss.

So you aren't discarding enough frames to have significant impact on the final S/N. And I don't think with your camera the difference in sub times is having significant impact, either. So that leaves the possibility of something going wrong during calibration. Maybe look closely at all your calibration frames. I'd also compare a bias to a 120s and 300s dark. If they aren't identical except for noise, you probably need to be using darks.

 

[fredvanner]

I'd also compare a bias to a 120s and 300s dark. If they aren't identical except for noise, you probably need to be using darks.

The very recent "emergent" idea that you don't need darks for some cameras should be treated as a pragmatic issue. Some modern CMOS cameras do have very low dark current, which means that for short enough exposures you may get almost as accurate calibration with bias frames as with matched dark frames. This is likely to vary from camera to camera so, as @cloudbait says above, the right thing to do is actually compare you bias frames with dark frames of varying exposure. Unless the darks are identical to the bias frames (and in particular have identical median values), you should be using dark calibration.

As an aside, since I believe I have seen examples of misunderstanding, if you decide not to calibrate with dark frames you absolutely must calibrate with bias frames instead (the option is not "I don't need dark calibration", it is "I can use bias calibration instead of dark calibration").

 

[ssa5]

IC1396 2023: https://drive.google.com/file/d/1DQeSwFomesTrQawQISCcI2ZBFs2YcovO/view?usp=sharing
IC1396 2022: https://drive.google.com/file/d/1duRjvIdB_WWJ-yr3OCCzOEAE9FVIjk8o/view?usp=sharing

The very recent "emergent" idea that you don't need darks for some cameras should be treated as a pragmatic issue. Some modern CMOS cameras do have very low dark current, which means that for short enough exposures you may get almost as accurate calibration with bias frames as with matched dark frames. This is likely to vary from camera to camera so, as @cloudbait says above, the right thing to do is actually compare you bias frames with dark frames of varying exposure. Unless the darks are identical to the bias frames (and in particular have identical median values), you should be using dark calibration.

As an aside, since I believe I have seen examples of misunderstanding, if you decide not to calibrate with dark frames you absolutely must calibrate with bias frames instead (the option is not "I don't need dark calibration", it is "I can use bias calibration instead of dark calibration").

I was aware in 2021 a lot of comments about the 2600 & 6200mm being so good that you did not need darks. I did create some darks, even created a master dark file, but when I went to use WBP with a dark the outcome was not good. I do not remember now what happened, I just know that I do never use them and up until now have never thought to use them.

I do use Bias always. Tomorrow as a matter of fact I am receiving a whole bunch of new equipment; Askar FRA300, ZWO 2600MC, and WO Flat68 to replace my existing Flat/Reducer. I am going to have to bring my current WO 91MM indoors, and had thought to create all new master flats, bias, and even darks with different combinations; i.e. flats for OSC with 91mm & 60mm, new Bias for MC and redo for MM, etc.

So you aren't discarding enough frames to have significant impact on the final S/N. And I don't think with your camera the difference in sub times is having significant impact, either. So that leaves the possibility of something going wrong during calibration. Maybe look closely at all your calibration frames. I'd also compare a bias to a 120s and 300s dark. If they aren't identical except for noise, you probably need to be using darks.

This does seem like a logical outcome. It is strange though this problem appears most with O3 filter (but not with S2). Back in 2022 I had the stupid idea of trying a 4 panel mosaic of IC1805. This is where problem of differential flexure introduced itself (most likely cause is my focuser), and solution was to drop from 5-7 min subs in favor of 3 min subs. I tried again in fall 2023, and this is where my problem first began. The 03 filter was one I just could not use. So a 4 panel mosiac became several thousand useless files (well I can still try to create using LRGB files and enhance with HA).

 

[cloudbait]

IC1396 2023: https://drive.google.com/file/d/1DQeSwFomesTrQawQISCcI2ZBFs2YcovO/view?usp=sharing
IC1396 2022: https://drive.google.com/file/d/1duRjvIdB_WWJ-yr3OCCzOEAE9FVIjk8o/view?usp=sharing



I was aware in 2021 a lot of comments about the 2600 & 6200mm being so good that you did not need darks. I did create some darks, even created a master dark file, but when I went to use WBP with a dark the outcome was not good. I do not remember now what happened, I just know that I do never use them and up until now have never thought to use them.

I do use Bias always. Tomorrow as a matter of fact I am receiving a whole bunch of new equipment; Askar FRA300, ZWO 2600MC, and WO Flat68 to replace my existing Flat/Reducer. I am going to have to bring my current WO 91MM indoors, and had thought to create all new master flats, bias, and even darks with different combinations; i.e. flats for OSC with 91mm & 60mm, new Bias for MC and redo for MM, etc.


This does seem like a logical outcome. It is strange though this problem appears most with O3 filter (but not with S2). Back in 2022 I had the stupid idea of trying a 4 panel mosaic of IC1805. This is where problem of differential flexure introduced itself (most likely cause is my focuser), and solution was to drop from 5-7 min subs in favor of 3 min subs. I tried again in fall 2023, and this is where my problem first began. The 03 filter was one I just could not use. So a 4 panel mosiac became several thousand useless files (well I can still try to create using LRGB files and enhance with HA).

So... I looked at your two files. I'm not seeing any significant difference in noise between the two. In fact, if it's not just averted imagination, I might argue that the 2023 image has slightly better S/N.

 

[nico1038]

Both images are very similar in my opinion.

If I apply noisxterminator at 0.4 as you do, then I can see the posterization in the 2023 image. But it is not real. It's only a visualization problem.
If you click on the 24-bit STF button it disappears.

 

[ssa5]

Both images are very similar in my opinion.

If I apply noisxterminator at 0.4 as you do, then I can see the posterization in the 2023 image. But it is not real. It's only a visualization problem.
If you click on the 24-bit STF button it disappears.

You are correct, and doing the 24-bit STF was the trick. After seeing this I went back to NGC2238 and was able to bring HA, O3, & S2 to non-linear mode finally, and the image looks 10x better since I was able to successfully (in my view) run NoiseXterminator multiple times.

While the artifacts or posterization may have been visual, for me it prevented running NoiseX until I was satisfied the image was clean enough. Now the gas clouds look much smoother, and the blending of the 3 'colors' looks immensely better.

Thanks to everyone

 

[cloudbait]

You are correct, and doing the 24-bit STF was the trick. After seeing this I went back to NGC2238 and was able to bring HA, O3, & S2 to non-linear mode finally, and the image looks 10x better since I was able to successfully (in my view) run NoiseXterminator multiple times.

While the artifacts or posterization may have been visual, for me it prevented running NoiseX until I was satisfied the image was clean enough. Now the gas clouds look much smoother, and the blending of the 3 'colors' looks immensely better.

Thanks to everyone

My take here... to be taken or left. There is no such thing as noise removal. All we can do is mask it, and always at the expense of losing real information. I would seldom apply any noise "reduction" tools to my individual channels. (And in PI, it is often important to preserve PSFs as accurately as possible, and noise reduction alters those.) I think you want to use noise reduction just once, after you've combined all your different channels and finished most of your other processing. I find it works best after I've stretched the data. Usually the very last thing I do. And that's the point where a tool like NoiseX has the most information to work with, and therefore can hide the most noise with the least damage to the real signal.

 

[ssa5]

My take here... to be taken or left. There is no such thing as noise removal. All we can do is mask it, and always at the expense of losing real information. I would seldom apply any noise "reduction" tools to my individual channels. (And in PI, it is often important to preserve PSFs as accurately as possible, and noise reduction alters those.) I think you want to use noise reduction just once, after you've combined all your different channels and finished most of your other processing. I find it works best after I've stretched the data. Usually the very last thing I do. And that's the point where a tool like NoiseX has the most information to work with, and therefore can hide the most noise with the least damage to the real signal.

I agree with your assessment of noise removal. I had originally saw somewhere (don't know where) that NoiseX should be "used as early as possible", so I had put that in my notes or workflow to do just after DBE, and before even going non-linear. Yesterday I decided to do a Google search and sure enough found the complete opposite (and what you said about doing it towards the end of processing).

Could have been a post at CloudyNights or some Youtuber's video, but one certainty is that I think I had NoiseX placed completely wrong in my workflow. I did redo NGC2238 last night, and was happy enough to have shared the outcome with some family and friends.

Downside to all this? Now I have no excuse but to revisit my 4 panel mosaic attempt I did last fall (to which I have to process L,RGB,HA,O3, & S2)

 

[chris.bailey]

OIII subs on objects with little OIII signal will always seem noisy. Back in CCD days 20 or 30 minute subs and a lot of them was normal. 3 hours to get decent S/N is not a lot, 8 hours would start to look different.

 

[Scott Badger]

I agree with your assessment of noise removal. I had originally saw somewhere (don't know where) that NoiseX should be "used as early as possible", so I had put that in my notes or workflow to do just after DBE, and before even going non-linear.

I’ve seen noise reduction before stretching recommended so as not to stretch the noise, making it worse…..but I agree with comments above that NR dramatically alters linear data. Just do an auto-stretch on a linear image after NX to see….. That said, if I see noise really ramping up while stretching, I’ll backtrack to just after the initial stretch, apply a modest NX (.3 or .4), and move on.

Cheers,
Scott