NGC7331 Processing Help - Image Artifacts

dusted996

Member
May 3, 2020
9
0
Hi,

Could anyone give me any pointers on why I'm getting the large scale and small scale noise in this image (& how to improve it)?

I'm guessing the large scale noise is light-pollution, but the small scale noise appeared after stacking in each individual channel. Cant seem to remove either sufficiently well with DBE.

Using an Atik Horizon Mono, LRGB filters, Bortle 6 zone.

Any help / tips on what to look into really appreciated - I've been following the pixinsight tutorials for RGB pre-processing (inc. subframe selection and drizzle integ etc). I've attached RGB combination & zoom plus the small-scale noise shown in green integration.

Cheers
 

Attachments

Last edited:

fredvanner

Well-known member
Apr 17, 2019
210
33
Wells, Somerset, UK
I've never seen anything like this in my images, but newbie (post below) appears to have a very similar problem. Since it seems to be an artefact that appears during stacking, it might help if you could post two or three consecutive single frames from the same filter (L might be the most informative).
 

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
The large-scale artifacts can be caused e.g. by light pollution or reflections or an unsuitable MasterFlat.

The small-scale artifacts are "walking noise" which is caused by not appropriate image calibration and not (or not sufficiently) dithering between light frames.

Unfortunately you didn't describe your approach of the preprocessing, the following informations are missing:
- did you apply ImageCalibration?
- was the image calibration performed on CFA data?
- which calibration frames were captured?
- how were the master calibration files produced?

You can check for yourself:
- is the MasterDark unclipped in the low range?
- are the calibrated light frames unclipped in the low range?
These informations would be valuable as well.

Bernd
 

fredvanner

Well-known member
Apr 17, 2019
210
33
Wells, Somerset, UK
I note that you used drizzle integration. I can imagine (but with no evidence) that some types of "systematically non-random" dithering might cause drizzle artefacts. Have you tried non-drizzled image integration? Do you get the same artefacts?
 

dusted996

Member
May 3, 2020
9
0
Thanks for the prompt replies - some more info:
  • Image Calibration - Yes - I've calibrated using Master Dark, Flat and Super Bias - I basically followed the tutorial here
  • Image Calibration was applied on CFA data - individual channel subframes
  • Calibration frames: Dark, Flat and Bias - all of which were combined together into masters.
  • Dark frames - combined different length dark images together into a master dark, which was used on all subframes; because the lights were taken across multiple evenings of varying exposure lengths and I didnt have exact matching dark exposures for each light.
  • Same artifacts without Drizzle? - yes unfortunately the walking noise appears in the pre and post drizzle integration frames.
Master dark unclipped in low-range? - Dont believe I did this (as per tutorial) but its possible I left this ticked by accident.
Calibrated light frames unclipped in low range - definitely not - just ran another integration to be sure and the artifacts remain.

I've added a few consecutive L frames (original un-calibrated / raw frames) to this post along with a GIF using Blink from green filter (post calibration) - 10 frames.
 

Attachments

fredvanner

Well-known member
Apr 17, 2019
210
33
Wells, Somerset, UK
I'm a bit puzzled. You say you are using an Atik mono camera with filters, but also that you were applying CFA calibration. Since CFA only applied to OSC colour cameras, these can't both be right. I get no artefacts aligning and integrating your three raw frames. I know this is only a short sample, but the effect is so strong I thought I might see something. Can you say what dither pattern you are using? (I confess I just leave it to PHD2, but it is more critical if you are going to use drizzle).
 

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
  • Image Calibration - Yes - I've calibrated using Master Dark, Flat and Super Bias - I basically followed the tutorial here
The Light Vortex tutorial about preprocessing says that the dark frames shall be calibrated with the superbias. This is bad advice because this approach involves the risk of clipping the dark frames (and hence the MasterDark). So deviating from this approach, the dark frames shall be simply integrated to the MasterDark.

  • Dark frames - combined different length dark images together into a master dark, which was used on all subframes; because the lights were taken across multiple evenings of varying exposure lengths and I didnt have exact matching dark exposures for each light.
Generally it is a bad idea to integrate dark frames of different exposure time together. Depending on the sensor (keyword: "amplifier glow") there are different possibilities to correctly calibrate your images. Please specify which sensor is utilized in your camera, is it a CMOS sensor, Panasonic MN34230?

Master dark unclipped in low-range? - Dont believe I did this (as per tutorial) but its possible I left this ticked by accident.
Calibrated light frames unclipped in low range - definitely not - just ran another integration to be sure and the artifacts remain.
No, this is a misunderstanding. I wanted you to check whether your available MasterDark (and also the calibrated light frames) are clipped in the low range. This can be done with two PixInsight processes: HistogramTransformation and Statistics. In the HistogramTransformation process, increase the horizontal zoom (to 40 - 100) and inspect the low range of the relevant images: is the peak clipped (= cut off) at the left side? In the Statistics process (important: the option 'Unclipped' has to be disabled, this is the default), check the value 'count (%)'. It should be near 100 %. The value (count (%) - 100 %) is the fraction of clipped pixels.

Bernd
 

dusted996

Member
May 3, 2020
9
0
@fredvanner - My misunderstanding on CFA - I have a Mono Atik Horizon (Panasonic MN34230 4/3" CMOS) camera using LRGB filters - I'll confess that I've never been brave enough to mess about with the default dithering settings in PHD. The Lum frames I attached were un-calibrated, so more data-points toward poor calibration I guess.

@bulrichl - Seems to all be pointing toward the way I processed the dark-frames - camera is indeed the CMOS Panasonic model you mentioned.

Yeh misunderstanding on the clipping front - just checked the master dark and count is at 88% - histogram also confirms there has been some clipping.

I've attached the master dark - understood from reading elsewhere the Amp Glow were the two large-scale artefacts on the left hand side?
 

Attachments

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
Yes, the MasterDark shows "amplifier glow" at the upper and lower left side. So ImageCalibration's option 'Optimize' in the 'Master Dark' section shall not be used for this sensor.

Please disclose all discrete exposure times that were used, then I can describe a procedure which probably will save these frames.

For the future, I recommend to determine the appropriate exposure time for an object first, then stick to this exposure time, and capture matching dark frames.

Bernd
 

fredvanner

Well-known member
Apr 17, 2019
210
33
Wells, Somerset, UK
At the risk of upsetting some enthusiasts...
Drizzle integration is a fascinating algorithm, and became indispensible for processing Hubble images; as a professional mathematician specialising in signal processing algorithms I find it really interesting, but...
... for most amateur astrophotographers it is an "icing on the cake" feature - squeezing a little bit of extra resolution out of a big enough stack of well-dithered subframes. I would advise getting your workflow robust and reliable with standard ImageIntegration before experimenting with drizzle.
 
  • Like
Reactions: fmeschia and pfile

dusted996

Member
May 3, 2020
9
0
@bulrichl- Optimize unchecked for Master Dark - noted for when I capture decent Dark Frames.

Exposure times used for light frames are below ( in future pick an exposure for a target and stick to it - noted & thanks so much - I have 15 hours of data you could help save... :
RGB: 60, 120, 180, 300
LUM: 120, 180 300

@fredvanner - noted maybe I should steer clear for the time being - I just found drizzle gets rid of lots of edge artefacts around stars which glare through in the final images, but understand its about getting the basics right first.
 

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
I propose the following trial:

1) If you have an insufficient number of dark frames with an exposure time of 300 s, capture some new in addition. Prepare a new MasterDark from these dark frames (no calibration, just integrate them) and save the result as "MD_300s.xisf".

2) Open the new MasterDark, MD_300s, and the MasterBias (or superbias), MB.

3) Use 3 PixelMath operations for the preparation of 3 additional MasterDarks: open the PixelMath process and input the following equation:

Code:
RGB/K: k*MD_300s+(1-k)*MB
In the 'Destination' section, enable option 'Create new image'.

In the 'Symbols' section, k has to be defined explicitely for each run. Use k=x/300 where x is the exposure time of the current group of light frames. So the following k values have to be used:

Code:
exp. time      k        Image Id
 60 s        0.2        MD_60s
120 s        0.4        MD_120s
180 s        0.6        MD_180s
Execute PixelMath 3 times with the specified k and Image Id, and save the results. In this way you end up with a total of 4 MasterDarks.

4) Calibrate the light frames in 4 groups according to their exposure time with the corresponding MasterDark (options 'Calibrate' and 'Optimize' disabled) and MasterFlat. No MasterBias or superbias must be used here.

This approach is not tested. Please let me know whether it worked.

Bernd
 
Last edited:

dusted996

Member
May 3, 2020
9
0
Thanks Bernd - Just gave it a quick try with the 61x120s Lights and 6x300sec Darks - realise I probably need to collect more dark data at 300s.

Using Trial Dark + Master Flat the walking noise appears to still be there. However when I use Trial Dark without Master Flat the noise appears to have gone. Seems maybe I've done something wrong generating / pre-processing the flat - I followed the same tutorial as above & info I read in Every Pixel Counts book.

Do you think collecting more Dark frames and correcting as per Trial would yield better results - or maybe I should try and live without the Flat & correct vignetting with DBE ?
 

Attachments

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
Using Trial Dark + Master Flat the walking noise appears to still be there. However when I use Trial Dark without Master Flat the noise appears to have gone.
No, not really. These two images are stetched differently: the STF Auto stretch is much more aggressive when the MasterFlat is also applied. So they are not comparable directly.

Seems maybe I've done something wrong generating / pre-processing the flat - I followed the same tutorial as above & info I read in Every Pixel Counts book.
It is true that the MasterFlat does not seem to match well. The flat field correction produced a bright round region in the center, surrounded by a darker ring and again a brightening in the most exterior regions of the image. Looking at the integration without flat field correction, my impression is: this is not normal vignetting (which should increase more and more when the distance to center is enlarged), it looks like a brightening of the central region by some straylight. Did you achieve a satisfying result of the flat field correction before with this specific scope/camera combination?

I'm sorry that the proposed trial did not seem to fix the issue. For further assistence I would need some of the original data (i.e. the new MasterDark, the MasterBias, the MasterFlat, and some of the light frames). Maybe that the application of CosmeticCorrection after ImageCalibration is necessary. If you are willing to upload the data to a filehoster and send the link, I can take a deeper look at it.

Bernd
 

dusted996

Member
May 3, 2020
9
0
No, not really. These two images are stretched differently
Rgr - good to know / learn.

Did you achieve a satisfying result of the flat field correction before with this specific scope/camera combination?
New scope / camera combo and this is first attempt - honestly I probably never really got it right with the last scope but results were good enough visually, so I ignored it. Just trying with the new gear to make some improvements throughout the process.

Thinking about it I did actually change the backfocus spacing half way through capturing the lights (consecutive nights) because I had it setup wrong - the flats were collected after changing the spacing.

Really appreciate the offer of further assistance - to save wasting your time I'll initially sort through the lights and make sure I am 100% sure the Flat frames are matching the equipment train / setup. I'll send over a link once uploaded.
 

dusted996

Member
May 3, 2020
9
0
Hi @bulrichl - I've uploaded some frames to dropbox and sent you a link to the folder.

Happy to send you the folder link @fredvanner if you have some time to investigate aswell.

Any extra insight into processing the frames would be appreciated.

Cheers!
 

bulrichl

PTeam Member
Nov 2, 2016
831
58
La Palma, Canary Islands
I took a look at the 1x1 binned data:

All light frames were captured at gain 1, offset 16, temperature -14.9 °C. The Lum frames were captured at 1x1 binning, exposure time 120 s. The red, green and blue filtered frames were captured at 2x2 binning, exposure time 60 s.

The superbias 1x1 has a mean of 271.0 ADU whereas the MasterDark 1x1 "MD_300s" has a mean of only 107.3 ADU. This doesn't match at all. Apparently the conditions for the capture of the underlying bias and dark frames were different. Generally it is crucial to capture light frames and all calibration frames with the same acquisition software, the same binning, the same camera gain and offset settings and the same sensor temperature. So it is not surprising that the "trial" MasterDarks which were created with PixelMath are not useful.

If you have used the same acquisition software for bias and dark frames as for the light frames (Artemis capture), you can look at the FITS header of the bias and dark frames and compare the conditions. Probably the bias and / or dark frames have to be repeated. Additionally, you'll need dark frames with binning 2x2.

Besides, the integration of the dark frames (300 s) was performed with Normalization set to 'Additive + scaling' and Pixel rejection algorithm set to 'None'. This is not correct. Like bias frames, the dark frames shall be integrated with Normalization set to 'None'. The Pixel rejection algorithm shall be chosen according to the number of frames. Pixel rejection is important for the integration of bias and dark frames in order to exclude cosmic ray artifacts.

The MasterFlat is way underexposed, the mean is 0.0019 in the range from 0.0 to 1.0. You should aim for a mean of about 0.5 in the flat frames (peak in about the middle of the histogram).

Bernd
 

dusted996

Member
May 3, 2020
9
0
Thanks Bernd - I'll work on the suggestions.

@fredvanner - link sent - clearly there are some issues with the way I've collected the calib frames which need solving but always happy to help out.

Thanks again both - really appreciated
 

fredvanner

Well-known member
Apr 17, 2019
210
33
Wells, Somerset, UK
1597692217330.jpeg
Integration of the R,G,B data you posted (I haven't included the L) using wbpp with your masters, followed by channel combine, ABE and PCC. I don't get any obvious artefacts (see crop).
1597692738922.jpeg