Newb PI Convert - Noise Assessment?

[f11]

Coming from about six different pieces of software to process my images, PI is a breath of fresh air as a single-stop shop to do it all.  But like many others have found, the learning curve is pretty steep and I've reached a point where I'm floundering.

I use PI BatchPreprocessing to create the calibration frames, calibrate and register the frames,  then use ImageIntegration to integrate the light frame.  I'm working only with luminance data to keep things as simple as I can for now.

My problem: my integrated images seem AWFULLY noisy.  I've  been watching Harry's videos (hey, THANKS man - I was one of those noisy "PI is too hard to figure out" and "where's the user manual" users when I first bought PI, and basically abandoned it for a couple years... your videos brought me back!) and Keller's PI series, and understand the workflow and which tools to be used to reduce noise.  But my images look NOTHING like those of other PI users.  Its a little frustrating - and its obviously something I'm missing.

As background, I typically image "away" from the city glow and as high in elevation as targets allow, have a -25C setpoint in my SBIG camera for darks and lights (ambient is typically 0C or colder), and use an EL panel to create flats with mean of ~30000 counts each x 20.  My system has an image scale of 0.68 asp bin 1x1, if that matters.

Could someone take a look at the attached integrated FITS image of NGC2841 (no other processing done) and tell me if the noise levels are typical, or unusually high?  Its a composite of 10 x 600s frames.  Is it possible that my calibration frames aren't doing their job?  Or is it just that I'm imaging from the edge of a city with a million residents and the sky noise is totally overwhelming my data?

Link below is to a Dropbox location, file is about 7.2MB.  Any suggestions or advice is gratefully appreciated.

Rod

http://db.tt/xY4vTR9F

[Josh Lake]

Hmm, I took a look and it does seem noisier than expected, especially at 10 x 600s. I'm wondering if it's just a function of the harsh automatic screen stretch with STF.

What do the calibrated masters in other programs look like when opened in PI? The same or better?

[f11]

Not sure what you mean - "What do the calibrated masters in other programs look like when opened in PI?"

Are you asking what master bias, master dark and master flat frames generated in other programs (CCDStack2 for example) look like when opened in PI?  I've never checked - if that's what you mean.  I'll have to do that with this specific set of calibration frames, and see what comes out.  But from memory, I believe (in general) that master calibration frames generated in PI are "cleaner" than those generated in CCDStack2 (just as an example), but I've never done a deliberate comparison.

Rod

[sreilly]

Have you looked at each individual image after calibration? Usually I cull the best images and only use those for the master frame. Have you checked each and used a method to decide which is the best data?

[Carlos Milovic]

Hi Rod. How many bias and darks do you take?

[f11]

Steve, all raw light frames in this case look pretty much the same.  When I load them into CCDStack2, the Image Manager reports the FWHM of the 10 frames to range from 3.24 to 4.39. Other than visual appearance of a sub (focus, clouds, tracking errors, etc), I usually base my include go/no-go decision on the FWHM reading in CCDS2 - if a frame is significantly different from the rest, its removed.  If I load PI calibrated lights into CCDS2, the FWHM data is replaced with "9999" so I can't assess if that measure has been affected, but otherwise all images look pretty much the same.

Carlos, I generally take 30 bias frames and 20 dark frames to make my master frames.  Dark frame sets are taken at each of 1x1, 2x2, and 3x3 binning; at -25C temperatures for winter imaging, at -20C for spring and fall imaging; I take the bias and flat frames at the same temp as the darks and lights, whether necessary or not.  The bias frames are usually taken just before starting a dark frame set.

Thanks for your considered help with this - really appreciated!
Rod

[Philip de Louraille]

I looked at the image with STF and you can clearly see several rows of data (mid bottom) that have dark noise. I am surprised they were not removed.

[Alejandro Tombolini]

Hi Rod, I made a try with your image.

First a DynamicBackgroundExtraction being careful with stars.



As there are several rows of data that have dark noise as Philip said, I correct a couple of them with CosmeticCorrection module. With more patience you can correct all of them.




Then generate a PSF to be use as external PSF in Deconvolution tool and a starmask to protect to cores of big stars during Deconvolution.




Apply Deconvolution. The change es very little in the cores of stars.



HT to clip and strech the image



HDRMultiscaleTransformation to achieve more details in the bright parts of the gallaxy, protecting the core with starmask.



Thanks to Gerald, now we can use this nice formula in pixelmath to correct dark pixels in the background replacing its value for the median.



Build a rangemask to protect background during ATWT



Apply ATWT to increase contrast in the galaxy.



New mask to be use with HistogramTransformation. Duplicate the image, invert, and apply HT to increase contrast between galaxy and background. Finally smoothing eliminating the four first layers.



Clip protecting the galaxy



Noise reduction with ACDNR protecting the galaxy with the mask included in the tool.



Finally HistogramTransformation to clip and decrese a little the background bright, protecting the galaxy with the previous mask. You can decrease the background even more, it will depend on your taste or the monitor you are using. In my case I still see it a little bit bright.



Final result.

[Philip de Louraille]

(Hand clapping)^2  !!!
Very nice flow very well documented!

[f11]

Holy Crap !!         

Um, sorry, Alejandro - that was rude of me.

I'm sitting here with my jaw hanging down... thank you VERY MUCH for all that work... I wasn't expecting anyone to actually process my image.  I was sorta hoping people could tell me if my raw data and calibration files were leaving me with enough "meat" to process and get a decent final image.  But your demo makes it clear that there is a lot that can be done with my data, noise as it is, to arrive at a reasonable result.  THANK you, sir - most generous of you!

The end result is still fairly noisy compared to most images I see here and elsewhere, but WOW - that result is WAY better than anything I was coming up with.  My site is a long way from being an ideal astro-imaging environment, and I was starting to think that I'd have to relocate it to a darker sky area... which still isn't a bad idea (although a daunting project looking at it from scratch!).

My take away here, aside from the fact that I have a LOT to learn about PI, is that 1) while noisy, my images are not a total write-off; and 2) if all my images are this noisy after being calibrated, I might follow the same sequence of steps you did to get a similar result.

May I ask how you figured out what steps to take and in which order, or is this just pure experience from working on a zillion images?  That is, is there an approach you always take to processing that tackles issues in a certain order?  Or is it as straight forward as what I see:

- DBE
- CosCorr to fix dark noise (does this mean my darks aren't doing their job completely?  perhaps I have to amp up the rejection during master creation?)
- PSF for use in DeConVol
- StarMask to protect bright objects
- DeConVol
- HistTrans
---> no longer linear
- HDRMST (using StarMask to protect bright cores)
- PixelMath to replace background dark pixels with median values
- RangeSelection to build range mask for use with ATWT
- ATWT#1 with mask to increase contrast in galaxies
- copy/invert to use as mask, use HistTran to increase contrast between galaxies and background in mask
- ATWT#2 to smooth mask --> how did you decide to remove the first four layers?? Is this just more experience?
- apply mask to image to protect galaxy, use HistTran to clip black end
- ACDNR applied with in-tool mask
- final HistTrans to clip background brightness

On the surface with this example, it appears your approach is: take out background noise; fix dark pixel noise; DeconVol; convert to non-linear; increase detail in bright objects; more dark pixel repair; increase contrast within galaxies; increase contrast between galaxy and background; black clip; final noise reduction; and final black clipping adjustment.

I'm sorta speechless... thank you again for your help.  I've got some SERIOUS homework to do here.

Oh yeah - back to my original issue - assuming this FITS file is representative of all my calibrated images, do I have an unusual noise problem?
Rod

[Josh Lake]

Yes, to answer your question, I think your data seems unusually noisy given your exposure times, number of calibration frames, etc.

What I was wondering, way back in the beginning of the thread (!), was whether you had opened the frames you got from other programs (the equivalent master to this one done with BPP) in PixInsight. If so, do they look the same when STF stretched? I'm trying to narrow down the workflow to see if it's something that you could change or fix or whether the noise is inherent in your data.

[Alejandro Tombolini]

Thank you Rod for your nice words, it is a pleasure if it is helpful for you.

May I ask how you figured out what steps to take and in which order, or is this just pure experience from working on a zillion images?  That is, is there an approach you always take to processing that tackles issues in a certain order?  Or is it as straight forward as what I see:

There are many discutions about that, but in general you have to consider that there are not recipes to follow, only general ideas regarding process to be done when the image is linear an non linear.
I always start trying to let the image as good as possible before stretching, because it is the key to recover faint details in the background.

Oh yeah - back to my original issue - assuming this FITS file is representative of all my calibrated images, do I have an unusual noise problem?

I agree that the picture is noisy, but not discouraged. Look at one of my tipical raw data, lot of hot pixels, gradients, etc. I think that each of us have different issues that can be solved to a greater or lesser extent with the different tools in PixInsight.

Saludos,
Alejandro.

[f11]

jlake, I'm not ignoring you... I took the same raw frames processed in PI to create the FITS image I posted, and processed them in CCDStack2 to generate a calibrated integrated master light frame.

Just for the heck of it, I subtracted the CCDS2 generated dark from the PI generated dark using PixelMath - and was surprised to see the result was a frame with structure, not pure black or white.  Running the cursor over the CCDS2 master dark, k varied from roughly 0.0012 to 0.0017, while for the PI master dark k varied from roughly 0.0168 to 0.0172 - a difference of magnitude 10!  In the difference frame, k varied from 0.0154 to 0.0158 plus or minus.

While doing that, I noticed that each of the last 4 of 10 raw light frames was darker than the last, so I did a second master light frame using just the first 6 raw light frames.  I realize that the normalization process in CCDS2 should take care of that difference, but I figured it wouldn't hurt to see if it made a difference.  It didn't seem to at first glance, and I didn't pursue it further.

Anyway, the calibrated and integrated lights from the two programs looked similar, but only on the surface. Running the cursor around the PIMasterLight k is about 0.32, while the CCDS2MasterLight k is about 0.13.  Usiing PixelMath again, the difference frame (subtracting the CCDS2ML from the PIML) contained a fair amount of data, with k measuring about 0.18 as expected.  Interestingly, the columns of vertical dark pixels noted in the PIML show up in the difference frame quite clearly. I assume this means the CCDS2 master dark removed these column pixels while the PI master dark didn't.  This could be user error, though (hey, its possible - I'm up to my armpits in files).

I've included DropBox links to the two new files: the CCDS2 cal'd and integ'd master light ("Mean NGC2841 Lum 1x1 DynCrop Reg.fit"), and the (PIML-CCDS2ML) difference frame, in FITS format if you want to see them:

CCDS2ML Frame: http://db.tt/tC3iOFcZ

Diff Frame: http://db.tt/ilGkYS7C


Alejandro, okay I understand... each image processing is a unique sequence of steps depending on the quality of the image and the problems it might have.  Fair enough.  And thanks again... that was quite a lesson you gave me! 

[bitli]

Interesting thread. The image is noisy IMHO. The question is where does not noise come from. May be it would help to look at parts of the process.

Are the original images more noisy? Is there wild differences in noise or illumination on the raw images (before any processing) ? This can be checked with Blink for example (were are looking for a big problem, not for some nuance in the last decimal). If the calibration/registration did not significantly and visibly lower the noise, there is a problem in the process.  If the raw image are extremely noisy compared to other people with similar equipment or your experience, there may be an acquisition/camera/weather/loading problem (although I do not see what this could be, it is better to be sure that you are looking for the problem at the right place - I sometime get this kind of images when there is a small high altitude haze, not really visible at naked eye)

A next step is use StarAlignment to align the raw images (usually this works fine enough). It is the even easier to see if some images are unexpected with Blink.

Then do an ImageIntegration of the aligned but uncalibrated images, using Average without rejection. The result may not be pretty, but it should have significantly less noise than the original images and, as far as noise is concerned, it is about as good as you can hope to get before any PI magic.

You can compare the raw integrated image with your processed image - they should have about the same noise (the processed image will have be slightly more noisy as flats and darks can only add noise, but should have far less systematic errors). If your processed image is much worse, then there is a problem in the dark/bias/flats or pre-processing steps. A similar small step approach may help identify the step(s) or image(s) causing the problem.

If you want to have significant numerical statistics from ImageIntegration you should crop the images to a common area before integrating the aligned images, otherwise the black border may confuse the statistics - I think.  This can be done withan ImageIntegration of the Minimum, then using DynamicCrop to select an area without the dark border, then using the ImageContainer to do the dynamic crop on all images.  There is surely a video somewhere on this step.

Here I would only have noise if I was triyng to image something ...
Clear sky-- bitli