PixInsight Forum

PixInsight => General => Topic started by: silentrunning on 2019 May 05 04:37:05

Title: Fixed pattern noise?
Post by: silentrunning on 2019 May 05 04:37:05
I've just encountered an annoying but somewhat interesting (in that I can't explain it) artefact on last nights images.

The left image below shows 10 stacked calibrated frames. You can see a vertical banding effect that I have not seen in any of my previous images. At first I thought this might be vertical banding in the master bias files but I have used these continuously without issues. Also to check this was not the case I aligned and stacked the uncalibrated raw images and the vertical banding remained so I figured it was either present in the raw data or created by the star alignment process. To add to the mystery I could not see the artefact in any of the individual raw frames.

I began to think maybe the camera was picking up noise from a power cable or something similar and began to think what had changed since my last imaging session. The only thing I changed was the camera rotation in order to get the desired framing of the object. By chance this was exactly square with the RA/DEC grid on the sky. Then I began to think as I only dither in RA could this pattern be a result of dithering occuring along fixed columns of camera pixels so that any fixed pattern noise perpendicular to this direction would not be ironed out by the dithering. The second image shows the calibrated images stacked without registration to show how the dithering has aligned with one axis of the camera. Can anyone confirm that my logic is sound and this is a possible cause for banding? In future i'll dither in DEC as well - something I have avoided due to large backlash issues.


Title: Re: Fixed pattern noise?
Post by: STEVE333 on 2019 May 05 10:02:36
"Silent" - Your logic makes sense to me. However, it is odd that you aren't able to see the banding in any of the single images.

I also have bad Dec backlash, but, have settled on using both RA and Dec dithering because it makes the data so much better.

I use PHD2 and use the Lowpass algorithm on the Dec guiding. I also allow 30 sec for settling in the BYEOS dithering section (I use BYEOS to control my Canon T3). Sometimes the Dec guiding jumps and takes a while to settle (which will usually ruin that frame of data), but, the improvement in the remaining data has been worth it for me.

I have a new CEM40 EC on order and, hopefully, the backlash problems will be a thing of the past.

Best of success with resolving this banding issue.

Steve
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 05 11:28:59
Thanks Steve

On very close inspection it is there in the individual frames, very close to the noise level but creeps up as more frames are integrated. So I figured it most be in the bias. Image on the left below shows the master bias and the one on the right after I have applied MMT to 8 layers and discarded the residual layer which shows the bands quite well (ignore the defect pixel lines). I guess sliding up and down these bands and adding the images together (dithering along that axis and integrating) accentuates these bands as previously deduced.

Apart from this oddity dithering has always worked perfectly for me. I guess in the past my RA direction has never been plum along that camera axis before. Anyways I have pretty much sorted the backlash issue in my DEC gearing so it's time to include DEC dithering as well.  :)

The question now is can anyone suggest away of removing the banding as, typically, the underlying data is not that bad for once!

John


Title: Re: Fixed pattern noise?
Post by: STEVE333 on 2019 May 05 12:51:13
John -

You might try the  CanonBandingReduction script. I believe it wants the "bands" horizontal but not sure about that. You can try it w/wo rotating the image 90 degrees.

I've only used it a couple of times, but, my banding wasn't as severe as yours. One time it did a nice job.

Just a thought

Steve
Title: Re: Fixed pattern noise?
Post by: vicent_peris on 2019 May 05 13:27:10
Hi,

CanonBandingReduction can do a good job on this image. But I just wrote a script that will completely eliminate this banding. I will release the script in the next weeks.


Best regards,
Vicent.
Title: Re: Fixed pattern noise?
Post by: STEVE333 on 2019 May 05 13:38:55
Fantastic Vicent.

Thanks for doing/sharing this with us. You have given a  lot to the PI community.

Much appreciated.

Steve
Title: Re: Fixed pattern noise?
Post by: Alejandro Tombolini on 2019 May 05 13:53:23
Hi silentrunning,

I used to have that problem taken pictures with DSLR at high temperature and mitigate the problem using Fourier.
While Vicent's script is ready you can open FourierTransform tool and apply on the image. It will generate two more images (DFT_magnitude and DFT_phase)
Open CloneStamp tool and on DFT_magnitude delete two vertical lines that you will see over the horizontal one. You has to see it as the attached image.  Once corrected apply CloneStamp and close it.
Open InverseFourierTransform and choose as First DFT componet: DFT_magnitude and as Second DFT component: DFT_phase. Apply Globaly.

Hope this helps.

Saludos, Alejandro.   
Title: Re: Fixed pattern noise?
Post by: STEVE333 on 2019 May 05 15:55:00
Very clever Alejandro -

I made a copy of John's image and applied your approach. Works like a charm! Not a vertical band in sight!!!

Steve
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 06 02:12:35
Thanks Guys

Vicent - I look forward to your script being released.

Alejandro, very interesting approach using the FT. I had a good play around whilst in "k-space" - I'm an MRI radiographer - adding individual bright spots and creating some very interesting banding at different angles, intensities and frequencies. I find the vertical banding in the original image is caused by the bright horizontal line in the FT.

I can't however reproduce the level of yours' and Steves' results. I find as I try to clone out the lines near the centre of the image I get banding effects that are worse than the original - lower frequency banding but with more contrast - no matter how careful I am to ensure cloning of neighbouring pixels. The best I can do is to leave the middle bits of the lines which leaves banding though much less obvious than the original and at a level that can disappear after judicial stretching. Image shows original, FT, Edited FT and Inverse FT.

I wonder if it's because the image I uploaded was a low resolution JPG screen shot and I'm working with the original fits file. Maybe I could upload it somewhere to see how you guys get on with it?

Indeed - cropped my original screen grab  to the image only and applied the technique - second image below. Strangely this time the brightest line is vertical and no horizontal line, completely different from the FT of  the FITS file. The two vertical lines that Alejandra mentions are there as well which confused me a bit as they don't appear in the FT of the raw image. As with Steve, the banding is all but completely gone when the two vertical splodges are clone stamped out. Much easier process on the jpg for some reason but it would nice to use the original data.



Title: Re: Fixed pattern noise?
Post by: vicent_peris on 2019 May 06 04:08:49
Hi,

Thanks, I just left a message in its corresponding board. Hope it will work for you!

https://pixinsight.com/forum/index.php?topic=13595.0 (https://pixinsight.com/forum/index.php?topic=13595.0)


Best regards,
Vicent.
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 06 04:57:51
Hi Vicent

I've got a break from work at the end of week so I'm looking forward to digging into your scripts. This will be the first time I have looked at PIs scripting ability but I was a s/w engineer for 15 years so It's an exciting prospect.

Just to get a feel for things I extracted the scripts and tentatively loaded the lineardefectdetection script into the editor and set the output directory. Just to get used to compiling and executing I ran it on my image - I know it's not what's it intended for but interesting first use of the environment, I'm looking forward to running it on an integrated non-aligned set of images to generate a defect map at some point.

Well anyways the output was interesting. The line model looked somewhat like the vertical banding in my original post so I couldn't resist subtracting it from my original image and hey presto big improvement!

Title: Re: Fixed pattern noise?
Post by: vicent_peris on 2019 May 06 05:09:29
Hi,

If you want to apply that pattern to the target image, write this formula in PixelMath: $T - line_pattern + med( line_pattern ) .

If forgot to say that these scripts should be applied to the linear image; any delinearization will deliver non-optimal results.

In your image, you don't need to detect any defective line since you want to simply remove the banding over the entire image, which takes always entire columns.

Best regards,
Vicent.
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 06 05:25:22
Thanks Vicent

Image below is the same processing done on the original linear image with your formula for subtraction.

My image has had my own defect map and clean up routines applied to it but I'm looking forward to detecting extra lines by using your routines as intended.

Really excited by this and looking into your scripts.

John
Title: Re: Fixed pattern noise?
Post by: vicent_peris on 2019 May 06 05:31:14
Could you please upload the image? It's a good test.

Thanks,
Vicent.
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 06 05:41:35
Sure.

How do I upload the original FITs file as it's bigger than 512KB?

John
Title: Re: Fixed pattern noise?
Post by: pfile on 2019 May 06 10:19:19
upload to google drive, dropbox, etc., allow the file to be read by anyone with the link, and post a link here.

rob
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 06 12:00:04
Ok

Hopefully you can now download it...

https://drive.google.com/file/d/1USCgBPJQNNKYIqiNQj-kdrGbrghV5ryF/view?usp=sharing

John
Title: Re: Fixed pattern noise?
Post by: pfile on 2019 May 06 12:18:47
yes - the link works

rob
Title: Re: Fixed pattern noise?
Post by: vicent_peris on 2019 May 08 04:41:29
Hi John,

I tried the script in your image and it does not work. The only reason I find is that the lines are not completely vertical. So, IMO, the fourier solution is the best one.


Best regards,
Vicent.
Title: Re: Fixed pattern noise?
Post by: silentrunning on 2019 May 09 01:46:15
Hi Vicent

Thanks, this is all very educational - I guess the dithering and integration essentially blurred what are residual vertical lines left from the bias.

If I perform an FT on my master bias and delete all but the middle horizontal line from it  and then perform an inverse FT I get what appears to be a model of the vertical data in the bias. I suppose if I could somehow align N copies of this vertical line model by using registration data from the lights I might end up with the same blurred vertical bands that appear in the integrated image - which came from N frames - which could then be subtracted out.

It's just out of interest that I'm pursuing this now but is there a way of acquiring the registration information from one set of data and applying it to a second set?

Many thanks

John