Using FFT to filter out very obvious periodic signal

[astrospotter]

Hello.  I have been using PixInsight for quite a few months now and love it very very much! Well done.

At this time I have a great need to take out some very obvious periodic noise that generates very visible but low level vertical bars every 64.5 pixels spacing horizontally in frames of size 3995x3205.  These were for M51 supernova and taken from a remote observatory that clearly had bad periodic interference in acquisition.  I will attempt to attach image of screen stretched power spectrum.   If I do an FFT (centered, power spectrum) I get very regular and distinct vertical bars on 500 pixel spacing (the bars are very thin and vertical in the frame.

I realize PixInsight folks have some sort of 'religion' about painting on any mask (or in this case FFT) so the clone stamp is all that is really there and is close to hopeless to edit this FFT nicely.
   What I wish to do is perhaps copy very long but thin vertical bars from this image that are very close to where the thin distinct lines are vertically.  Then paste the thin copy/paste segment over the obvious periodic 'bad thing'. 

It is very difficult to save then edit this in different program due to other programs I have only support 16 bit data AND I cannot see the un-stretched power spectrum in the other program as it will be all black.     All these things mean I am trying to figure out how to do this in PixInsight.

Since I am a programmer I would even entertain thoughts on a script that could do this copy of adjacent 8 or so pixels on a horizontal line just prior to the bad lines.  (hope you follow that).
Is there an example of pixel manipulation based on X-Y coordinates in some script online or in PixInsight shippable scripts somewhere?

I realize I must not remove the central vertical line and only the other ones that are vertical in the power spectrum.

I am attempting to attach jpeg of power spectrum
Thanks for any thoughts,   Mark Johnston, a devout PixInsight user

[astrospotter]

Getting stuck with more issue that I expect is because IFFT is maybe rarely used I am hitting them.

In doing inverse FFT I have found that once I do a save of a FITS file for DFT_magnitude then the inverse FFT will always fail and complain of a 'Parse Error' as follows:
Invalid boolean literal: 'true    '

I feel that the save of the DFT_Magnitude file alters the Fits data and makes PIDFTCNT become a little bit different as instead of showing    true  (no single quotes) it then shows  'true   '.  Something has happened to it's internal 'type' so my next IFFT then fails in Parse Error.

Cannot figure out how to avoid this and if I cannot avoid this I will not be able to do my inverse fft because after 100 or so  uses of the clone stamp to edit out periodic elements PixInsight will then pop its cork (die).   So I am really getting stuck here.

I have tried to use the Replace feature of FITS header to specify true in assorted ways including some java syntax and usage of  1   or TRUE or double quotes and other things, no luck.

I attach a bitmap with the left being prior to saving the file and right after saving file where the PIDFTCNT has changed to 'true   ' which might be the reason for parse error.     

[avastro]

Hello,
Did you have tried converting your image in tif format?
That could do the trick.

Antoine

[Carlos Milovic]

Hi
For the first problem, you may try to build a map of the bad pixels, and then use the defectmap process, or the cosmeticcorrection script. They should replace the values that are causing those problems.
I would not bother too much if the clone stamp works here. The notch filter (suppressing some particular frequencies) has no further basis, so go ahead.

[oldwexi]

Hi Astrospotter!
<Is there an example of pixel manipulation based on X-Y coordinates in some script online or in PixInsight shippable scripts somewhere?>

The Process PixelMath allows read and replace of pixels with selective x-y coordinates like in a programm.
Have a look at:
http://pixinsight.com/tutorials/PixelMath/en.html

Simple example of HotPixel removal i use in raw CFA files:
x = XPos();
y = YPos();
xm1 = x-2;
xp1 = x+2;
ym1 = y-2;
yp1 = y+2;
p12 = Pixel( $T, x,   ym1 );
p21 = Pixel( $T, xm1, y   );
p22 = Pixel( $T, x,   y   );
p23 = Pixel( $T, xp1, y   );
p32 = Pixel( $T, x,   yp1 );
mean = (p12+p21+p23+p32)/4;
iif(p22 > mean * 1.2,mean,p22)

Aloha Gerald

[Juan Conejero]

Hi Mark,

Invalid boolean literal: 'true    '

You have found a bug in the FourierTransform tool. I apologize for this; will try to release a bug fix update ASAP. Workaround: save a project file instead of a FITS file; this won't modify existing FITS keywords.

As for manipulating the magnitude FFT component, Gerald has pointed out a good solution: PixelMath. For example, a simple PixelMath expression:

iif( x()%500, $T, 0 )

can be used to replace each column every 500 pixels with black pixels. Don't forget to disable the 'Rescale result' option of PixelMath. Setting pixels to zero acts like an ideal notch filter in this case, which is not an optimal solution. A better approach is replacing each column with the mean of its two adjacent columns:

iif( x()%500, $T, (pixel( $T, x()-1, y() ) + pixel( $T, x()+1, y()) )/2 )

To avoid replacing the central column you can introduce an additional condition:

iif( x() == trunc( width()/2 ) || x()%500, $T, (pixel($T, x()-1, y()) + pixel($T, x()+1, y()))/2 )

where trunc() is the 'truncation to integer' function. Of course, much more sophisticated things can be done with relatively simple JavaScript scripts in PixInsight, but I think PixelMath is basically all you need to do what you want with these images.

[pfile]

minor threadjack - if i take the FFT of some image and get the phase and magnitude plots, then i do some operation to the magnitude plot (like, say, subtracting a magnitude plot of another equally sized image), the IFFT tool tells me that the new magnitude plot is not a valid input image for IFFT. no other details.

what's the deal with that? i should be able to take the inverse FFT of two arbitrary images, right? the result could be meaningless, but...

[astrospotter]

Thanks all!   I love PixMath and have been looking for an excuse to use my software background on PixInsight scripts so here I go ...

Note to pfile:   Juan has explained in this thread that there is a bug in modification of fits header info and he will address as time permits.

Note to Carlos:  Sadly what you have said sounds quite nice but is greek to me at my level of PixInsight.  But I will try to sort it out in terms of how to go about what you suggest.   My key issue is it is VERY difficult to form any sort of map or painting out of very low level signals because I do not have 'screen transfer function' an other tools.   I may just try to do this in imagej which does have some crude paint for such masking.

Juan:  You (and the whole team) have done such an amazing job on other things that I am only glad to have helped in finding one bug in the rather uncommon workflow involving use of FFT/IFFT workflow.  I will use project in meantime as you suggest.  Thanks.

Broken record mode:  a round clone stamp is extremely painful for this sort of task as the circle must be tiny but yes I agree, it would basically work.  I also know that any sort of paint, even for masks and this sort of filter issue is considered a 'sin' per the philosophy of the PixInsight creators and I respect that view and understand why the view holds moral weight in areas other than FFT masking.   

Possible solution for consideration in future rev:
A clone stamp mode for 'rectangular' with a user-defined rectangular stamp (user enters x and y much like current radius) would greatly ease FFT power spectrum edits for true periodic noise and may not violate the views of the PixInsight philosophy.  Just a thought for some later date.

Thanks again for all the inputs,
Mark Johnston

[pfile]

Note to pfile:   Juan has explained in this thread that there is a bug in modification of fits header info and he will address as time permits.

well, what you have is something different, i think. i'm getting this error even if i never try to save any file. if i just execute a pixelmath expression on the magnitude, then IFFT does not like the magnitude image any more.

[astrospotter]

oldwexi:  Thanks for getting me started in thinking about pixelmath as a tool I could use on my own to add yet more power to PixInsight.

I stumbled with a few very fundamental issues as I have not used PixelMath yet with my own individual taylored routines but I got it to work and do exactly what I had envisioned I had needed it to do which was confirmed by looking at output  dft mag and phase and I also tried things in real/imaginary.

My key issue that was VERY hard to figure out (due to inability to see a full basic example) is that for all the variables you use in a given routine you MUST in the PixelMath  'Symbols' field place each one so they will be defined.  This is very unique and conceptual concept of PixInsight only and finally I realized that is why I kept getting I think it was   no lvalue or some such error.   I was incorrectly thinking that PixelMath would work like a 'basic' interpriter and automatically define values in the formulas with init values of 0 or whatever.   you MUST define all vars used in the PixelMath 'Symbols' field for it to work.

Anyway, that was 'noob stuff' but was highly frustrating for a few hours.

Oh and all in all even though I got PixelMath do modify images (a big step for me and I love it)  I was unable to eliminate my noise issue.  I will revisit the cal frames and so on as now I feel something in calibration is maybe why I am getting this issue.

But thanks again for the head start and it was very nice step to take to even farther recognize the extreme power of PixInsight for future PixelMath tasks.

Mark