Removing Noise from an Image

[bulrichl]

Sorry, I need one further information and forgot to ask the question before:

You can configure SGP to save CR2 or FITS or both formats. Are you using CR2 files or FITS files for the processing in PixInsight?

Bernd

[ngc1535]

Yeah...perhaps I should not have suggested a cause for the flipping.

Bernd's suggestion that the DSRL camera does this because it has a tilt sensor (most people like to have their friends with their feet below their heads) and this information is encoded in the image header to be interpreted by some software (or maybe even outputted by the camera as such).

Regardless of the reason- the main point is that data frames and calibration frames need to be in a one-to-one relationship when using them. Subtracting a dark from a rotated light frame (or a rotated bias/dark..) will appear to "work" at first glance since it will get rid of the pedestal..but it will not map to the individual pixels and basically just give you noise. The thing this, this is a relatively subtle error and not easy to catch at first glance.
-adam

[pfile]

the thing is though when working with CR2 files one should have the DSLR reader set for "pure raw" in which case the metadata containing the camera orientation is completely ignored. so you can't end up with the calibration frame and the light having different orientations.

however, if you are for instance using SGP to capture lights and saving as fits, and used some other tool to capture darks, you have one real problem and one potential problem. the real problem is that when SGP writes a fits from a DSLR, it 'expands' the data to fill the 16-bit integer space. but when PI opens the CR2, it will not do this expansion. so the frames are completely incompatible with one another. the potential problem is that PI's FITS reader is set to the opposite direction than SGP's FITS writer. that can definitely cause the problem that adam is describing.

if you stick with CR2, you should be fine in this respect. if you use FITS, and as long as you capture lights and darks/bias/flats in the same program, you will also be fine. you may find that your images are flipped from what you would have expected, but at that point it is just a cosmetic problem.

rob

[BobinBend]

Bernd and others,  I am capturing my lights with SGPro in CR2 format. My calibration data is also captured as CR2 BUT with another capture tool, BYEOS. I currently create my Flats before every AP session also using BYEOS but my darks and bias subs are several months old.

Are you guys saying that I should create all my calibration data using SGPro? Should I be using CR2 format only?

I feel the problem is related to the Meridian flip because when I stacked the raw files from both sides of the flip I get the red noise anomaly, but when I stack the raw subs from only one side of the flip the red noise disappears.

Your thoughts?
Bob

[pfile]

CR2 should in theory be fine, no matter the capture application. however of course BYEOS and SGPro gain the ability to write CR2 files by incorporating the Canon SDK (software development kit) and it is possible that BYEOS and SGPro are using different versions of that SDK. so there's some chance the CR2 files are not 100% the same between both applications but i feel that the risk here is low. the SDK changes when canon releases new cameras but it's very likely that the suppport for a given camera is unchanged from version to version of the SDK.

the meridian flip is almost certainly a red herring. as long as you are using the right settings in PI, there's no reason why the data coming off the camera should be flipped pre- or post- meridian flip. i don't believe any canon camera actually changes the data written to the CR2 file based on the orientation sensor. i can't say that with absolute certainty but i think we'd have heard about it by now if they do.

i suppose the meridian flip could be related to the culprit, if for instance your sky conditions differ wildly between the western side of the meridian and the eastern side.

rob

[bulrichl]

1) Short-term measure
I assume that you did not change the 'RAW format preferences' until now, and that you don't use a rotator for compensating the orientation after M.F. If the issue that you observed is caused by the flip information that the camera stores in the images, you will see it when blinking the uncalibrated light frames in PI: are the images captured after M.F. upside down (indicating no rotation in PI) or in the same orientation as the images captured before M.F. (indicating a rotation by 180° in PI)?

The settings for the RAW module in PixInsight will have an effect only when processing RAW image files (CR2). Since you are using CR2 files, please check the current settings of the RAW module in PixInsight:
Click in the menu 'VIEW/Explorer Windows/Format Explorer'. The Format Explorer then shows a list of format categories. Double click on 'RAW' opens the 'RAW Format Preferences'. Take a screen section of the 'RAW Format Preferences' and show it here.

With this information, the most reasonable way to process all of your present project's light frames to yield a low-noise integration should be obvious.


2) Long-term measure
For future projects it would be worthwhile to revisit which file format is favorable: CR2 or FITS. SGP allows you to save either format, CR2 or FITS (or to save both). I consider using FITS as superior since there is valuable information in the FITS header. Only one important information is missing in the FITS files: the debayer pattern (for the 60D: 'RGGB'). So using the FITS format, you will have to input the debayer pattern 'RGGB' manually when debayering.

If the rotation that you observed is caused by the flip information that the camera stores in the images, this would only happen with the CR2 format AND the wrong setting of the 'No image flip' option in 'RAW Format Preferences', but it would not happen with the FITS format.


It is right to use only one file format. As Rob wrote, mixing images in CR2 and FITS format will not work. So if you decide to change the file format to FITS, you will have to capture new calibration files in the same format.

Regardless of which file format you decide to use in future, in my view it is not a good idea to capture light frames and calibration frames with different software - this could give rise to incorrect image calibration. So yes, I suggest you to use only one acquisition software for all frames in future.

Bernd

[Juan Conejero]

So yes, I suggest you to use only one acquisition software for all frames in future.

That's true... and in the near future PixInsight will be an option for image acquisition directly in XISF format, thanks to our integration of the INDIGO platform, on which we are working.

[bulrichl]

I tested different settings with my Canon EOS 600D (= Canon Rebel T3i) regarding the automatic rotation of images and also looked into the camera manual (should have done this before). The results are:

In this Canon model, which is very similar to the 60D, automatic rotation only affects the upright representation of images that were captured in portrait orientation. A rotation by 90° will be executed only if
- the auto rotate function is enabled in the camera (this option can be enabled either for display on the camera screen and on the computer or only for display on the computer), and
- the 'No image flip' option is disabled in PixInsight.
No rotation will be executed if the camera is rotated by 180° during capturing.

So I conclude that the camera flip information can be excluded as a possible cause of this issue.

Bernd

[BobinBend]

bulrichl,

Thanks for your suggestions.

I assume that you did not change the 'RAW format preferences' until now, and that you don't use a rotator for compensating the orientation after M.F. If the issue that you observed is caused by the flip information that the camera stores in the images, you will see it when blinking the uncalibrated light frames in PI: are the images captured after M.F. upside down (indicating no rotation in PI) or in the same orientation as the images captured before M.F. (indicating a rotation by 180° in PI)?

Attached is an image of my Raw format preferences and I don't use a rotator for compensating the orientation after a M.F. When looking at my light subs with Blinking the images are NOT upside down but ARE flipped horizontally. Also when I examine a single light sub after I've stretched it, the background noise appears to be clear. The noise seems to be introduced only after the integration of the light subs.
Thanks,
Bob

[bulrichl]

Bob,

your settings in RAW format preferences are exactly as they should be for astrophotography (Pure Raw), don't change anything here.

Now it's going to get interesting:

When looking at my light subs with Blinking the images are NOT upside down but ARE flipped horizontally.

Are you saying that the uncalibrated light subframes are mirrored after the meridian flip?

Also when I examine a single light sub after I've stretched it, the background noise appears to be clear. The noise seems to be introduced only after the integration of the light subs.

That's confusing, I expected that the noise in the post-M.F. images was introduced by calibrating them. Did you compare a post-M.F. image before and after ImageCalibration as well?


It seems to me that this issue can be resolved only when you will upload 4 images to a filehoster (I will gladly take a look): two pairs of uncalibrated (CR2 format) and calibrated (XISF format) light frames, one pair before and one pair after M.F.

Bernd

[BobinBend]

Bulrichl,

Yes the uncalibrated light subs are mirrored after the meridian flip. I discovered this by looking at the raw subs in Windows explorer using the Windows Photo application. I also loaded all the raw images into PI's Blink process and can see the mirrored images as well. I confirmed that these subs become mirrored right after the M.F.. I also verified that my Canon 60D Auto Rotate is in the "OFF" position. Should I be concerned about the mirrored images? I would expect to see some difference after the M.F.

For integration I used the BatchProcessing script to integrate the subs after loading my darks, flats, bias subs as well as my lights. Therefore I was not able to see if the calibration steps alone introduced the noise anomaly. I think my next step is to do the longer version of registration, calibration, integration to see what my results are each step of the way.

With the holiday coming up I will probably wait until this weekend to resume this investigation.
Thanks again for your insights!
Bob

[bulrichl]

Bob,

the auto rotate function of the camera is definitely not involved here. There should not be some mirrored images when they are captured in one session, with the same acquisition software, opened and displayed with the same processing software with the same settings. For me this is a mystery up to now.

The BatchPreprocessing script also does save the calibrated light frames, doesn't it? The data have to be stored in order to be further processed (debayering, registering, integration), I guess they must be somewhere on the hard disk, unless they have been deleted already.

For now I wish you a nice Independence Day, enjoy the holiday! Maybe we can find out more when the celebration is over.

Bernd

[STEVE333]

Yes the uncalibrated light subs are mirrored after the meridian flip.

Hi Bob - I don't want to jump in the middle of this, but, after the meridian flip the light subs should be rotated 180 deg (not just flipped horizontally). Trying to calibrate subs that are just flipped horizontally will definitely introduce noise.

Have no idea what could cause this, so, that's it for me.

Steve

[BobinBend]

Steve and Bernd,

I did a little more investigating and after Steve's comment about the light subs being rotated 180 degs and brought two subs into PI (from different sides of the M.F. and rotated one of them. Sure enough it matched the other. So I was mistaken about the mirror flip. Sorry about that. Thanks Steve.

Bernd, I also looked at the Temp folders from my BatchProcessing run and found the Calibrated and Registered folders. The Calibrated images are without the noise anomaly while the registered folder ( with _c_d_r extension files) contains the noise anomaly in each of the files. So it appears for some reason that when I attempt to register the raw images from both sides of the M.F. they produce registered subs with this added noise. Of course integrating these subs will also produce noise in the integrated image.

Do either of you have an explanation for this and what is the solution if I want to use all the subs for my final integrated image?

Thanks,
Bob

[pfile]

rather than just eyeballing things, open two corresponding _c_d and _c_d_r images and run the NoiseEvaluation script on both and see if they really have significantly different noise.

in short there's really no way that StarAlignment can add noise to your images. it might introduce artifacts depending on how it is configured, but not noise in the strictest sense.

rob