Image Calibration Problems

[cs_pixinsight]

First off I'd like to introduce myself.  I have used Pixinsight LE in the past to process some off the tripod type shots when I dabbled in astrophotography.  I decided to make a curved rod barn door tracker in a serious attempt to see if I liked it enough to spend the big bucks on some serious equipment.  I'm using an unmodified Canon 400D DSLR with the Canon 28-70 2.8L lens to get some wide fields of Orion from my backyard.  Light pollution isn't the best here, but I figured it was low enough to test things out without purchasing light pollution filters.  On an average night the sky fog limit is probably VLMag 4.6-4.7, so my camera settings are typically ISO 16000, F6.3, 60s @70mm.  This leaves the back of camera histogram about 30-40% away from the left edge for all channels.

In this process I started with Deep Sky Stacker (DSS) and all was good until I started doing longer sessions without resetting the barn door.  The 1.5 hour sessions (90 subs) all had obvious vertical banding in them even after calibration with bias and dark frames taken immediately after the imaging session.  These calibration frames should be within 5-10 degrees F.  I went as far as 100 bias and 100 darks in an attempt to get rid of the banding with no success.

So, I decided maybe DSS was the culprit and started looking at other software.  Pixinsight was one I wanted to try due to my previous experience with the LE version and I also looked into Nebulosity 3.  First up was Nebulosity and it did a better job than DSS, but the banding was still evident.  I then turned to PI hoping that its image calibration and integration routines would be the best of all and I would leave all this banding behind and have much better final stacks.  Unfortunately, that hasn't been the case and I'm unsure of where my issue(s) lie.  See the animated file attached for comparisons of the 3 final stacks comprised of 89 lights, 40 bias and 40 darks.  To me, DSS is the worst, PI in the middle (this was Winsorized Sigma Clip (4,2)) and Nebulosity3 best (pardon the Nebulosity trial bars).

In my PI attempts, I have followed the DSLR work flow, Vincents tutorial and read every forum post I could find on calibration.  Here are some of the options I have tried in an attempt at figuring out why I can't get a clean stack:

• Use/Removed input hints
• Tried different stacking methods and thresholds that go with them
• Added a pedestal=40 - which helped.  Without this the final stack was extremely noisy. But why do I need to do this and how do I optimize this value, since 40 was just a guess.
• I even went as far as using PixelMath to calibrate in an attempt to find where things went wrong.

I believe my biggest hangup at this point, being new enough to all of this, is I don't know if what I'm getting at each stage is valid or not.  For example, is my master bias valid?  Visually inspecting a debayered version of it, an uncalibrated master dark and a calibrated master dark, it sure seems like they match up well and the bias signal is being removed.  I'd assume the lights are getting processed correctly too, but based on the intregrated light stack I'm not so sure.

In any case, I'm hoping some kind soul on the forum will be able to help me troubleshoot this so I can at least match what Nebulosity3 is generating (but hopefully much better).  Let me know what information you would like to see and I'll be happy to provide it.


Thank you, Craig

[astropixel]

Craig. Welcome. You are on the right track.

The Canon banding reduction script will help you there, but the best way to reduce banding is to dither your images.

The red streaks through your image indicate that your curved rod tangent arm is reasonably accurate. Consequently, your images are aligned very nearly edge to edge, which produces this effect. This is an acquisition issue and very difficult to fix with calibration or processing.

Dithering, that is moving the camera very slightly, say 5 or 10 pixels between each frame will eliminate the streaks (that is, one pixel spilling over into the same pixel in the next image).

Turning your curved rod tangent arm off for about 5 seconds between shots will emulate dithering. This method is not as ideal as a box spiral for instance, because every so often you will need to recenter your target - but it should solve the streaking and reduce banding.

Your images should look flatter and be easier to work with.

Referring to the attached images. To remove the streaking, I took a second set of the same area that were dithered and mixed them up with the first set. The final image was presentable - only just - see second image.

The third image is a properly dithered set.

ap.

[cs_pixinsight]

Astropixel, thank you for the information.  I would have never thought a tangent arm tracker would be accurate enough to need dithering.  I have an idea to test out your theory and will try to post some results as soon as I can, probably tomorrow. 

Do you think it would be possible to post a larger size for your examples?  It's hard to see the details at their current size.

Also, do you have any ideas on the pedestal needing to be increased from 0?  I just pulled 40 out of a hat, so actually having some type of methodology on what to set it to would make me feel better about it.  Average/Mean/StdDev of the dark or bias frame perhaps?

Thanks again, Craig

[astropixel]

Craig. I have attached the source images - the third was acquired with a spectrum modified camera and looks a little different.

I would, for the time being, leave the pedestal at default, unless you really need it. I never use it myself, because my dark sampling is high and exposure times longer. As with all things PI it is experimental. Try 10 or 15 to start with.

ap.

[JamieInCLT]

Hi Craig-

I am having trouble w/ calibration also. 

Can you post a picture of what your master dark looks like?  I think that mine is too noisy, removing more than it should...

[cs_pixinsight]

Astropixel, thank you for the larger size samples.  Your first image named rhoophstreaky.jpg definitely looks simlar to the streaks in my final stack, especially the my DSS stack.  I see two main issues with my final stack:

• The streaks which you have identified (diagonal in my sample image)
• Vertical banding - which looks like incomplete bias substraction to me

To make those vertical bands more obvious, I created another animated gif which shows the following:

• 1st frame:  A stack of 10 consecutive subs starting with a PS overlay where I see the veritical bands
• 2nd frame:  Same 10 sub stack without the overlay
• 3rd frame:  A stack of 10 subs, skipping every other sub - again with the PS overlay
• 4th-10th frames: same pattern but the stacks contain every 3rd sub, every 5th sub and every 10th sub

This was meant to simulate the dithering you mentioned I should try in your first reply.  Looking at the gif, I believe that as the number of skipped subs is increase both issues seem to go away.  Every 3rd sub seems to get rid of a majority of the problem, but every 5th and every 10th helps even more.  Going back to my DSS registration settings, I noted the differential x and y  position.  In the first 10 subs of my session, dX=-1.56 and dY=-2.01 and in the entire 89 sub session it was dX=3.61 and dY=-29.07, but dX changed sign midway through the session (my curved rod must be off a bit).  The dX actually ranged from -7.37 to 3.61 over the 90 minutes.  This averages out to a dX=.122px and dY=.323px per 1min sub.  This means every 10th sub, which seems to have elimated the problem, shifted approximately X=1.2px and Y=3.2px.  Every 5th sub, which still leaves a bit of the banding visible, shifted by X=.6px and Y=1.6px.

Do you feel this test is representative of dithering or do you think the banding will show up as more subs are stacked and noise is reduced to make it more visible?

The other interesting item is this test was the hot pixels which didn't get removed with the dark subtraction.  There are 5 red and 1 green hot pixels in this crop.  3 red hot pixels are located in the lower left quadrant.  You will notice that they don't get removed completely until I was using every 5th frame in the stack.  It appears that the dark scaling must not be removing the hot pixels completely and I'll need to resort to the CosmicCorrect script to remove them.

The last item I would like to get your thoughts on is the comparison between the Nebulosity and PI results.  They used the same subs, but Nebulosity seems to have eliminated the banding to a higher degree and the noise level seems much lower too.  Do you concur and would you have any suggestions on getting PI to match or improve on the Nebulosity result?

Sorry for bombarding you with more questions, but I would like to better understand what I'm seeing in my final result.

Thanks again, Craig

[cs_pixinsight]

Jamie, I'll try to post an image of my dark later tonight.

Craig

[astropixel]

Hi Craig. I have seen red streaks both vertical and diagonal, it all depends on the general direction of drift during imaging. Banding is not evident in my first image, but it's there all the same. Seperating the channels, R is dominated by banding.

I have not used nebulosity. Perhaps pixel rejection is more aggressive. PI does however, present your data faithfully. Banding and bad pixels are masked with adequate dithering. Integrating images without calibration, produces remarkably flat, artefact free images.

Good calibration in PI requires adequate sampling of lights and calibration frames. My view is that most calibration / processing issues with DSLR images in PI can be traced to acquisition. Adequate SNR and good sampling of all frames (particularly calibration frames) appears to be the key.

Your gif illustrates this well. There is a hot pixel near the bottom and middle of the frame. It is masked as the pixel distance between the frames increases. With a gap of 5 frames it is not visible. That's the distance to dither your frames. There is less noise and there are no red streaks.

ap.

[cs_pixinsight]

Jamie, here's my master dark after calibration with my master bias and an output pedestal=40.  Without the output pedestal the master dark becomes mostly white, probably because the bias subtraction is causing negative numbers.

I hope this is helpful.