Bad vignetting in spite of using flats

[headdown]

Hello folks,

I've been using Pixinsight for a while, and have a workflow that has been working well for me.  Until the latest image that is.  I am processing about 18 hours of the M77 region, imaged with a Takahashi FSQ 106ED.  As usual, I used the batch preprocessing script with a super bias, fresh flats and dark frames.  After the script calibrated, debayered, and registered the frames, I manually used ImageIntegration to integrate the 219x5m subs.  The result is not good, as you can see from the attached screen shot.  While it is nice and clean, and has great potential, the vignetting is terrible.  The light strip at the bottom of the frame is a problem that I have always had using the Tak with the Canon 6D.  I have not found a way to remove that strip with calibration, and have in the past reluctantly accepted that I have to crop it from the final image.  But that I can live with, and it is nothing new.  The really exaggerated vignetting is new though.  It is almost as if the flats made the vignetting worse instead of removing it as usual.  I use this scope and camera combination all the time and have never seen this before.  The flats usually remove any minor vignetting completely. I am at a site in Arizona with a SQM of 21.5, and there should not be any serious gradients at this dark site.
There is a warning visible in the process console: Inconsistent observer values - metadata not generated.  I have never seen this warning before.  Can anyone tell me what this means?
A last mystery is the red (hydrogen?) arc in the lower left of the frame.  I have not seen this in other images of this area.

Any advice much appreciated.

Dean

edit:  I checked a sub before and after calibration, and sure enough, the flats are doing the job and removing the vignetting, and even the strip at the bottom of the frame that has long been a problem.  See attached screenshot.  So what is happening that the vignetting/gradients are being reintroduced to the image during integration?

[pfile]

the warning just means that the OBSERVER keyword in the input files was not the same in each file, so PI doesn't know which one to use to put in the FITS header of the integration.

the arc is probably the reflection from a bright star just outside the sensor FOV.

since the OBSERVER keyword exists, it sounds like you're not using CR2 files as input - are you using a capture program that generates FITS? if so did you take all of your calibration frames with that software and save them all as FITS as well?

rob

[headdown]

Hi Rob and thanks for the reply,

Okay the observer message seems irrelevant to my problem then.  I am using SGP and do capture FITS.  And yes all calibration frames were taken as FITS by SGP.  And I just checked in CdC and I can't see any bright stars that are nearby but out of the frame.  Please see edit that I just made to original post.

Dean

[headdown]

And I just checked and after debayering the calibrated image above, the strip at the bottom gets very light again, and will have to be cropped.  But the rest of the image seems okay, with no vignetting.  So it seems to be in integration that somehow the vignetting returns.

Dean

[headdown]

More strangeness.  When I plate solve the integrated image it gives me 7.72 arc sec/pixel.  This is way off...it is actually 2.55 arc sec/pixel with the Canon 6D and Takahashi FSQ 106ED.
I don't know what happened to change the resolution, or if/how it relates to my issue.

Dean

[pfile]

i've definitely seen this kind of problem before - the individual subs look OK but when integrated, flattening problems are evident. i think this is just because the SNR in any one sub is too low for your eye to really see any subtle problems, but in the integration the SNR is pretty high and very small differences are more readily visible.

did the platesolve actually find the real center of the image?

i guess we need to go back to basics - what is the length of the flat subs and how were they calibrated? if they are really short then maybe the flats should be calibrated separately using only a master bias.

rob

[headdown]

Hi Rob,
Oops..the plate solve is not a problem after all.  I must've changed the resolution when I changed it to JPG to upload to Astrometry.net.  I converted it again, and solved it again and the pixel scale is correct at 2.55.

The flats are 1 second exposures, using a iPad on dim flashlight mode with a white t shirt stretched tight over the dew shield.  This usually works very well for me.  They were calibrated by the preprocessing script using the default settings...I think it used Windsorized sigma clipping.  25 flats, 40 darks, and a super bias were used.  And as you can see in the pic, they flats seemed to do a good job on the individual subs.  Your theory may be correct in that I am integrating a lot of subs at 219, so it wouldn't take much in each to accumulate.  Tomorrow I will be processing the IFN and M81 and M82 with about 300 subs, so if this happens again we will know that somehow the large number of subs is contributing to the problem.  I have gone as high as 160 subs in the past with no issues at all, so this is strange.  Except for integrating more subs than usual, I can't think of anything else that has changed in my work flow, which usually gives good results.
Dean

[pfile]

i guess you should check the console while BPP is running - BPP attempts to apply the master dark to the flats by scaling it but since the flats are so short it may not be finding a scaling factor. in this case i think it just subtracts the dark from the flat without scaling it, which may not be the best calibration since the dark signal in the flats is probably pretty negligible. also the lack of temperature regulation in DSLRs always makes dark/light/flat matching somewhat difficult, so it's possible that your darks don't match the flats temperature-wise...

i have never used an iPad for flats - don't know what the backlight LED PWM frequency is. but most people would say that making the panel dimmer and the flats a little longer is a good idea to eliminate any shutter shadow effects and panel flicker effects. 1s isn't horribly short, though. anyway it might make sense to take some 1s flat darks and calibrate your flats with those, but the bias alone would probably work OK.

btw that shadow at the bottom is the shadow of the mirror when it's fully retracted. some people actually remove those from their DSLRs but of course that renders the viewfinder useless. usually this is only a problem with faster optics but the FSQ106 has a massive image circle so not surprised to see it there.

rob

[headdown]

What is strange is that this is my usual work flow and method and exposure for flats, and it always worked great.  And I just now integrated only 16 of the 219 5 minute subs of M77 using ImageIntegration with the same setting as for the 219 subs, and it gave the same result.  Huge vignetting.  So the problem is not the high number of subs and a slow accumulation of gradients.  And the 1 second exposures with the iPad have always worked fine before.  Watching the process console if I recall it builds the master flat, then subtracts the master dark from the flats.  Once in a while it gives the message about the dark being incompatible with target frame, etc, but usually not.

That is good to know about the mirror shadow..thank you.  I never use the viewfinder anyway and would love to get the mirror removed to eliminate that strip.

I am still imaging tonight and will take another set of flats.  Not that it is likely to help, but I am running out of ideas.  The integrated image looks very good except for the vignetting, and I would really like to get it properly processed and not waste it.
Dean

[dld]

I'd had a shadow at the bottom side of my integrated lights when using my 6D with an 135mm lens at f/2 and I was a bit disappointed. Now I am taking flats using my laptop's monitor and a thick white paper. I am taking flats in a dark room (usually the next day) at different camera orientations and positions to average out any paper inhomogeneities  (yes, I could see those!) and brightness unevenness of the monitor. By adjusting the brightness of the monitor, I am targeting at exposure times near 1 second. I also use the mirror lockup setting of the camera during taking lights and darks (I don't know if this actually helps!)

Here is a helpful resource on taking flats with a DSLR: http://www.myastroscience.com/proper-flats-with-dslr

I have found that if after LP subtraction you have only this annoying dark strip remaining, the CanonBandingReduction script can help!

[headdown]

Hello did,
It sounds like we use similar method to take flats.  I use an iPad with the brightness as low as it goes, and with the t shirt stretched tight over the dew shield at night I get the 1 second exposure. I think I will likely just get the mirror removed if it is not too costly to do so.

Dean

[GJL]

I don't want to interfere in the whole discussion, but I notice that the two posted images are mirror-inverted against each other. Are all pictures including flats properly registered?

Gerhard

[dld]

Hello Dean,

I wouldn't do anything before eliminating other sources of errors like the one Gerhard notices. I would go back and revise my procedure for taking and integrating flats. Maybe 1 sec is not enough to provide a good signal at the vignetted edges of the frame. Maybe the number of flats are low, resulting a low SNR at the vignetted parts of the frame. Finally I would seek advice from other users with the same telescope/camera.

[headdown]

Hello Gerhard and did,

The subs are properly registered, and I used the preprocessing script to calibrate them.  I assume it applied the flats in the proper orientation.  And the above calibrated image looks good, with no vignetting visible after the flats had been applied. As for my procedures taking flats, it has not changed, and it has worked very well for the last two years or so.  I judge the exposure duration by watching the histogram on test images, and I aim for the flat data to be roughly in the middle.  I have used from .3 to 1 second with no issues until now, so I don't think the problem is in my procedures for taking flats.

But I think I might have figured out what happened.  I think the gradients may be coming from the lights in my RV.  I have heavy shades, and I keep the lights low when imaging.  But in this rare situation I was imaging over the top of the RV at some point in the night, and with the  f5 Tak, it may have been picking up my own light pollution.  On the same nights, I was imaging pointing in the other direction, away from the trailer.  That image has a gradient that is likely from the RV town of Quartzsite.  ABE did an amazing job of removing that gradient with one click, but I have had no such luck removing the gradient on this image, though I will keep trying.

And I will of course be relocating the scope and not imaging in the direction of the RV again.  If the problem disappears as fast as it appeared, I think I will have found the answer.  That won't be happening for a few more weeks, and I will report back when I have an answer.  Thanks again to all who contributed to this discussion.

Dean

[dld]

Hello Dean,

Good news then! The RV lights could be the source of the arc at the bottom left corner. I am glad that you found the source of your problems!

Clear skies,
dld