Having trouble with compound light pollution gradients

[macnmotion]

One more thought. Could it be that the Flat is fine, and that flat calibration is merely revealing something that we are capturing on the Light image? I don't know, some type of reflection/shadow of the spider or something else? Just a thought.

[pfile]

just a couple of quick thoughts - i have not been able to look at the images you posted.

at SRO iTelescope has a 24 inch planewave and they have a large EL panel on the wall behind it. i think the panels exist, but they may be have to be made custom.

i guess all of your telescopes have refracting elements in the optical path, right? reflections are certainly possible, and the flats that adam describes would probably more closely match the reflections, if they are actually present.

rob

edit: you can see the panel in the opening seconds of this video, in the first large shed: https://www.youtube.com/watch?v=Q0bTLD_ci1A

[pfile]

oh - one more sanity check - i was reading your blog and you mention night vision cameras. hopefully the IR illuminators in those cameras are turned off while you are imaging?

what happens if you make a stack of uncalibrated lights? are the gradients still there? looking at your master flat it seems OK...

rob

[macnmotion]

Congratulations on being the only person who reads our blog.

The IR on our cameras is off - we're pretty diligent about that. We only use it at night to make sure our domes are opening and closing properly, or if there is some issue with the system and we need to take a look.

Based on the samples I posted, I'm at a loss as to why we get that pattern that kind of looks like shutter when our flats are 6+ seconds. It can't be the shutter, has to be something else. If I saw a hint of that pattern in the uncalibrated stack I'd feel better that it was light pollution but I don't see that.

[ngc1535]

First I want to thank you guys for taking the time to talk through this with me.

Indeed we are probably going beyond the realm of PI alone.

We hadn't considered night flats. I'll read up on this. Note that we are not using flat darks, we're using the same darks used for calibrating our lights.

This is fine- and PI will  basically find that for a 6 second flat there is no thermal signal to optimize and it will simply subtract the bias. Please oh Please... for the sake of this thread... post or make available your master dark (the one you used to calibrate this data). To improve the analysis, perform a 3x3 median filter (convolution) on it. Please.

This is a large scope, we haven't found any type of EL panel specifically made for this size (we do have EL caps we can use for our RH 300's, but not for our 400mm RiFAST and 600mm RiLA).

They exist. Likely I win the prize for the largest aperture in this thread with a 0.8m using what used to be called the Alnitak FlatMan XL

 We instead use an LED TV, and play a "white movie" on it through VLC on an attached computer. We cannot mount it on the dome, as the domes are clamshells.

Hmmm... isn't one of the shells the "inside?" Anyway, the lift thing works. You will find that the "white" you are getting from that LED TV is likely not going to characterize the night sky well and your flats will not be "perfect"... but that is a later consideration and not your larger problem at the moment.

 Instead, we have it on a mini lift to bring it up so that we can point our scope horizontally at the screen. We use a sheet of diffuser paper over the screen to help even out the lighting.

For uniform illumination of your chip- double scattering is almost always necessary. Two surfaces or diffusers. I used to hang milk plastic in front of the telescope while looking at another source of diffuse light. But this isn't a remote solution... it is an on-site thing.

 And we use brightness controls on the TV and in VLC to get it in range for each filter. We are targeting ADUs in the high 20 thousands (ranging from 27,000 - 29,000).

If anyone has any ideas on how to improve our flats within the limitations of our setup, we'd love to hear them. Especially if we could remotely control the process (really just the light source, as everything else is already automated).

The EL planel is probably best. Twilights are an option.. but they have other challenges.

That said, I did do some more sleuthing. I've found some answers, and am left with some questions. As I said, we have flats from both Jan 12 and Jan 20. All our subs were shot in between. I created Lum stacks from each of the four nights we shot, unregistered, so that any dust motes would add together and be easier to see. For each night I calibrated once with the Jan 12 flat, once with the Jan 20 flat. I found that I need to use the Jan 12 flat for subs shot on Jan 12 and 13, and I need to use the Jan 20 flat for subs shot on Jan 16 and Jan 18. That should at least solve most of the dust issues. Following are 4 comparison shots (all using PI boosted autostretch):

The issue is the chevron thing and dark bit in the top left of your images is not present in your flats. This nor does it appear in your raw data. I know I am not Sherlock Holmes... but darn it...I demand to see your dark! Geez (might as well reveal your bias as well)

I then took the Jan 18 subs and calibrated in different combinations of flats, darks, bias. Here is a labeled screenshot showing all combinations, as well as the master flat used during calibration.


All .xisf files for all of these comparisons are in this folder:
https://drive.google.com/drive/folders/1J3ZXsHzZXEezDu1Ok87y_cq2SgVdTGg8

As you can imagine we would like the best calibration solution possible considering our circumstances. We spend a lot of time on acquisition, and a lot of time on processing, it's a shame to be handicapped at all.

[macnmotion]

Hello ngc1535,

I've uploaded master dark and master super bias here:

https://drive.google.com/drive/folders/1u2PMvn6VScZtDnWpfQG9JOCKxEe249y0?usp=sharing

I don't see a Median process in PixInsight (although I see a description of it in the PI documentation). I ran a MorphologicalTransformation Median 3x3, not sure if that's what you meant. It's also in that folder.

We've been in touch with Alnitak several times this past year. They have discontinued their XL EL panels. At this point we are unable to find a source for a quality EL panel large enough for our purposes. All of the leaves of the clamshell move, and disappear behind the base, so anything mounted to a leaf would crash into the base. The base is low and would require our mount to point the scope below the horizon, which is past current safety limitations of the mount. Yes it can point down but we would have to disable the safeties. Building an easel that can hold an EL panel at the right height and angle isn't an issue, finding an EL panel is.

About the diffusers: Thanks for the suggestion to use two layers, we can try that. Also, is it possible that a crease on the diffuser might be causing this "chevron" angled gradient? I've been told that apparently the diffuser sheet was stepped on before it was installed leaving a crease that is within the FOV during flats.

Thanks again.

[ngc1535]

Hi Mac:

Ok. I don't see the issue (being a source of light) in your dark... but I did notice a couple of things...

1. That dark has the extension of FIT. Is there anything else that was done to it? I almost looks like a single dark frame that yes, is normalized (0-1)- but something is weird. The dark should look like the master bias and have roughly the same kind of ADUs... (since the dark is a thermal signal plus the bias and those hot pixels). But your dark is an order of magnitude SMALLER in values compared to the bias. It appears as if the darks were bias subtracted and this is a thermal frame?? This would not be the standard way to process data- because PI (via BPP or perhaps your own hands on method) is expecting a dark that it will subtract a bias from.  Looking at the FIT header I can see you combined 32 frames with the Linear Fit rejection method. Fine... but I think something happened before then? The difference in counts between your dark and flat needs to be explained. By the way, I am using the same chip.

2. Yes, I meant in MT...thanks.

3. A crease in a diffuser will not cause a feature like what you seeing. Your diffused sources of light are so out of focus... that black spots, holes...etc- really don't matter (with regards to this mystery anyway).

So, unfortunately I did not solve the mystery- but I wonder if can win a consolation prize if I am correct there is still something wrong with your darks.

Again, please let me know so I can exhaust all of my wrong guesses.

-adam

[bulrichl]

Hi Mac,

like Adam, I can also imagine that an incorrect calibration (preparation of master calibration files and/or settings in the light frame calibration) is causing all the trouble. Perhaps my guide will help: https://pixinsight.com/forum/index.php?topic=11968.0

Bernd

[macnmotion]

Hi Adam and Bernd,

I've been calibrating in PixInsight for a couple years the same way, using workflow from Light Vortex Astronomy. I do not use the BPP script. I do calibrate my dark subs with a master bias before integrating a master dark. But I do not recalibrate the dark master while calibrating the lights. There is no double calibration happening.

I am using the new local normalization process and including that data when I integrate my lights, perhaps that did something. However, in my many tests since that first run I just used Scale + Zero Offset for normalization.

In short, I'm using the same preprocessing steps I use on all of our data.

[pfile]

ah - LN can be dangerous. if the reference frame is not perfect, then all of your frames can be compromised by the bad reference frame. also if the scale is set wrong in LN artifacts can be introduced.

so is that chevron thing still present in the images integrated with Scale+Zero Offset still there? i have lost track.

rob

[bulrichl]

I've been calibrating in PixInsight for a couple years the same way, using workflow from Light Vortex Astronomy. I do not use the BPP script. I do calibrate my dark subs with a master bias before integrating a master dark. But I do not recalibrate the dark master while calibrating the lights. There is no double calibration happening.

For a pre-calibrated MasterDark it seems to look OK to me; nevertheless I generally don't recommend to pre-calibrate dark frames or the MasterDark - the reason is explained in the guide. In your MasterDark, 0.12 % of the pixels are clipped, but that is not relevant here.

Maybe uneven illumination by the light source used for flat frame acquisition is the reason? I would give sky flats a try, at least in order to exclude that flat frames are causing this issue.

I don't have experience with local normalization yet.

Bernd

[pfile]

OP has said that sky flats are not an option due to their equatorial location, but i agree that as a test it is likely a good idea. one should be able to get a few sky flats at dawn and dusk, enough to debug this problem anyway. if OP is using ACP for acquisition this is very easy to set up and run.

rob

[macnmotion]

Hi Rob,

Yes the chevron is still there when using Scale + Zero Offset (used for all the various test stacks during this debugging process).

We don't use ACP, we are now using Voyager. I'll take a look to see if Voyager can automate the sky flat process enough to get even a few flats for testing. Thanks.

[macnmotion]

Bernd, I've read your article (and will read others) regarding calibration of dark dubs before dark integration. I will consider altering my procedure to avoid possible clipping. Thanks.

[macnmotion]

I thought I'd bring this thread full circle and share the results. 420 subs totalling 28-1/2 hours of LRGB, from two scopes each with its own light pollution gradients :-)

Here is NGC 2683. I had to greatly modify my standard processing workflow to deal with a number of issues.