Splotchy background after Batch Preprocessing and DBE or ABE

[sharkmelley]

I’ve used the up-bottom choice in Batchpreprocessing (I already was, that is). When I turned it off and left the debayering at RGGB, I got a very BLUE image, and the darks didn’t do their thing, so that wasn’t a good choice. I also switched it to GRGB and got the same as my original result. No luck here.

I think we need to take a look at your FITS files.  If you use Google Drive to upload one each of the light, dark, flat and bias files then I'm sure we can work out the correct settings.

Mark

[alanrock]

Here you go--thanks in advance!
https://drive.google.com/open?id=15wKi7OebRUzOuz-_5utSH3yHyFzlEWbL

[msmythers]

Just wanted to take a look at the frame 8 light sub for screen door effect. All I did was debayer(RGGB,VNG), ABE,PCC,HT and curves with mask. I didn't try hard with ABE since this was just a single frame without calibration as the amp glow on the right shows. Colors look fine and no screen door effect.

I'll let others talk to the calibration.



Mike

[pfile]

same flow here, RGGB/superpixel + up-bottom, no problem debayering; no screen door, color seems correct. maybe there is a problem with the calibration?

rob

[alanrock]

OK, so maybe we should talk about my settings in Batch Preprocessing. Do I need to use the same rejection algorithm for Darks, lights, Bias and flats, or does it depend on the number of files for each? I'm using Winsorized Sigma Clipping for everything although I do not have the same number of files for everything (obviously). Also, I am using Sigma High and Sigma Low of 3.0 for everything but lights--that was different. Any other settings I should be looking at for things to adjust? Would it be helpful to share a log file?
Thanks again, hopefully we can figure this out!
Here's the log file: https://drive.google.com/open?id=1vjlNwSsflnRfZuQaD1tXZ3HIfAKS2Csp

[pfile]

i guess what would be instructive is if you posted one of the _c_d and _c_d_r files from the batch preprocessing folder, that would show whether or not BPP did the right thing along the calibration and registration steps.

as long as you have enough frames (~10 - ~15) windsorised sigma clipping should be OK for all frames. 3.0 is pretty reasonable for Hi/Lo sigma. if you open the masters that BPP creates i think it will show the rejection maps and you can see if too much was rejected (or not enough)

rob

[alanrock]

I uploaded a c_d and a c_d_r file into the shared folder. Interestingly, when I open the c_d_r file and do an ABE on it, it does NOT look splotchy all. What do you think that means?
The files are in this folder: https://drive.google.com/open?id=15wKi7OebRUzOuz-_5utSH3yHyFzlEWbL

[pfile]

well so the _c_d and _c_d_r files you posted appear to be undebayered. although of course they should be, or at least the script thinks it attempted to debayer them or else it would not give the suffix "_d".

one thing that strikes me as odd is that the amp glow in the dark file appears on the left side of the frame, while it appears on the right in the light frame. this could indicate some incompatibility between the calibration frames and the lights with respect to how they were written out to disk (up-bottom vs. bottom-up). the filenames for the flats/bias/dark look like they were generated by Maxim to me. if so, that's likely your problem right there - maxim is writing out the fits files differently than SGP and so they are not compatible with one another (though i suspect with some massaging you could make them compatible.) so it could be that the files are debayered, but because the bayer matricies are reversed between the calibration frames and lights, the debayered data is coming out looking weird. in particular i can see that in the flat that the green channel is much weaker than the other two channels, which might end up leaving a checkered imprint even on the debayered data.

anyway one way to test this theory is to debayer, align and integrate the lights without calibrating them and see what you get. i suspect it will work fine and you'll need to recapture your calibration frames.

rob

[alanrock]

When I look at the c_d_r file though, it's quite green which would seem to indicate it's been debayered, wouldn't it?
I'll have to look at my calibration files again though. I could've sworn they're all created in SGP, but when I look at the FITS header, it says Maxim for some of them. I probably shouldn't mix and match them.
And, if the c_d_r file looks fine (not splotchy), wouldn't that seem to indicate that it's the step after that that's the problem? And the step after that is...?

[pfile]

c_d_r (and c_d) are indeed green but the tell is the checkerboard pattern when you zoom in... that's not normal. ordinarily this is what an undebayered file looks like but in this case i think it's because the dark (and more importantly the flat) bayer pattern is reversed from the lights when read in to PI.

rob

[pfile]

well now i'm pretty confused - for fun i used your flat/dark/bias subs to calibrate the light sub you posted and it's coming out "fine" in the sense that the debayered file looks proper, with no screen door effect. so either your calibration masters are really different from the subs you posted, or my theory about the flat is wrong.

if you can manually calibrate and debayer the images OK with the 3 masters indicated in the log, then all i can guess is that for some reason, the BPP script has failed to debayer the images, despite calculating a filename that indicates it was debayered. i have never seen that before though.

but for sure the dark sub you posted does not match properly since the amp glow does not match.

rob

[alanrock]

I just started to run the BPP again using the darks that were created in SGP, even though the only flats I have were created in Maxim, and I noticed that the last time I ran it I had the debayer set to BGGR, probably because I've tried so many different things. Ugh. Anyway, I'm running it again right now even though the flats are from Maxim. When it's done, I'll repost those files.

[sharkmelley]

I uploaded a c_d and a c_d_r file into the shared folder. Interestingly, when I open the c_d_r file and do an ABE on it, it does NOT look splotchy all. What do you think that means?
The files are in this folder: https://drive.google.com/open?id=15wKi7OebRUzOuz-_5utSH3yHyFzlEWbL

On the contrary, your a c_d and a c_d_r files DO have that same grid pattern.  Look at them at a scale of 4:1 (not 1:4) and you'll see the pattern.  For the avoidance of doubt, I'm looking at the files ngc7320_600sec_1x1__frame14_c_d.xisf and ngc7320_600sec_1x1__frame14_c_d_r.xisf.

However, your light file (ngc7320_600sec_1x1__frame8 copy.fit) and flat file (Flats-0015F1 copy.fit) provide the key to this mystery.  These two files are not compatible i.e. they have not been created with the same settings.  The CFA pattern RGGB works fine with the light frame but not with the flat frame.  The flat frame seems to need GBRG or GRBG.  The problem is that when you calibrate the light frame with the incompatible flat frame you end up with a mess that no single CFA pattern can possibly debayer successfully - you will always have a grid pattern.

In addition, if you examine the dark frame the "starburst" amp glow is on the left hand side but in the light frame it is on the right hand side.  So the dark frame is also definitely incompatible with the light frame.  With the bias frame it is impossible to tell.

You need to re-examine your acquisition of light frames and calibration frames to determine what has caused this left/right flip.  Were these files all shot at the same time with the same software?  Did you change some settings somewhere?

Mark

[alanrock]

So it sounds like I need to resolve this screen door effect before I need to even bother caring about the splotchy effect I was talking about in the beginning, is this right? As previously stated, my flats were created with Maxim, but my bias, darks, and lights were created in SGP. When conditions permit, I'll redo my flats in SGP (I'm using a Hyperstar on a C9.25, so I cannot just create flats any time I want to-conditions need to be right). When I zoom in on a light, I see a checkerboard effect which I assume is normal, but when I zoom in on a c_d_r file, I guess I'm seeing the screen door effect. Check out the attached files. I guess I shouldn't bother wasting my time until I get new flats done, does that sound right?
And to reiterate, the screen door effect and the "splotchy" overall look are two different issues, although they may be related.

[alanrock]

Screen?