Poor results after applying master flat to my lights

[rtemen]

Hi, all.
Well, I have been viewing several tutorials on the process of calibrating and stacking.
In most cases, when the tutorial shows the resulting light after the masterflat has been applied, there is almost no vignetting left, and the background seems very even all across the field.
When I do these procedures, I get a horrible result. (At least I think they are horrible.)
I am attaching one of my lights, one flat, the masterflat and a calibrated result.
Could anyone look these over to see if there is a basic problem with any of these files?
Any suggestions on what might be wrong would be appreciated. Or if the files are ok, then what I might be doing wrong.
I have tried both Automatic and Dynamic background extraction and I frankly cannot see any difference in the results.
Thanks for any assistance.
Here are the files.
https://www.dropbox.com/s/xjn8f142tpifjuw/M63_FLAT_Tv13200s_800iso_00018.CR2?dl=0
https://www.dropbox.com/s/7hu8vficvc5sfnb/M63_LIGHT_180s_800iso_00008_c.xisf?dl=0
https://www.dropbox.com/s/chfsw2elwnta8sv/M63_LIGHT_180s_800iso_00008.CR2?dl=0
https://www.dropbox.com/s/dfg1sq1gnk0s0j2/MasterFlat.xisf?dl=0

[oldwexi]

Hi rtemen!
Tried to analyze your issue.
Cant reproduce exactly your workflow as during calibration you used
Master_Bias, Master_Dark and Master_Flat2 (!!!).
None of them are accessible for me on the dropbox.

However,
Find attached a screencopy to see what i tried with your submitted data.

Find in the first upper row at left a diagonal profile cut through
your single flat,  right beside the profile through my corrected Flat version,
and at most right a profile through your so called Masterflat  which is more or less unusable
because of the very dark corners.

In the second row at left is the calibration result  using your single flat and single light, you see because
of the flat beeing to dark you get bright corners.
right beside in the middle the result using the single "corrected flat" which gives a well calibrated light (c_3).
What did i do?  I added a pedestal of 0.08 to your single flat and saved it as f2.
(Above is the corrected flat profile cut and the PicelMath expression to correct your original flat)
In the second row most right is the calibration result when using your "MasterFlat". Clearly it has
to much bright corners as the corners in the Masterflat are to much darker than the centre.

In the 3rd row, at left your debayered result, in the middle my debayered result.

In the 4th row bottom left your original light, in the middle the histogram of the corrected flat f2.

Here the link to the screencopy
http://www.werbeagentur.org/oldwexi/flatcorr.jpg

Gerald

P.S.: For me it looks like you had different bright single flats which lead to the funny masterFlat.

[rtemen]

Hi, Gerald.
Thanks for looking in to these for me.
I wanted to let you know what I am using.
I have a C11 and my Canon T3i.
These shots were made after attaching my Meade Focal Reducer.
The pictures that I take have what I call a doughnut in the middle.
The very center of the shot is one shade. Then there is a torus(doughnut) around that which seems a little brighter, and then extremely dark area to the rest of the edges.
When I asked a few questions of the place where I bought the reducer, the answer was sort of like 'that is how they work and your flats will compensate for it'.
So far, I am not able to do anything, as you can see from my pics.

Are you by chance in the U.S. where we might chat on the phone?
I would probably understand some of your descriptions if you translated some of the terms since I am new to PI.

Rich

[oldwexi]

Hi Rich!
From my wording in the previous reply you can recognize that english is not my native language.
So, please apologize if my descriptions for the screenshot "what to do to improve the flats"
sounds confusing.

I am living in Austria/Europe so a normal phone call will cost a fortune...
therefore that wont be the solution.

What i can offer is:
You provide the missing MasterBias, Master Dark and MasterFlat2 and maybe a few more single flats
and single lights so i can produce a video which should explain step by step what i did to correct your
MasterFlat for better calibration. (but still in Austrian English)

As i have also bought once a Meade reducer i know what your are talking about, because
i use this piece of hardwarenow as a stand for my hot coffecup. Couldnt find another use which makes sense for me...
However, as you can see in the screenshot in the middle column in the second and third row,
it shows a complete flat calibrated raw image and calibrated debayered flat color  image.
So there should be an easy way (as shown)  to make your lights usable.

Gerald

[rtemen]

Hi, Gerald.
Well, here is a link to a folder with everything in it. The biases and the darks are not here, just their master versions.
https://www.dropbox.com/sh/s0tzxp3waa42o2e/AADNebu0Pt8rFc1ohUkODO0pa?dl=0
Thanks for offering to look at this.

Rich

[oldwexi]

Hi Rich!
Downloaded the stuff.
But, the Master Dark and MasterBias are debayered stretched RGBs. Therefore they are not usable
for proper calibration. 
Calibration has to be done on the raw mono CR2 files with mono master files. Not with stretched RGB calibration Master files.
The bias and darks are important because you have a very noisy camera
with thousand of warm/hot pixels which need to be removed.
Usually the warm pixels can be reduced a little with ImageIntegratioin if there are enough light images there.

So i need raw darks frames and raw biasframes . Also i need more of the lights for better clipping the
artefacts of the camera.

Anyhow, for the future you need to double the exposure time of the flats. They are to dark!
Guiding needs also to be improved very much.

I am trying my best, but the for me so far the existing material does not allow me to go deeper...

Gerald

[rtemen]

Hi Gerald.
WOW! I do not recall seeing any tutorial that tells me anything about using mono for bias and darks.
So this is all news to me.
Is there a tutorial that you know of that teaches how to do these things to get the proper bias and dark frames?

Rich

[rtemen]

I do not know how to make the bias and dark frames monochrome.
I set my camera to monochrome and take a picture.
When I open the file with PI it is in color.
I took a regular picture with my camera of my office.  It is monochrome when viewed on my camera.
After downloading my picture to my computer and use the Windows viewing programs, it shows monochrome.
When I open it in PI, it is color.

Please help???

Rich

[pfile]

setting the camera to monochrome does not do anything.

all CR2/NEF files are inherently greyscale before they are debayered. all you need to do is make sure that you are using "raw cfa" as a format hint in any process that is going to open the DSLR file.

if you want to open the files on the desktop as CFA/greyscale then go into the format explorer, double-click on DSLR_RAW and set it for "pure raw" by clicking the button at the bottom of the dialog box, then click OK.

rob

[rtemen]

Thanks, Rob.  That did the trick on my main test.
So, should all of my Image Integrations and Image Calibrations and final stacking in this pre-processing phase all be done in the non-debayered mode?
Then, during image processing, will things get changed back into color for the final picture processing?

Rich

[pfile]

that is correct: the flow for an OSC of any kind is calibration, debayer, register, integrate. calibration is done on the "raw" monochrome file and only after calibration are the images converted to 3-plane RGB files by the debayering process.

rob

[rtemen]

Rob.
My newbie status is going to show here big time.
First, what is OSC?
Second, I am not sure what a couple of your steps are and when they are done.
Here is what I have been doing for the past year.  Please tell me where to adjust and where de-bayering takes place.

I use ImageIntegration on my Biases to make a MasterBias.
I use ImageIntegration on my Darks to make a MasterDark.
Then I use ImageCalibration on my flats to apply the MasterBias and MasterDark to the flats making my calibrated flats.
I use ImageIntegration on the calibrated flats to make the MasterFlat.
Then I use ImageCalibration on the Lights applying the MasterBias, the MasterDark, and the MasterFlat.
Then I use Star Alignment on the calibrated lights to make registered lights.
Then ImageIntegration on the registered Lights for the final integrated file.

So, that all said, how does this match with your statement of "calibration, debayer, register, integrate"?
Are all of my steps above included in your calibration step before debayer?

Thanks for your assistance.

Rich

[pfile]

OSC = "one-shot color" which is a term that describes any CCD capable of taking a color image with one exposure, that is, sensors with bayer color filters applied above the sensor. besides DSLRs there are dedicated astro cameras that use bayered sensors (like the STF-8300C).

inbetween "Then I use ImageCalibration on the Lights applying the MasterBias, the MasterDark, and the MasterFlat." and "Then I use Star Alignment on the calibrated lights to make registered lights." you need to debayer the lights. you can use the script called BatchDebayer to do a bunch of calibrated lights at once.

so then you need to resume your flow from StarAlignment, but choose the debayered lights. then use ImageIntegration on the registered and debayered calibrated lights.

rob

[rtemen]

Thanks, Rob.
I'll work on this in the morning.

Rich

[oldwexi]

Hi Richg!
Please note,  before you dont have the raw mono darks and raw mono bias frames
everyrything else is future music.
Also you need more than 5 lights to get rid of the thermal noise (warm pixels) left after the
calibration.

So first step - make in minimum 10 darks, store them on the dropbox.
Second step - make 10 bias and store them on the dropbox. (The bias are only needed for the flatcorrection)
Third step -  put another 10 lights on the dropbox.

After this -  you and we can try to go further. Otherwise it makes no sense

because the trailing warm pixels in the stacked image will destroy the whole image.
Trailing pixels because you did not guide the images so the objects move during exposures
from lower left to the upper right. Which gets you also very much elongated stars...
Here a resampled example how your stacked data look like when the dark do not fit
http://www.werbeagentur.org/oldwexi/trailing%20leftover%20of%20warm%20pixels%20after%20stacking.jpg

So, lets have step 1 - 3 done than we can see what can be done to get more out of your M63 images.

Gerald