Ha Calibration Woes . . .

[Terry Danks]

Thanks guys. I am getting confused. I DID notice in my previous post that the master darks from the new vs the old versions of PI were of differing orientation. (Never though I'd actually be glad my STL developed its first column defect last summer!)

But I don't know why this is only a problem on the two machines that are running up-to-date versions of PI and not on the old machine running a very old version?

Also, I tried flipping the dark and re calibrating. Sorry to say I got the same result, i.e., horribly dark, clipped calibrated subs.
I have attached two panels. One with the master dark as produced by the latest version of PI compared to a single sub and another run with the master flipped to match the sub orientation.
Same poor result for each effort.

Finally, I checked and see that the master dark produced on the new machines for my 10 min LRGB subs is also flipped when compared to the lights . . . but my lights calibrate well.
Totally confused. 

[ngc1535]

2. I have seen problems like this when people use a GEM for acquiring data. At some point data is rotated due to a flip- and this breaks the correspondence with the dark/bias/flat.

 you're not going to rotate the lights first and then apply darks and biases to them, unless one has seriously misunderstood how the calibration pipeline is supposed to happen

>> This is exactly what some people inadvertently do. This case can be distinguished from the flipping of the axis which is a mirror and not a rotation. It is easy to accidentally align images before calibrating. Many people do not know the pattern of a dark/bias is baked in...and that the orientation matters (which is what I was trying to point out by displaying the darks). Terry should have enough information now to report back what the answer was/is.

3. I guess I should explicitly say that biases and darks have fixed patterns in terms of that gradient (that is what is giving away the problem). I will go a step further and say that the XISF dark is likely in the wrong orientation. Gradients for all STLs I have used are darker at the top and brighter towards the bottom. Heh...  I guess it pays that I have that much experience eh?

You should be able to produce a master dark in XISF that is in the correct orientation and will calibrate your Ha. If this same dark DOES NOT calibrate your other LRGB data..it means *that* data is rotated/mirrored..whatever.

Rob: You didn't consider it..but I did. lol this is why I wanted to see a comparison..I did not want to assume (correctly) that the darks were the same.

as i said, the problem is going to be the fits reader direction - the XISF dark would be in the wrong orientation because the fits reader direction did not match between writer and reader when the darks were read in to be integrated. which is OK of course if *all* the frames (calibration and light) are read in with the same fits reader direction, and were all written with the same fits writer direction... but somehow along the way, something must have changed in terry's case such that the lights have been read in with the opposite fits reader direction from the calibration frames.

I agree- if a flip of axis is shown. Just need to see the two darks (or other data) and say. The answer is right there.

rob

[pfile]

at this point it's hard to know if your masters are flipped or not, or how they got that way. probably what you should do is just upload a few dark subs and a few bias subs (straight from the camera) with a Ha light sub and someone can take a look.

rob

[bulrichl]

Compare the 'Coordinate origin' setting of the FITS module on the two computers: double click on Format Explorer/FITS. Are these settings different? To make things even more complicated: the global setting will be overruled by the corresponding Format hints if used.

When using the BPP script, the 'Global options' setting 'Up-bottom FITS' is decisive.

It is important to understand that these settings makes an impact only when loading a FITS file. If 'Upper left corner (up-bottom)' is selected, the orientation of the data will not be altered. If 'Lower left corner (bottom-up)' is selected, the orientation of the data will be mirrored vertically. This is done during loading of the FITS file.

If you use different 'Coordinate origin' settings on the two computers, a (dark, bias or flat) master file in XISF format that was produced on the other computer will not be suited to calibrate your FITS files.

The 'Coordinate origin' setting should ALWAYS be so that the resulting images are true sided. The most simple test for this is when you take one image of text and check whether the text is readable in the image. Note that mirroring vertically is not the same as rotating by 180 degrees: by mirroring the side-correctness will be changed, by rotation not.

Bernd


P.S. There seems to be some linguistic confusion as well. I don't like the term 'flipped' at all. Better use 'mirrored' and 'rotated' in order to differentiate between these operations clearly.

[Terry Danks]

I'm out of my depth here.
I "went back to basics." By that I mean . . .

1/I uninstalled PI v. 1457.
2/ I reinstalled a previous version, v. 1448.
3/ I loaded the BPP with .fits files as they come from the camera, for all frames, lights, darks, flats and bias frames.
4/ I then ran the BPP twice. Once with the "up-bottom FITS" box checked and again with it unchecked.

Same result each time. It was evident, from the break in the column defect, that checking/unchecking the "up-bottom FITS" box DID "vertically mirror" the resulting master dark. New masters were created on each run. Still, the resulting calibrated sub looked the same, i.e. severely clipped.

Attached is a single calibrated Ha frame, with the histogram displaying the stretch from the STF default. Also attached is the same single un-calibrated sub with its histogram from the STF default.

Not sure what to make of Bernd's suggestion? After I double click on Format Explorer/FITS. I attach that panel also. That panel is identical to that on both the machine with the current version of PI and the old machine with an ancient version.

[ngc1535]

Terry,

As I recall... you said that on the "new" machine you were able to calibrate LRGB images with darks- but not the Ha.
If this is correct...

1. Show the dark frame you used with a raw L (or R or whatever) frame that you know calibrated properly, and a raw Ha frame that did not (we have seen quite a few of them...but just for completeness). I am assuming this one dark was used to calibrate both data (Ha and L). This would be helpful.
2. In addition, you really need to make the files available to be examined, there isn't enough information in the screen shots. Looking at the FITs headers will also be informative.

-adam

[Terry Danks]

Hi Adam:

Yes, it is correct that my LRGB calibrations are fine but, no, it is NOT correct that the same darks were used to do so. The LRGB subs are 10 minutes duration. The Ha subs are 30 minutes. So, different darks.

But we have moved well beyond those anyway as the problem persists both with . . .

Totally new master darks integrated from fits files as they come from the camera and
A fresh install of PI. In fact, two fresh installs, v. 1448 and 1457.

I have uploaded the master calibration files to Dropbox. Also one of the uncalibrated fits lights, and the lousy calibrated file that resulted from the run.
Link . . .
https://www.dropbox.com/sh/26f9mbae6ccpt48/AAAaaC5h8plLwv93P7DQz9lga?dl=0

[ngc1535]

Terry,

I will try to look at the data later...
But regarding the darks- you can scale a single dark (optimize) to calibrate data of different exposure times. In this way you can eliminate one more variable.

It would be good to upload a "GOOD" dark with a raw data file that you know calibrates properly.

-adam

[ngc1535]

Just a quick note.
Did you notice in your FITs header for the your M82 image- it says "Is the image flipped." TRUE ?

I thought you were providing the raw data?

-adam

[bulrichl]

Terry,

referring to your uploaded images:
1) Ha Light:   median 943.0
2) MasterBias: median 928.8
3) MasterDark: median 983.6

Master files are not mirrored compared with Ha light frame. However note Adam's comment about the FITS keyword 'FLIPPED'.

As the median of the Ha light frame is less than the median of the MasterDark, it is essential to use an output pedestal. Calibration using the ImageCalibration process works (no MasterBias, MasterDark: don't calibrate, don't optimize, MasterFlat: don't calibrate) with the maximum possible output pedestal of 1000 seems to work fine, leaving 2941 pixels that have an intensity of zero.

I added a vertical plot of column 2004 of the calibrated Ha light frame. There is no vertical gradient in the calibrated image (see Vert_Plot_col_2004).

I suppose that calibrating your Ha images this way will result in a nice image after integration, despite the pixels with zero intensity.

Bernd

[pfile]

so we're back at square 1, which is good i guess.

juan would have to comment if the FLIPPED keyword has any effect in PI, and if it does, if the behavior changed from release to release.

rob

[bulrichl]

so we're back at square 1, which is good i guess.

Agreed 

juan would have to comment if the FLIPPED keyword has any effect in PI, and if it does, if the behavior changed from release to release.

I gave it a try and changed the value of 'FLIPPED' from "T" to "F" with a Hex editor, saved and opened the modified image in PI. There was no change of the representation of the image in PixInsight v1.08.06.1457. Cannot test older versions of PI right now.

Bernd

[ngc1535]

Terry,

referring to your uploaded images:
1) Ha Light:   median 943.0
2) MasterBias: median 928.8
3) MasterDark: median 983.6

Master files are not mirrored compared with Ha light frame. However note Adam's comment about the FITS keyword 'FLIPPED'.

As the median of the Ha light frame is less than the median of the MasterDark, it is essential to use an output pedestal. Calibration using the ImageCalibration process works (no MasterBias, MasterDark: don't calibrate, don't optimize, MasterFlat: don't calibrate) with the maximum possible output pedestal of 1000 seems to work fine, leaving 2941 pixels that have an intensity of zero.

I added a vertical plot of column 2004 of the calibrated Ha light frame. There is no vertical gradient in the calibrated image (see Vert_Plot_col_2004).

I suppose that calibrating your Ha images this way will result in a nice image after integration, despite the pixels with zero intensity.

Bernd

Something else from my personal experience. The fact that the dark has a higher level than the data frame of the same time hints at an issue.
The STL has significant RBI. If a dark is taken immediately after flat field images (for example)- this can raise the effective pedestal of the dark. I would not expect this to be an issue with a master dark that is combined - since the RBI does eventually fade. But just for a future  note Terry... do not take darks immediately after light frames if possible.

The other bit of information I cannot discern from the master dark and the data- is the correspondence of the temperature. I wouldn't expect this to be the issue- but it is certainly something to check.

-adam

[Terry Danks]

Well, I'm learning stuff here for sure.

Regarding the flipped notation. That puzzled me as it was my intent to upload a file as it came from the camera.
I am imaging as I write this and it seems all my subs, as they come from the STL, now have Flipped=T! I went back and looked at some from a couple of years ago and they had Flipped=F.
When this happened, I have no idea. And I have no idea as to why it happened either. Or if I should do anything about it?

I was not aware that I could use a 10 minute dark for a 30 minute sub. Do I understand correctly that you are telling me that is what "optimize darks" actually does. It would be great to not have to take 30 minute darks for my NB!

All my subs, including darks, flats and bias frames are taken at -20C.

[ngc1535]

Concerning the scaling of darks...technically if you have characterized the dark current well- it is true that you can use a 10 minute dark to calibrate a 30 minute exposure.
HOWEVER, scaling in that direction multiplies the errors. And given PI's noise analysis for determining the factor, I suspect it is asking for trouble. It is far better to, say, create 30 minute darks..and then use a master to calibrate all exposures equal to or less than it.
There are some side effects of using this strategy.

1. Benefit: You can use a master dark to calibrate data taken for significant periods of time (read as months). Regenerate master biases more frequently.
2. Con: Hot pixels will not not subtract away nicely (due to either exposure differences since their "dark current" may not be linear, or due to changes over time). The combination of cosmetic correction and dithered data with proper rejection during image integration will result in clean images.

This has been my mode of operating for many years. Sure, a single time matched dark will get rid of the hot pixels... but *this is not the important part!*

-adam