Ha Calibration Woes . . .

[Terry Danks]

I am experiencing problems calibration all Ha frames.

Two things . . .

1/ Only Ha frames are giving me grief. LRGB calibrations are fine, and . . .
2/ I went back and redid some Ha frames I just calibrated a few days ago with no problems but now they come out terribly!

This is on two machines. Same "new problem" on both.

Attached is a screenshot with the "good" calibration on the left and the "bad" on the right. Default STF settings on both. All backgrounds in calibrated Ha frames are now clipped, or almost clipped.



I have used the Batch Preprocessor, set to "Calibrate only. I have also tried producing new flats, darks and bias frames in case my masters had somehow become corrupted, but I still get these very dark calibrated frames.

To my knowledge, I have not changed any settings on either machine but am at a loss as to what is causing this sudden misbehaviour.
Any suggestions?

[pfile]

try a couple of things -

1) using ImageCalibration, just use the master dark (no bias)
2) if that works fine, there's some problem with the bias (for some reason the ADU values in the bias exceed the ADU values in the dark)
3) if that doesn't work fine, try adding an output pedestal when calibrating the lights. (the ADU values in the dark exceed the ADU values in the light)

rob

[Terry Danks]

Many thanks for helping me out yet again, Rob.

Adding a pedestal (1000!) worked. The subs though have a tremendous gradient (see attached).
DBE pretty much wipes that out but this all leaves me puzzled as to what in hades I've done to cause this to be necessary?
It also means abandoning the Batch Preprocessor completely and calibrating and applying a defect map manually.  (Poor me!  )
As I stated in my first post, this is the first time any of this has happened and there must be something I've done to cause it. Same Ha frames 4 days ago were fine going through the Preprocessor and having it apply the defect map.

[pfile]

well sometimes what happens with narrowband exposures is that there's just no sky signal in the background, so the background is essentially the same as a dark. in that sense what's happened to you is normal - its like subtracting a dark master from a dark sub, so you end up with some pixels with negative values, which get clamped to 0.

but, that gradient is weird. i wonder if there is some problem with the darks.

is this a CMOS camera? if so is it possible the gain is different between the lights and darks? or maybe there was a driver software change between the lights and darks?

rob

[ngc1535]

How about a simple light leak during the acquisition of the darks?
(Darks taken in the day, next a computer monitor at night... etc etc).
The question is easily answered. Take the master dark and apply a median 3x3 filter. (it will help to visualize things).
Is the gradient in the dark?
-adam

[Terry Danks]

Many thanks again, Rob and thanks too, Adam.
Camera is an SBIG STL CCD.

I have now involved three machines. Two running the latest versions of PI. These two both have this weird problem.

The third machine has "Stone Age" PI on it, v.01.08.01.1079. Obviously, this machine is rarely used any longer for anything, but still works.

First problem with it was that it predates .xisf. Furthermore, PI's Batch Format Conversion will not convert my .xisf masters to .fits. So I loaded the original calibration subs (.fits) into the "Stone Age" version of PI and . . . voila! The Ha calibration seems fine!

Now, I could have somehow, inadvertently messed up a setting in my main machine, thereby causing this. But I see no likelihood of similarly messing up my second machine?

And calibration is fine on the third machine, the one with the ancient version of PI.

As I've stated earlier, I've already tried producing new masters on the two machines running the latest PI version in case my masters somehow "tuned bad," but get the same lousy calibrations after doing so. But all is fine on the machine running the ancient version of PI.

So, I'm flummoxed. 

[pfile]

are the light/dark/flat subs i16 or are they some other format?

rob

[Terry Danks]

Sorry, Rob, but I don't understand the question. "i16?"

I took the subs, flats, bias and darks directly as they came from the SBIG STL into the old version of PI and all is well.
I believe all are 16 bit ".fits" files.

Taking these very same files into the latest version of PI produces the wonky calibration.
Attached is a single sub put through both the old PI and the latest version. No CC on either and default STF for both.

[pfile]

sorry, yeah, i16 = 16-bit integers (it is how PI refers to that format). was just curious if they were floating point images as lots can go funny there, but not so with i16.

anyway i've never seen a regression in ImageCalibration so what's happening is kind of strange...

rob

[ngc1535]

Sorry, Rob, but I don't understand the question. "i16?"

I took the subs, flats, bias and darks directly as they came from the SBIG STL into the old version of PI and all is well.
I believe all are 16 bit ".fits" files.

Taking these very same files into the latest version of PI produces the wonky calibration.
Attached is a single sub put through both the old PI and the latest version. No CC on either and default STF for both.

Can you post a picture of the dark (the master dark, the one that is being used in calibration) that physically resides on your old machine?
Next to it, post the dark (the master dark, the one that is being used in calibration) that physically resides on your problem machine.
-adam

[Terry Danks]

Hi Adam:

Attached is a screenshot of the two master darks.
The .xisf is, of course, produced on one of the machines running the latest version of PI.
The .fits master was produced on the older machine that still gives me good calibrated Ha subs.

[STEVE333]

I'm out of my depth here, but, I did notice that if you vertically mirror one of the images (fits or xisf) they look quite similar.

Just sayin ...

Steve

[pfile]

ok so likely the fits reader direction is different between the machines?

i didn't focus on this initially... since on a given machine as long as the masters stay on that machine everything should be consistent. but i guess if you move any fits files between machines/instances of pixinsight with different settings then you can run into this problem. this was one of the reasons that juan developed XISF.

rob

[ngc1535]

Hi Adam:

Attached is a screenshot of the two master darks.
The .xisf is, of course, produced on one of the machines running the latest version of PI.
The .fits master was produced on the older machine that still gives me good calibrated Ha subs.

OK. So I think we can conclude that the calibration process itself (that is adding, subtracting..etc) is not the issue on the old or new machine.
The issue deals with the reason the darks are incompatible with the data.
The fact that you have two distinctly looking darks is a pretty big issue (which is why I gently tried to pull you in this direction.)

My guess is that Steve is correct. These are likely identical darks if you did create them from the same files. I am assuming this is a master dark. If it is a single dark- then certainly there is a geometric transform (rotation, mirror...) in play.

Unfortunately it is degenerate to determine whether the axis/origin is being affected on either the darks or the actual Ha data. The confusing part is that your calibration problem on the new machine should happen for all data... not just the Ha.

Conclusions:
1. Your Darks on the two machines do not match. This is very odd and the explanation is the answer to your problem. It is as if there is a mirroring or rotation of 180 degrees. I can't tell with this small images based on the hot pixels which it is.
2. I have seen problems like this when people use a GEM for acquiring data. At some point data is rotated due to a flip- and this breaks the correspondence with the dark/bias/flat.
3. I guess I should explicitly say that biases and darks have fixed patterns in terms of that gradient (that is what is giving away the problem). I will go a step further and say that the XISF dark is likely in the wrong orientation. Gradients for all STLs I have used are darker at the top and brighter towards the bottom. Heh...  I guess it pays that I have that much experience eh?

You should be able to produce a master dark in XISF that is in the correct orientation and will calibrate your Ha. If this same dark DOES NOT calibrate your other LRGB data..it means *that* data is rotated/mirrored..whatever.

Rob: You didn't consider it..but I did. lol this is why I wanted to see a comparison..I did not want to assume (correctly) that the darks were the same.
 

[pfile]

2. I have seen problems like this when people use a GEM for acquiring data. At some point data is rotated due to a flip- and this breaks the correspondence with the dark/bias/flat.

uh, too much time on big AltAz research telescopes??? using a GEM won't break correspondence between biases and darks - nothing the telescope, camera or the mount can do will ever do this. the only thing that can break this correspondence is reading the file in backwards relative to how it was written. the sensor is the sensor and it doesn't matter how you hold the camera, the data comes out the same.

and just a meridian flip can't break correspondence with the flats. the only thing that will do that is if you use a camera rotator to unrotate the field after the meridian flip. then you need new flats since the camera to telescope orientation is different.

the flips that are the result of registration should take place after calibration; you're not going to rotate the lights first and then apply darks and biases to them, unless one has seriously misunderstood how the calibration pipeline is supposed to happen.

3. I guess I should explicitly say that biases and darks have fixed patterns in terms of that gradient (that is what is giving away the problem). I will go a step further and say that the XISF dark is likely in the wrong orientation. Gradients for all STLs I have used are darker at the top and brighter towards the bottom. Heh...  I guess it pays that I have that much experience eh?

You should be able to produce a master dark in XISF that is in the correct orientation and will calibrate your Ha. If this same dark DOES NOT calibrate your other LRGB data..it means *that* data is rotated/mirrored..whatever.

Rob: You didn't consider it..but I did. lol this is why I wanted to see a comparison..I did not want to assume (correctly) that the darks were the same.

as i said, the problem is going to be the fits reader direction - the XISF dark would be in the wrong orientation because the fits reader direction did not match between writer and reader when the darks were read in to be integrated. which is OK of course if *all* the frames (calibration and light) are read in with the same fits reader direction, and were all written with the same fits writer direction... but somehow along the way, something must have changed in terry's case such that the lights have been read in with the opposite fits reader direction from the calibration frames.

rob