Help With Gradient Analysis

[Giddy]

It is almost certainly modulating the lights by PWM (fixed frequency, variable duty cycle). You may be able to see this by simply waving the tablet around and noting the pulsation. This is a primary reason to try and get your exposure time up on the order of a second, because then each pixel will average out to the same (large) number of cycles.

Got it. Given the time involved in making mistakes, would you mind confirming this is a reasonable workflow?

Anytime:
- 100 bias, ISO to match lights, fastest exposure time (once per ISO, save for future use)

Just before lights:
- 40 flats, ISO 100, light adjusted so exposure is 1-2s (save settings and replicate each session)
- 100 bias, ISO 100, same exposure as flats (one time, save for future use)

Darks
- No darks

If this is correct, what would the stacking process be? Do I need to stack the flats separately, or will WBPP be able to use the correct bias files for lights and flats?

If it's not correct... and I hate to be so needy, but I'm getting a bit lost... could someone please just tell me exactly what to try and I will do it.

 

[cloudbait]

Got it. Given the time involved in making mistakes, would you mind confirming this is a reasonable workflow?

Anytime:
- 100 bias, ISO to match lights, fastest exposure time (once per ISO, save for future use)

Just before lights:
- 40 flats, ISO 100, light adjusted so exposure is 1-2s (save settings and replicate each session)
- 100 bias, ISO 100, same exposure as flats (one time, save for future use)

Darks
- No darks

If this is correct, what would the stacking process be? Do I need to stack the flats separately, or will WBPP be able to use the correct bias files for lights and flats?

If it's not correct... and I hate to be so needy, but I'm getting a bit lost... could someone please just tell me exactly what to try and I will do it.

Not an expert on using "photography" cameras for astroimaging. If you're not using darks (and you should test to see if you don't need to by confirming that bias frames and long exposure dark frames look the same) then the ISO for the bias frames should match that for the flats and lights. If you use darks, the bias ISO only needs to match the flats. You may or may not need flats every session, depending on the stability of your system. If the optics stay sealed, flats can be stable for a long time.

WBPP can manage everything. If you have calibration frames over multiple nights that have to match lights over those nights, you'll need to use its keyword system to make sure the right things go together.

 

[Giddy]

Not an expert on using "photography" cameras for astroimaging. If you're not using darks (and you should test to see if you don't need to by confirming that bias frames and long exposure dark frames look the same) ...

Yes, my entire week has been consumed by trying to test exactly that. I was told on CN forums that with a "photography" camera I shouldn't use darks. Since the camera cannot be temp controlled, the darks do more harm than good, or so I was told. Since all my research said darks would lead to better results, I figured I would be scientific and try and test it.

My goal is to find a workflow that is "best" for what I have and then compare using darks with not using darks. I'm just having trouble getting to a good baseline. I also do not know how to compare results. For example, which of these is "better":

Amazingly addictive, fun, but frustrating hobby. Reminds me of golf in many ways.

 

[fredvanner]

Before testing on images, there are simple checks that you can do just by comparing your darks with your bias frames. Find somewhere where you can keep the camera at an approximately constant temperature, then capture a set of bias frames (they are cheap, so I usually take 50). Then take some darks at exposures that you are likely to use especially longer exposures. For this test you only need to take three or four at each exposure (so that you can check them for consistency). Integrate your bias frames to construct a masterBias.
Then, in PixInsight, check the masterBias (mB) against the values in Dn (an n second dark) for all the different values of n that you have captured. If the values of Dn are not virtually identical to mB, you probably need to darks. If the median value of the Dn increases systematically with n, you definitely need to use darks. I have never seen a photographic camera that doesn't need darks (but then I've only had a couple, and they are not exactly cutting edge). My low dark current cooled CMOS ASI2600MC can get away without darks at short exposures, but I still capture darks for most exposures longer than 60s.

 

[Giddy]

Before testing on images, there are simple checks that you can do just by comparing your darks with your bias frames. Find somewhere where you can keep the camera at an approximately constant temperature, then capture a set of bias frames (they are cheap, so I usually take 50). Then take some darks at exposures that you are likely to use especially longer exposures. For this test you only need to take three or four at each exposure (so that you can check them for consistency). Integrate your bias frames to construct a masterBias.
Then, in PixInsight, check the masterBias (mB) against the values in Dn (an n second dark) for all the different values of n that you have captured. If the values of Dn are not virtually identical to mB, you probably need to darks. If the median value of the Dn increases systematically with n, you definitely need to use darks. I have never seen a photographic camera that doesn't need darks (but then I've only had a couple, and they are not exactly cutting edge). My low dark current cooled CMOS ASI2600MC can get away without darks at short exposures, but I still capture darks for most exposures longer than 60s.

Thanks! I think I get all of that. I'm taking the images now. That definitely seems like a much easier test... I'm about 50 hours in to this, lol, so clearly I don't mind investing a little time in finding the "right" answer.

When you wrote "check the masterBias (mB) against the values in Dn", what values do you mean? Is that the Statistics process or something else?

 

[fredvanner]

Is that the Statistics process or something else?

That is certainly the first check to do - particularly comparing the median and the MAD values (are the images essentially the same level and the same spread of values). A more detailed check is to look for variation across the image. For this you use PixelMath. Applying the expression:

abs(Dn-mB)

will generate an image with the absolute value of the difference between the dark and the masterBias. This should be very close to zero (so will need an STF, possibly a boosted STF), and should have no pattern across the image. If it appears that the darks are systematically higher signal than the mB, then applying:

Dn-mB

will generate an image with the value of the difference between the dark and the masterBias. This should be roughly uniform across the image. If the median value for these subtracted images increases linearly with n (the duration of exposure), it is firm evidence that dark calibration is required.

 

[Giddy]

That is certainly the first check to do - particularly comparing the median and the MAD values (are the images essentially the same level and the same spread of values). A more detailed check is to look for variation across the image. For this you use PixelMath. Applying the expression:

abs(Dn-mB)

will generate an image with the absolute value of the difference between the dark and the masterBias. This should be very close to zero (so will need an STF, possibly a boosted STF), and should have no pattern across the image. If it appears that the darks are systematically higher signal than the mB, then applying:

Dn-mB

will generate an image with the value of the difference between the dark and the masterBias. This should be roughly uniform across the image. If the median value for these subtracted images increases linearly with n (the duration of exposure), it is firm evidence that dark calibration is required.

Thanks so much! That will be helpful. I just got done taking the darks so I'll start analyzing them.

In the meantime, to return to my second image issue, I just restacked the data with the correct ISO bias frames but I still have the same bright central area. There was a flip during imaging so I tried stacking without the flipped images and still got the same result. All stacked results look virtually identical.

Is there anything else I can check? I'm trying to stack very small selections of data to see if that provides a clue.

 

[Giddy]

For the dark test, I took 100 fast bias frames and stacked them in WBPP. I compared that to the 4th of 4 frames shot at 60, 120, and 180s.

Bias

60s Dark

abs(Dn-mB)

I'm not sure what pattern to look for...?

 

[fredvanner]

What about the 120s and 180 s darks?

 

[fredvanner]

Note also that you can see hot pixels (also indicated by the max = 65532 in the dark). One of the functions of dark calibration is to remove hot pixels.

 

[cloudbait]

Note also that you can see hot pixels (also indicated by the max = 65532 in the dark). One of the functions of dark calibration is to remove hot pixels.

That's what caught my eye. The mean stayed very similar, meaning the actual dark current is low. But the number of outliers shot way up, seen in that max value as well as the deviation. True hot pixels don't correct out very well (but CC will take care of them), but in this case I think there are a lot of warm pixels that will. And I'll bet the longer exposures will really have a bunch of those. Darks are probably warranted.

 

[fredvanner]

The mean stayed very similar, meaning the actual dark current is low.

However, if 10 ADU of dark current at 60s turns out to be 20 ADU at 120s and 30 ADU at 180s this needs to be calibrated out. Even 10 ADU is worth subtracting (albeit with a pedestal, later removed, to prevent clipping).

 

[cloudbait]

However, if 10 ADU of dark current at 60s turns out to be 20 ADU at 120s and 30 ADU at 180s this needs to be calibrated out. Even 10 ADU is worth subtracting (albeit with a pedestal, later removed, to prevent clipping).

Agreed. It's just an interesting sensor behavior. Overall low dark current but very high dark current in a small (but probably significant) number of pixels. In fact, not what we usually see with CMOS sensors. Of course, it's also possible that we're not seeing true "raw" data. The raw files from most consumer cameras are still processed to some extent, with things like factory programmed fixed pattern removal and forms of dark subtraction. And the black level of 4K is very high (1K in the native 14-bit space), and would seem to be an artificial offset of some kind.

 

[fredvanner]

In fact, not what we usually see with CMOS sensors.

The Nikon Z7 does indeed have a relatively modern CMOS sensor, but optimised for terrestrial photography where managing dark current is not a high priority issue.

 

[fredvanner]

And the black level of 4K is very high (1K in the native 14-bit space), and would seem to be an artificial offset of some kind.

Indeed. I'm certain that this is simply the camera fixed bias offset.