Author Topic: Bad Photos or bad workflow? Need help.  (Read 5026 times)

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Bad Photos or bad workflow? Need help.
« on: 2018 April 26 01:10:13 »
Hi everyone,

after trying several things in many long evenings I hope to get help here in this forum.

Here is my workflow (photos are taken with the Sony Alpha 7s on a 10" Meade SCT):
-----------------------------------------------------------------------------------------------

- master bias out of 50 bias frames

- bias-subtracted master dark out of 30 darks with same exposure as lights (10min)
   ->Image Calibration (applied master bias with checkbox "Calibrate" enabled)
   ->Image Integration

- bias-subtracted master flat out of 50 flats with a flat field box
   ->Image Calibration (applied master bias with checkbox "Calibrate" enabled)
   ->Image Integration

- Image Calibration of 30 10min exposures of M51 (applied master bias, master dark, master flat with checkbox "Calibrate" disabled)
  and then integrated with Image Integration to integration.xisf

(all Image Integration settings out of PixInsight tutorials)

- debayer of integration.xisf

- Dynamic Alignment (Star Alignment gave bad results, because guiding did not work properly)

--> At this point I share the link to my debayered and aligned xisf-files really hoping someone will bring these files to a nice picture of M51,
and can explain what I´m doing wrong.
--> Please, please, please  :embarassed: :embarassed: :embarassed:
Link: http://gofile.me/3WOrk/Z25mFpxuG

What I did furthermore:
- Dynamic Crop, Screen Transfer Function, Background Neutralisation, Automatic Background Extraction, Colour Calibration, Histogram Transformation, HDR Multiscale Transform and Colour Transformation.
But here is my result attached:  :-[

Thanks in advance,
Armin

Offline bulrichl

  • PixInsight Old Hand
  • ****
  • Posts: 488
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #1 on: 2018 April 26 02:01:02 »
Hi Armin,

I downloaded the file "_2018_04_20_01_35_12_10_c_d_registered.XISF". The data are severely clipped both in the R and G channel, not in the B channel (see appended histogram and statistics; the horizontal zoom is 800, i.e shown is only a small detail of the histogram at the extreme left). This is very strange, and I cannot imagine why this should happen. For sure, something went quite wrong in your image calibration.

If I understand you correctly, you are using a pre-calibrated (bias-subtracted) MasterDark. This is not advisable. See my guide https://pixinsight.com/forum/index.php?topic=11968 for explanation and description of a correct image calibration. If you have questions after having read this, I am gladly willing to help.

Bernd

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #2 on: 2018 April 26 02:30:06 »
Hi Bernd,

but why should this registered.xisf be clipped in the R and G, because the colour calibration is done afterwards?

I will read your tutorial ...   :)

Thanks, Bernd!

Armin

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #3 on: 2018 April 26 04:17:58 »
Aah, just a moment.
I´ve forgotten to tell that I´m using a Hetuch IDAS Filter against light pollution in my street.
But this filter should not clip colours!

Offline Greg Schwimer

  • PixInsight Addict
  • ***
  • Posts: 210
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #4 on: 2018 April 26 09:41:53 »
I have an A7S as well. Great camera.

I see the clipping as well. It's in all 3 channels but predominantly in the R & G channels. You can see this with Statistics. The % row should be very close or equal to 100%.

Code: [Select]
            R          G          B
count (%)   49.98985   56.27555   95.65478

The filter should not be a factor.

Assuming you still have the individual calibration frames, go ahead and run everything through the BatchPreprocessingScript without using any of the masters you created previously. How does that result look?

« Last Edit: 2018 April 26 13:21:54 by schwim »
- Greg
Scottsdale, Arizona, USA

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #5 on: 2018 April 27 00:35:13 »
Hi,

at first I´ve a question: What means "pre-calibrated (bias-subtracted) MasterDark"?
In the picture below is my understanding. It is the calibration of the flats, but I´m doing the same for the darks.
So I did not pre-calibrate the darks, right?

Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

@Greg: I´m still having the calibrations frames. And I´ve tried the BatchPreprocessingScript. Wow! Fantastic tool. I did not know it.
But at the end it had problems with integrating. I had a bad autoguiding, so I did first dynamic alignment with all pics, but the script had nevertheless problems. I attachted the error message and an example of a dynamic aligned pic. It looks strange.

Offline bulrichl

  • PixInsight Old Hand
  • ****
  • Posts: 488
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #6 on: 2018 April 27 04:20:01 »
Hi Armin,

Quote
at first I´ve a question: What means "pre-calibrated (bias-subtracted) MasterDark"?

A pre-calibrated MasterDark is made either
- by subtracting the MasterBias from the Dark frames and integrating the calibrated Dark frames or
- by integrating the Dark frames and subtracting the MasterBias from the resulting integration.
Both approaches are to be avoided. Please read my guide https://pixinsight.com/forum/index.php?topic=11968 , I will not repeat it here.

Quote
In the picture below is my understanding. It is the calibration of the flats, but I´m doing the same for the darks.
So I did not pre-calibrate the darks, right?

Of course, you did: when you applied the same procedure to the Dark frames. Besides, checking 'Calibrate' in the "Master Bias" section is unnecessary - you don't want to calibrate the MasterBias. (You can remove the interrogation marks from the comments in your first screen dump, they are both correct.)

Quote
Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

I used Canon DSLR cameras for some years, and I can tell you that there is no clipping at all if you do the calibration process correctly. However, I cannot speak for Sony cameras as I never had one.

A wrong calibration will damage the data. I guess that the clipping that you observe is only a result of a wrong calibration procedure. That should be a good motivation to improve it.

Bernd

Offline sharkmelley

  • PTeam Member
  • PixInsight Addict
  • ***
  • Posts: 239
    • View Profile
    • Mark Shelley Astrophotography
Re: Bad Photos or bad workflow? Need help.
« Reply #7 on: 2018 April 27 05:19:25 »
Quote from: ArminPro
Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

The Sony A7S (and Sony cameras in general) do not clip their blacks in the raw files.   It was certainly a problem on many Nikon cameras where bias frames and dark frames would clip at zero and for some of them a hack did exist to prevent it.  More recent Nikon cameras do not have the same issue because they have a non-zero bias.

Mark
Takahashi Epsilon 180ED
H-alpha modified Sony A7S
http://www.markshelley.co.uk/Astronomy/

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #8 on: 2018 April 27 05:55:13 »
Hi Bernd,
In the meanwhile I´ve read it.
And this is the extract which I will use when using a DSLR without temperature control and have flats with very short exposures:

1. Generate a master bias by simply integrating  the bias frames, and make a superbias.
2. Generate a master dark by simply integrating  the dark frames.
3. Genaerate master flat = flats - master bias.
4. Calibrate lights with master bias, master dark and master flat. "Calibrate" and “Optimize” checkboxes for master dark enabled.
5. Remaining hot pixels must be corrected with the CosmeticCorrection process directly after completed calibration process.

One questions to the blacks which sharkmelley mentioned: Does the Sony automatically generate black (=darks?) after the exposure?

Armin

Offline Greg Schwimer

  • PixInsight Addict
  • ***
  • Posts: 210
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #9 on: 2018 April 27 08:45:47 »
Quote
One questions to the blacks which sharkmelley mentioned: Does the Sony automatically generate black (=darks?) after the exposure?

If you have Long Exposure NR enabled on the camera it will take a dark frame after each exposure and use that to calibrate the light frame. I'm not clear on the exact process the camera uses. I'm not sure this works in bulb mode either.

Regardless, for AP you want this OFF.
- Greg
Scottsdale, Arizona, USA

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #10 on: 2018 April 27 10:18:11 »
 :'( :'( :'( :'( :'( :'(

I have another problem!
After dynamic alignment I get in the attached pic a strange structure.
If I nevertheless do calibrating and integrating this structure gets worse.

I must do dynamic alignment because of star deviations so that star alignment could not handle it.

Does anyone know this problem?

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4638
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #11 on: 2018 April 27 11:18:05 »
it looks to me like you are trying to register un-debayered images...

rob

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #12 on: 2018 April 27 12:53:03 »
I must first debayer before I can do dynamic align?

Thanks for the hint! :)

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4638
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #13 on: 2018 April 27 12:53:56 »
yes - the flow should be:

calibrate, debayer, align, integrate

rob

Offline ArminPro

  • Newcomer
  • Posts: 24
    • View Profile
Re: Bad Photos or bad workflow? Need help.
« Reply #14 on: 2018 April 28 03:48:21 »
Hi,

I did the calibrations, which Bernd recommends, and now it looks much better, as you can see!
Thanks to Bernd and the others!

But there are still some patterns, I don´t understand, the galaxy is noisy, and I thougth with 26 exposures with each 10 min the result should be even better.
I did no colour transformation, because with every setting I made it worse.

So I share again a link with all steps in the process and also the masters I used. Mabe you find again a big mistake.  :embarassed:

http://gofile.me/3WOrk/fRitvRsqB

Regards,
Armin