PixInsight Forum

PixInsight => Image Processing Challenges => Topic started by: ArminPro on 2018 April 26 01:10:13

Title: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 26 01:10:13
Hi everyone,

after trying several things in many long evenings I hope to get help here in this forum.

Here is my workflow (photos are taken with the Sony Alpha 7s on a 10" Meade SCT):
-----------------------------------------------------------------------------------------------

- master bias out of 50 bias frames

- bias-subtracted master dark out of 30 darks with same exposure as lights (10min)
   ->Image Calibration (applied master bias with checkbox "Calibrate" enabled)
   ->Image Integration

- bias-subtracted master flat out of 50 flats with a flat field box
   ->Image Calibration (applied master bias with checkbox "Calibrate" enabled)
   ->Image Integration

- Image Calibration of 30 10min exposures of M51 (applied master bias, master dark, master flat with checkbox "Calibrate" disabled)
  and then integrated with Image Integration to integration.xisf

(all Image Integration settings out of PixInsight tutorials)

- debayer of integration.xisf

- Dynamic Alignment (Star Alignment gave bad results, because guiding did not work properly)

--> At this point I share the link to my debayered and aligned xisf-files really hoping someone will bring these files to a nice picture of M51,
and can explain what I´m doing wrong.
--> Please, please, please  :embarassed: :embarassed: :embarassed:
Link: http://gofile.me/3WOrk/Z25mFpxuG

What I did furthermore:
- Dynamic Crop, Screen Transfer Function, Background Neutralisation, Automatic Background Extraction, Colour Calibration, Histogram Transformation, HDR Multiscale Transform and Colour Transformation.
But here is my result attached:  :-[

Thanks in advance,
Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 26 02:01:02
Hi Armin,

I downloaded the file "_2018_04_20_01_35_12_10_c_d_registered.XISF". The data are severely clipped both in the R and G channel, not in the B channel (see appended histogram and statistics; the horizontal zoom is 800, i.e shown is only a small detail of the histogram at the extreme left). This is very strange, and I cannot imagine why this should happen. For sure, something went quite wrong in your image calibration.

If I understand you correctly, you are using a pre-calibrated (bias-subtracted) MasterDark. This is not advisable. See my guide https://pixinsight.com/forum/index.php?topic=11968 (https://pixinsight.com/forum/index.php?topic=11968) for explanation and description of a correct image calibration. If you have questions after having read this, I am gladly willing to help.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 26 02:30:06
Hi Bernd,

but why should this registered.xisf be clipped in the R and G, because the colour calibration is done afterwards?

I will read your tutorial ...   :)

Thanks, Bernd!

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 26 04:17:58
Aah, just a moment.
I´ve forgotten to tell that I´m using a Hetuch IDAS Filter against light pollution in my street.
But this filter should not clip colours!
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 April 26 09:41:53
I have an A7S as well. Great camera.

I see the clipping as well. It's in all 3 channels but predominantly in the R & G channels. You can see this with Statistics. The % row should be very close or equal to 100%.

Code: [Select]
            R          G          B
count (%)   49.98985   56.27555   95.65478

The filter should not be a factor.

Assuming you still have the individual calibration frames, go ahead and run everything through the BatchPreprocessingScript without using any of the masters you created previously. How does that result look?

Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 27 00:35:13
Hi,

at first I´ve a question: What means "pre-calibrated (bias-subtracted) MasterDark"?
In the picture below is my understanding. It is the calibration of the flats, but I´m doing the same for the darks.
So I did not pre-calibrate the darks, right?

Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

@Greg: I´m still having the calibrations frames. And I´ve tried the BatchPreprocessingScript. Wow! Fantastic tool. I did not know it.
But at the end it had problems with integrating. I had a bad autoguiding, so I did first dynamic alignment with all pics, but the script had nevertheless problems. I attachted the error message and an example of a dynamic aligned pic. It looks strange.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 27 04:20:01
Hi Armin,

Quote
at first I´ve a question: What means "pre-calibrated (bias-subtracted) MasterDark"?

A pre-calibrated MasterDark is made either
- by subtracting the MasterBias from the Dark frames and integrating the calibrated Dark frames or
- by integrating the Dark frames and subtracting the MasterBias from the resulting integration.
Both approaches are to be avoided. Please read my guide https://pixinsight.com/forum/index.php?topic=11968 , I will not repeat it here.

Quote
In the picture below is my understanding. It is the calibration of the flats, but I´m doing the same for the darks.
So I did not pre-calibrate the darks, right?

Of course, you did: when you applied the same procedure to the Dark frames. Besides, checking 'Calibrate' in the "Master Bias" section is unnecessary - you don't want to calibrate the MasterBias. (You can remove the interrogation marks from the comments in your first screen dump, they are both correct.)

Quote
Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

I used Canon DSLR cameras for some years, and I can tell you that there is no clipping at all if you do the calibration process correctly. However, I cannot speak for Sony cameras as I never had one.

A wrong calibration will damage the data. I guess that the clipping that you observe is only a result of a wrong calibration procedure. That should be a good motivation to improve it.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: sharkmelley on 2018 April 27 05:19:25
Quote from: ArminPro
Regarding the clipping in the Sony A7s. I´ve heard, that most DSLRs do this, even in the RAWs. It is an internal step after the exposure. Is it possible to disable this feature, maybe with a hack? But nevertheless I see wonderful pics from galaxies in the net made by DSLRs.

The Sony A7S (and Sony cameras in general) do not clip their blacks in the raw files.   It was certainly a problem on many Nikon cameras where bias frames and dark frames would clip at zero and for some of them a hack did exist to prevent it.  More recent Nikon cameras do not have the same issue because they have a non-zero bias.

Mark
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 27 05:55:13
Hi Bernd,
In the meanwhile I´ve read it.
And this is the extract which I will use when using a DSLR without temperature control and have flats with very short exposures:

1. Generate a master bias by simply integrating  the bias frames, and make a superbias.
2. Generate a master dark by simply integrating  the dark frames.
3. Genaerate master flat = flats - master bias.
4. Calibrate lights with master bias, master dark and master flat. "Calibrate" and “Optimize” checkboxes for master dark enabled.
5. Remaining hot pixels must be corrected with the CosmeticCorrection process directly after completed calibration process.

One questions to the blacks which sharkmelley mentioned: Does the Sony automatically generate black (=darks?) after the exposure?

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 April 27 08:45:47
Quote
One questions to the blacks which sharkmelley mentioned: Does the Sony automatically generate black (=darks?) after the exposure?

If you have Long Exposure NR enabled on the camera it will take a dark frame after each exposure and use that to calibrate the light frame. I'm not clear on the exact process the camera uses. I'm not sure this works in bulb mode either.

Regardless, for AP you want this OFF.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 27 10:18:11
 :'( :'( :'( :'( :'( :'(

I have another problem!
After dynamic alignment I get in the attached pic a strange structure.
If I nevertheless do calibrating and integrating this structure gets worse.

I must do dynamic alignment because of star deviations so that star alignment could not handle it.

Does anyone know this problem?
Title: Re: Bad Photos or bad workflow? Need help.
Post by: pfile on 2018 April 27 11:18:05
it looks to me like you are trying to register un-debayered images...

rob
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 27 12:53:03
I must first debayer before I can do dynamic align?

Thanks for the hint! :)
Title: Re: Bad Photos or bad workflow? Need help.
Post by: pfile on 2018 April 27 12:53:56
yes - the flow should be:

calibrate, debayer, align, integrate

rob
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 28 03:48:21
Hi,

I did the calibrations, which Bernd recommends, and now it looks much better, as you can see!
Thanks to Bernd and the others!

But there are still some patterns, I don´t understand, the galaxy is noisy, and I thougth with 26 exposures with each 10 min the result should be even better.
I did no colour transformation, because with every setting I made it worse.

So I share again a link with all steps in the process and also the masters I used. Mabe you find again a big mistake.  :embarassed:

http://gofile.me/3WOrk/fRitvRsqB

Regards,
Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 28 07:36:01
Hi Armin,

I am sorry, but your master calibration frames seem all to be wrong:
- The Superbias has these constant vertical pattern, and its intensity values don't suit to the MasterDark. There was no Bias frame, so I cannot help.
- The MasterDark has too large intensity values compared with a Light frame. There was no Dark frame either.
- The MasterFlat has these constant vertical pattern as well, presumably because the flats were calibrated with the (defective) Superbias. (The flat frames are underexposed.)

In the calibrated light frames the red channel is severely clipped, and the blue channel is somewhat clipped as well.

In your M52a.jpg I see horizontal colored banding and a strange constant vertical pattern or grid.

I guess the error has been made during preparation of the master calibration frames, but I don't know exactly what went wrong. Please. proceed according to the guide.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: dvonhand on 2018 April 28 07:55:05
Hey,

Are you dithering your frames at all?  Dithering is moving the camera by a couple pixels between each frame so that the same point in the image hits different pixels in the camera sensor each time.

I've found that when using one-shot color sensors I have to dither otherwise I get color artifacts in the image.

You may also want to check your individual master light frames (e.g. the master red, master green, and master blue) to make sure there aren't artifacts in those images.  If there are artifacts in those images then any of the channel combination and color corrections won't work or will be harder to do.  It looks like you have some kind of moire pattern in your master light frames.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 28 08:52:13
Sorry, my son has birthday party, so just sharing the links of bias and darks:

http://gofile.me/3WOrk/GEK8snLDO
http://gofile.me/3WOrk/Iol8LpwJy

I know dithering, but at the moment I don´t dither.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 29 04:10:05
Hi Bernd,

you´re right. Something strange happens with my calibration files.

I examined my master_bias and zoomed in and saw a strange pattern. I used exactly the integration settings, you recommended.
I compared it with the master_bias made with the BatchPreprocessing script, which looks different.
Do you know what happens there?

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 29 08:45:39
Hi Armin,

I took a look at one Bias frame (DSC02160.ARW) and a Dark frame (DSC02220.ARW). They look OK to me.

However all of your master calibration frames are defective. In my guide I didn't describe the prepration of the master calibration files in detail but referred to Vicent's tutorial https://www.pixinsight.com/tutorials/master-frames/ . I guess your error was, that in the integration of bias and dark frames, in the section 'Image Integration', you didn't set 'Normalization' to 'No Normalization'. Please check, that all the settings are made like in Vicent's tutorial when integrating Bias, Dark or calibrated Flat frames. (The settings for the calibration of Flat frames are different again!)

I suggest the following approach:

1) Delete all master calibration frames, delete the calibrated Flat frames, delete the Integrations and the images that were derived from the Integrations.
2) Prepare MasterBias and MasterDark according to Vicent's tutorial. 
3) Calibrate the Flat frames, deviating from Vicent's tutorial, only with the new MasterBias (section 'MasterDark' shall be unchecked, no MasterDark needed in this step).
4) Integrate the calibrated Flat frames with the settings in Vicent's tutorial.
5a) With the new MasterDark and MasterFlat, calibrate your light frames with the settings in my guide, image 2, left hand side.
or
5b) With the new MasterBias, MasterDark and MasterFlat, calibrate your light frames with the settings in my guide, image 2, right hand side.

I am curious about your results.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 29 10:51:45
Hi Bernd,

in my last post I sent the integration setting, too. There youj can see, that I´ve set it to "No normalization".
But I do, as you recommend. I´ve deleted all the files. But integrating the bias and afterwards the darks to a master, there is this pattern again.
Please see the zoom into the master_dark in this post. It is the same pattern as in the master_bias, which I posted in the last post.

I know Vincent´s tutorial. I´ve read it also.

Also strange, I told before, when I´m calibrating with the BatchPreprocessing Tool, these pattern don´t exist.

I think it does not make sense to continue with the work flow, if we don´t know the reason for this.

Are there any other sytem settings or options?

Nevertheless thanks, Bernd.

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 29 12:10:01
One additional insight:

I don´t get these pattern for master bias and master dark when generating with the BatchPreprocessing script.

But there is pattern in the master flat when generating with the BatchPreprocessing script.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 April 29 12:16:23
The hatch pattern is the bayer matrix and can only be seen in an non-debayered frame that has captured light. On bias and dark frames I believe you won't be able see the matrix because there is no light on the sensor to reveal it. On flats and lights you should will be able to see it.

I'm not sure why it shows up in one bias master. Maybe some flats got into the mix?
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 29 12:53:59
Hi Greg,
I do not see it on the bias and dark frames, but on the bias and dark masters.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 29 17:21:34
If your settings in the integration module are correct when integrating bias and dark frames, the only possibility that I can imagine is (like Greg already suggested) that there are defective bias and dark frames respectively. You can easily check that with 'Blink', generating the statistics. Then compare median and mean values and inspect in detail deviating frames (-> histogram).

If you don't find the reason this way, I'm helpless.

Anyway - it is not plausible for me at all that the problem occurs with the Integration module and not with the batch processing script.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 April 30 02:13:05
Sorry, I really can't reproduce what you experience.

Finally I loaded 2 of your bias frames (DSC02160_bias.ARW and DSC02190_bias.ARW), made a copy of each and integrated these 4 (actually only 2) files with ImageIntegration and the suggested settings. (The only reason for copying the files is, that ImageIntegration refuses to integrate less than 3 files.) The result was completely as expected: no vertical pattern, see attached screen-dump.

I want to add, that I have installed the new RAW module, see https://pixinsight.com/forum/index.php?topic=12126.0 . Maybe that the old, dcraw-based DSLR_RAW module has problems with Sony raw file format ARW? Perhaps you try the new RAW module as well. However, how should it be that the batch preprocessing script doesn't show the problem? I can't imagine that either. So the reasoning remains: either bad settings or some bad files on your side.

Are you working with the current version of PixInsight (01.08.05.1353), and did you install all updates for modules?

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 April 30 04:22:08
Aaaaaaaah!
At least one success! Thanks !!!
After the RAW update my master_dark_new looks like this:

 :) :) :)

I will do the next steps and give answer.

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 April 30 10:49:39
Is that new master debayered - it shows that it is (RGB in the header)? You don't want it to be. Check the settings for the DSLR RAW format handling here:

     - View->Explorer Windows->Format explorer
     - Select DSLR_RAW
     - Select Edit Preferences
     - Select Pure Raw, then OK.


The preferences window should look like below:




Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 01 12:55:59
Hi Greg,

just coming home from a trip ...

No, it was not debayered, but the "pure raw" option was not enabled.
So I enabled "pure raw" and generated the masters again. Now in the header there stands "gray". Is this o.K.?

Another question is to the xisf: Are the following preferences o.K.?

Armin

Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 May 01 13:50:01
So I enabled "pure raw" and generated the masters again. Now in the header there stands "gray". Is this o.K.?

Hopefully that helps. Masters should not be debayered, else you're not getting an accurate representation of every pixel due to the interpolation done during the debayering process.

Quote
Another question is to the xisf: Are the following preferences o.K.?

They look OK to me. I checked mine and see my xisf preference menu differs from yours a bit. Screen shot of mine is below. These are the defaults. I've never needed to change them.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 01 14:05:50
O.K., thanks.

Here is my result after integrating the lights with subtracting all masters, Automatic Background Extraction, Background Neutralisation, Colour Calibration, Histogram Transformation, HDR Multiscale Transform and Curves Transformation.

In my opinion it is not nice regarding 5 hours exposure. What do you think?
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 May 01 14:27:22
Post a fresh link to the unprocessed master light frame so we can have a look.

For comparison, this is a 2 year old master light of the same object. Total time was 200 minutes, exposures were 1200s each - way longer than necessary. Since the image is from the same camera (A7S) it might be useful for comparison purposes.

https://www.dropbox.com/s/glq3ak1upagwwmh/light-BINNING_1.xisf?dl=0

Note that if you open that frame and look at the Statistics process you'll see 100% count for the R, G, and B channels. If those numbers are less than that for your masters you'll want to find out why. Most likely something went wrong during calibration.

Processed version is below, also from 2 years ago.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 02 00:13:24
Hi Greg,

this picture is hundred times better than mine! Wow!

Here is the new link: http://gofile.me/3WOrk/HaHC1jRfL

The problem is, that the deviations from picture to picture is great, so that StarAlignment did not work.
So after ImageCalibration (with all masters) and then debayering, I had to use DynamicAlignment (before I could integrate), which needs a long time.

So if you do a new ImageCalibration or Debayer, I could do the DynamicAlignment, in order you will not spend too much time.
Or you align only 8 pictures which is the minimum.

Thanks in advance for this great offer!

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 May 02 01:39:05
I downloaded the file integration.xisf. First thing I noticed is the total pixel count is a bit less than the full sensor size. Is this image cropped? Statistics don't reveal any clipping <- this is good. An unlinked STF gives a decent result. I see the remnant of a satellite or airplane path from left to right, passing just in front of M51. If you can't get rid of it completely during integration or processing maybe consider dropping the frame that is contributing it. At first glance I'd say the integration looks pretty good.

I did a quick and partial processing on it to see what I could get at easily. I was able to get a decent result in not too much time. A much better result can be achieved for sure.  Screen capture below. Here's what I did:

 - DBE ( several passes )
 - Photometric Color Calibration
 - TGV Denoise w/ mask & support for L and chrominance
 - stetched w/ Histo Transform
 - Extracted L mask, apply inverted
 - Curves Transformation for contrast
 - HDR Multiscale Transform (mask not inverted here)
 - Curves Trans for saturation & final tweak

I've attached the process container as well.

I think you have something decent to work with here. More data is always nice to have if you have the time for it.



Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 02 02:35:58
You processed this picture out of my integration.xisf ???!!!Wooooooooow !!!!!  Very nice!!!!!

Yes, I did a dynamic crop.

And yes, there was a satellite passing M51. I thought this will be processed out through the integrating process?
Then I will drop it.

I think I´ve to watch some tutorials, because some processes you did, I don´t know by now.

What do you mean with "more data is always nice ..."? More exposures?

I used Curves Transformation, too. But what do you mean with CT for contrast and CT for saturation?
Please, maybe you could send a screenshot of the CT or is it implemented in the process container?

Thanks a lot!!

Regards,
Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 May 02 02:41:54
Hi Armin,

without doubt, Greg's tips are very valuable for the post-processing steps. However, you are still having a severe problem with your calibration, and this must be resolved first!

I had a look at your master_bias_new.xisf, master_superbias_new.xisf, master_dark_new.xisf and master_flat_new.xisf. MasterBias, Superbias and MasterFlat are OK now, but the MasterDark isn't, it is severely clipped. Once again: this seems to be a pre-calibrated MasterDark, obtained by either calibrating the bias frames and integrating the calibrated bias frames or by calibrating the integrated bias frames. Both of these approaches are not recommended!

You should delete your MasterDark (master_dark_new.xisf) and build a new one in this way:
Simply integrate the dark frames with the settings recommended in https://www.pixinsight.com/tutorials/master-frames/ . Finished, that's it.

Then proceed with these calibration master files performing the light frame calibration (settings: see https://pixinsight.com/forum/index.php?topic=11968 , image 2, left hand side or image 2, right hand side.

I guess the flat field correction can be improved yet, by increasing the exposure time for the flat frames by a factor of 3.5 to 4.0. However, whereas the modified procedure for building the MasterDark is an error correction, this is an optimization.

Good luck!

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 02 03:05:37
Hi Bernd,

I really really did simply an integration of the dark frames. With exactly the same settings I did for the integration of the bias frames.
I did it immediately after integrating the bias frames.  :'(
And I deleted the old one before. :'(
Could it be, that different ambient temperature of the darks and lights generate this problem? Because I generated the darks two days after the lights.

For the Image Calibration of the light frames I see one little difference to my settings: I did not enable "Evaluate noise" in the Output Files section.
I will change this.
And I enabled "Calibrate" and "Optimize" in the Master Dark section, because when I remember right, you recommended it in your tutorial when using DSLR without temperature control and have flats with very short exposures.

Flats: I have chosen the exposure time for the flats in that way, that the histogram is in the middle.  :-\

Armin


Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 May 02 03:57:04
Hi Armin,

Quote
I really really did simply an integration of the dark frames. With exactly the same settings I did for the integration of the bias frames.
I did it immediately after integrating the bias frames.  :'(
And I deleted the old one before. :'(
Oha, I'm sorry, now this is my fault - your Master Dark (master_dark_new.xisf) isn't clipped at all. I apologize for my overhasty statement.

Quote
For the Image Calibration of the light frames I see one little difference to my settings: I did not enable "Evaluate noise" in the Output Files section.
I will change this.
This is not necessary, it will not change anything.

Quote
And I enabled "Calibrate" and "Optimize" in the Master Dark section, because when I remember right, you recommended it in your tutorial when using DSLR without temperature control and have flats with very short exposures.
As I don't detect any "amp glow" in your MasterDark, I guess this is the way to be preferred as well.

Quote
Flats: I have chosen the exposure time for the flats in that way, that the histogram is in the middle.  :-\
Yes, but in the middle of what? The middle of the camera's histogram? DSLR cameras show an already stretched histogram, so this is not correct. You could take a series of frames with different exposure time (bracketed exposure?) and view the image statistics or the histogram in PixInsight. Your camera has a resolution of the AD converter of 14 bits (corresponding to a maximum value of 2^14 - 1 = 16383).

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 02 04:31:00
Hi Bernd,

yes, I meant the histogram of the camera. So the histogram of the unstretched picture should be in the middle?
I will do a series of exposures and watch the histogram in PI.

By the way, where can I see the histogram in PI? In Histogram Transformation?

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: Greg Schwimer on 2018 May 02 09:24:53
What do you mean with "more data is always nice ..."? More exposures?

I used Curves Transformation, too. But what do you mean with CT for contrast and CT for saturation?
Please, maybe you could send a screenshot of the CT or is it implemented in the process container?

By more data I mean more exposures. This will help reduce noise further and also help raise the signal to noise ratio for the image, especially the dimmer parts.

The process container has the processes I used. I'd skip the TGV Denoise part for now. The mask I use is a simple extraction from the integrated image.
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 May 02 10:59:43
Hi Greg,

I could open the Process Manager and see all your processes. By double clicking I can open each process and see which settings you´ve made.

Did you just do 8 times the same DBE? I cannot see any difference when processing this.

I don´t understand "The mask I use is a simple extraction from the integrated image." But I think I´ve to watch a tutorial for generating masks, right?

And I don´t understand what you did in that many Histogram Transformations (HT).
In my processing I took the "New Instance" button once from the SCT and laid it to the HT in order to stretch the picture.
But I will also read more about the HT.

Thanks,
Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 May 02 11:05:15
Hi Armin,

Quote
yes, I meant the histogram of the camera. So the histogram of the unstretched picture should be in the middle?
I will do a series of exposures and watch the histogram in PI.
The Peak in the histogram of the raw flat frames shall be about in the middle of the data range of the camera. In your case (14-bit ADC) this means: 0.5 * 2^14 ADUs = 0.5 * 16384 ADUs = about 8200 ADUs. In PixInsight you can view it either in the histogram or in ImageStatistics (mean or median).

Quote
By the way, where can I see the histogram in PI? In Histogram Transformation?
Yes, in the Histogram Transformation module. The image in question must be opened, and in HistogramTransformation you select that View, from which the histogram shall be displayed.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 June 08 14:10:10
Hi Bernd,

yesterday I did some flat frames with various exposure times: 0.2s, 0.3s, 0.4s and 0.5s

In the histogram module I can see, that with rising exposure time the histogram moves from the left corner a little bit to the middle. But it is still far away from the middle. Also in the ImageStatistics the mean values go from 260 (0.2s) to 650 (0.5s) which is far away from 8200.

So I have to chose such an exposure that I will get 8200? I think this would be some seconds.

Armin
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 June 09 01:56:45
yesterday I did some flat frames with various exposure times: 0.2s, 0.3s, 0.4s and 0.5s

In the histogram module I can see, that with rising exposure time the histogram moves from the left corner a little bit to the middle. But it is still far away from the middle. Also in the ImageStatistics the mean values go from 260 (0.2s) to 650 (0.5s) which is far away from 8200.

So I have to chose such an exposure that I will get 8200? I think this would be some seconds.

Hi Armin,

yes. However, one additional point is important: the data that I gave are valid only whith ImageStatistics set to 16-bit [0,65535].

At the beginning of this thread you wrote that you are taking the flat frames with a flat field box. Exposure times of a few seconds seem unusually high then. Did you dim the flat field box?

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 June 09 03:37:03
Quote
At the beginning of this thread you wrote that you are taking the flat frames with a flat field box. Exposure times of a few seconds seem unusually high then. Did you dim the flat field box?
Yes, I dimmed it a little bit. Should I drive with maximum performance?

Quote
However, one additional point is important: the data that I gave are valid only whith ImageStatistics set to 16-bit [0,65535].
I´ve set the ImageStatistics to 14-bit, because you wrote that I´ve got a 14-bit ADC in my Sony. :-\
When set to 16-bit in ImageStatistics I get 1000 (2s) to 1800 (5s).
Title: Re: Bad Photos or bad workflow? Need help.
Post by: bulrichl on 2018 June 09 04:37:46
Quote
However, one additional point is important: the data that I gave are valid only whith ImageStatistics set to 16-bit [0,65535].
I´ve set the ImageStatistics to 14-bit, because you wrote that I´ve got a 14-bit ADC in my Sony. :-\
When set to 16-bit in ImageStatistics I get 1000 (2s) to 1800 (5s).
The setting of 16-bit [0,65535] in ImageStatistics is correct if you keep in mind that the maximal values will be scarcely 2^14-1 = 16383 due to the 14-bit ADC.

The resulting values are still surprisingly low. My flat field box is rather bright, and I take flat frames (Canon EOS 600D = Rebel T3i) at ISO 100 and 1/60 s, undimmed.

Quote
At the beginning of this thread you wrote that you are taking the flat frames with a flat field box. Exposure times of a few seconds seem unusually high then. Did you dim the flat field box?
Yes, I dimmed it a little bit. Should I drive with maximum performance?
Yes, I would use it undimmed and go for about 8000.

Bernd
Title: Re: Bad Photos or bad workflow? Need help.
Post by: ArminPro on 2018 June 09 12:53:34
Quote
Yes, I would use it undimmed and go for about 8000.

No, I used it undimmed and got mean=7500 for an exposure of 1/50s and ISO=800.
Thanks!