Newbie - Why do my colours looks so different

[midnightlightning]

Hi,

I got the PI trial yesterday and have processed my M31 data twice following the Harry Newbie tutorials but got very different colour results. I prefer the Yellow to the Blue version.
Can anyone help with where or what I did to lose the yellow core?


EDIT - the processes I used were Channel Combine, Linear Fit, Colour Calibration, Background Neutralisation ,DBE, SCNR, HDR Multiscale Transform, ACDNR, Histogram Transform, Curves.

I'm not sure of the exact order.

Thanks

Jon

[Geoff]

Well, for a start you should have done DBE then BN before colour calibration.

[ngc1535]

Hi Jon,

In many cases cookbook type workflows ("do these things in this order with these settings") will give you an answer- but as you have indicated you may not know why you have a particular result.
This is often deals with the details of processes and how you used them. (For example, with Color Calibration, seeing the method you used, with the previews you chose, might be informative). In addition- the use of Linear Fit to achieve color balance is in the best case unnecessary- and in the worst case results in exactly the kind of issue you have a question about.

For others that are reading this- I posed this question a "few" years ago- but I was just starting my journey in the understanding of PI.
https://pixinsight.com/forum/index.php?topic=7393.msg50115#msg50115
I am better positioned to now argue my point concerning Linear Fit. Juan perfectly explained how Linear Fit works. The part that was not addressed concerned whether Linear Fit will always result in the proper color balance for the image (in the same way Color Calibration and Photometric Color Calibration do). The answer is no. And things might be unnecessary or weird to combine LF with Color Calibration in order to achieve the result. LF will tend to remove color biases and make images more white by equalizing the brightnesses in the channels. Some objects and fields have real color biases.

And just to be a contrarian- I do not think that DBE should come before BN in a typical workflow. BN simply applies offsets to the color channels- and if it is possible to chose a good background patch known to be "the sky." - Resulting image is really the one to apply DBE to. BN is a linear operation. DBE is not (it is a model with different results depending on tolerances, multiplicative/subtractive..etc etc).

-adam

[Geoff]

And just to be a contrarian- I do not think that DBE should come before BN in a typical workflow. BN simply applies offsets to the color channels- and if it is possible to chose a good background patch known to be "the sky." - Resulting image is really the one to apply DBE to. BN is a linear operation. DBE is not (it is a model with different results depending on tolerances, multiplicative/subtractive..etc etc).

-adam

Good point Adam

[midnightlightning]

Thanks, sounds like Linear Fit may be implicated, I will experiment.

If I have to understand the math behind PI processes, as in that link, I won't be taking PI any further though.

[RickS]

If I have to understand the math behind PI processes, as in that link, I won't be taking PI any further though.

It's perfectly possible to use PI without digging into the maths and plenty of people do.

[Geoff]

If I have to understand the math behind PI processes, as in that link, I won't be taking PI any further though.

As Rick said, you don’t have to know the maths but it’s nice to know that it’s there if you want to dig deeper and understand what’s in the black box.

[ngc1535]

Midnight,

Your response is a bit of a catch-22. You don't have to understand the "math" to use PI- but in order to answer your question of "where or what"  (or in my rewording of your question "Why did I lose the color in the core of M31?"), some deeper discussion of how processes work is necessary. I probably should have made my tangential comment on Linear Fit a separate thread. It just seemed to be on-topic to a larger audience. And as you saw, two people can suggest opposite ideas. The only recourse to understand if the order of the workflow you used is connected with the result you achieved- especially in the face of two opposite suggestions- is to look at how a process works (which is typically mathematically). This applies to any image processing software in general.

I actually think Linear Fit isn't the major factor in your result- I suspect one of Color Calibration, SCNR, or HDRMT (applied to all the color channels rather than luminance for example) are more likely the greater influences on your resulting color.

Color Calibration: If you choose mostly the core of M31- and this is easy to do if you choose the threshold method of getting the relevant signal- as the predominant color of the core is yellow- you will transform the core (and image in general) into something more white or bluish. I suspect this is the method you used. You might experiment by making the white reference a single small background galaxy in the image- or use the structure detection method (which will basically just end up using the stars).  The vagaries of Color Calibration can be greatly minimized by using Photometric Color Calibration. This process didn't exist at the time many of the tutorials you find in the wild were made. It is a relatively hands-off approach that gives a consistent answer based on a reference catalog of stars. It is super awesome.

There are things to consider for each of the other processes as well... but I don't want to go on more tangents!
-adam
 

[midnightlightning]

Hi Adam, thanks for your very helpful response, I am working hard on this stuff and hopefully learning but quite a steep curve.
I'm not sure what order I ran the processes in my original post, I have since written down a basic workflow which I will adapt but didn't do this for the first couple of attempts.

The image is RBG only, I haven't got as far as adding in Lum and Ha yet (next on my list), so HDRMT would have been applied to the RGB as a whole.

I think I only used SCNR as a final step to remove a green cast.

ColourCal. I used White and Background Previews.

The white was made up of 7 preview "slices" so that I covered the whole of M31 (being on a diagonal it was the only way I could think to do it) and then aggregated them with the Preview aggregation script (whatever its called). Lower limit set just above background reading, which was high - background was something around .0025 where I expected around .007. Upper limit left at default.

Background. Upper limit .01, lower limit default.

Structure detection off.

I checked both reference graphs and thought they looked ok - checked against the Harry PI video tutorial.

If there is anything obviously wrong with the above do let me know.

I have read about the photometric colour calibration but not had chance to try it yet - sounds excellent.

The nicest surprise I have found to date is Blink and the Image analysis script (sorry I don't have PI on this laptop, one of the limitations of the trial, and cant remember the name). Using these I identified that a significant number of lights were sub standard. The mist came in one night and I didn't notice as I was monitoring from indoors!!!! Anyway, they are a major incentive for me to purchase PI in themselves.

I used these today to analyse the results of an experiment with calibration frames. I used various combinations of Darks, Flats, Dark Flats and Bias in APP pre-processing. I usually use all 4 types but my trial showed there was no difference that I could detect in quality whether I used all four or just flats/dark flats with either Darks or Bias. The one without Bias looked "milky" in Blink compared to the others and had lower star count but identical noise and FWHM, but when I processed each stack I couldn't differentiate visually. I suspect the milkiness was removed by the LP wipe (I used Star Tools for a quick process as it only takes a couple of minutes per stack). Not scientific but interesting.

Thanks again for your detailed reply.

Jon







 

[ngc1535]

Just as a final follow-up- I do believe your Color Calibration choices were not necessarily helpful. Although M31 is a spiral galaxy... it really isn't an SBc and certainly not an Sc... it is too extended and inclined..and the yellow core will contribute too much to the resulting color balance. Just run Color Calibration with Structure detection with the default... and I bet you will be happier. PCC is really the ultimate arbiter.
-adam

[midnightlightning]

LOL, the tutorial never mentioned the technique only applied to specific galaxy classifications - how was I supposed to know that

Will try again using structures, thanks again.

Jon

[ngc1535]

LOL, the tutorial never mentioned the technique only applied to specific galaxy classifications - how was I supposed to know that

Will try again using structures, thanks again.

Jon

Yes, there is some theory behind using that method that you will want to understand.
More of those pesky details.
The white balance method you were using assumed you are selecting signal that is the average spectral characteristic of the integrated light from a spiral galaxy (SBc or Sc). This is a good white reference for reasons that are explained in many places (best place is the PCC documentation). But I claim your selection was not necessarily a good one.
Using the average colors of the stars in the field (structure detection) will likely be closer. To be clear... it isn't that M31 has to be dealt with differently because of the object it is... you use a spiral galaxy as a white reference for all objects. It is just that M31, in my opinion, does not make a good white reference!
-adam
-adam

[midnightlightning]

Thanks Adam, I had a go with PCC and got a much better result, still work in progress.
Jon

[ngc1535]

Great news!

[midnightlightning]

I have reprocessed (several times) and think I am finally making progress, a few things I tried seem to have helped.
I discovered that my background mask for Background Neutralisation contained stars - fixed.

There is a blue bias in my data which I handled by an unlinked STF and applied it to HistTransform, where I backed off the stretch a bit.
Colour Calibration was via PCC - which I really like.

There was a lot of green in the image, I removed it early on with SCRN and found this seemed to slightly reduce the blue bias - I tried with and without SCRN and definitely better with.

I have perhaps overdone saturation, especially blue, but I am happy with it for now.

I added a stack of noise playing with HDR but have mitigated it to some extent with ACDNR - I need more data which I will get if the clouds ever part!

Just need to add Lum and Ha now