NormalizeScaleGradient: Bookmark website now!

Status
Not open for further replies.
John,

After integrating the .nsg files produced by NSG, I get warnings like the following:
** Warning: Inconsistent Instrument:Telescope:FocalLength (FOCALLEN keyword) value(s) - metadata not generated.
** Warning: Inconsistent Observation:CelestialReferenceSystem (RADESYS keyword) value(s) - metadata not generated.
** Warning: Inconsistent Observation:Equinox (EQUINOX keyword) value(s) - metadata not generated.
Click to expand...

What I find is that FOCALLEN and EQUINOX have consistent values in the input files to NSG, but inconsistent values in the output files.

In the case of RADESYS, this keyword is completely absent from the input files, but a similar keyword, RADECSYS is present in most. RADECSYS happens to be missing on the precise set of files where the other two keywords are copied over undisturbed.

Here is an excerpt from the first 10 images in my dataset. This was generated by the FITS Data View script, which puts question marks where the FITS keyword is missing.

Do you know why this might be happening?

Thanks,
John

IndexOutput FilenameFOCALLENRADESYSRADECSYSEQUINOXInput FilenameFOCALLENEQUINOXRADECSYS
001w89_M81_M82_Light_120_secs_001_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_001_c_cc_d_B_a_r.xisf6752000FK5
002w98_M81_M82_Light_120_secs_002_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_002_c_cc_d_B_a_r.xisf6752000?
003w98_M81_M82_Light_120_secs_003_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_003_c_cc_d_B_a_r.xisf6752000?
004w99_M81_M82_Light_120_secs_004_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_004_c_cc_d_B_a_r.xisf6752000?
005w99_M81_M82_Light_120_secs_005_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_005_c_cc_d_B_a_r.xisf6752000?
006w98_M81_M82_Light_120_secs_006_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_006_c_cc_d_B_a_r.xisf6752000?
007w101_M81_M82_Light_120_secs_007_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_007_c_cc_d_B_a_r.xisf6752000FK5
008w99_M81_M82_Light_120_secs_008_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_008_c_cc_d_B_a_r.xisf6752000FK5
009w98_M81_M82_Light_120_secs_009_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_009_c_cc_d_B_a_r.xisf6752000FK5
010w101_M81_M82_Light_120_secs_010_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_010_c_cc_d_B_a_r.xisf6752000FK5
 
johnpane said:
John,

After integrating the .nsg files produced by NSG, I get warnings like the following:


What I find is that FOCALLEN and EQUINOX have consistent values in the input files to NSG, but inconsistent values in the output files.

In the case of RADESYS, this keyword is completely absent from the input files, but a similar keyword, RADECSYS is present in most. RADECSYS happens to be missing on the precise set of files where the other two keywords are copied over undisturbed.

Here is an excerpt from the first 10 images in my dataset. This was generated by the FITS Data View script, which puts question marks where the FITS keyword is missing.

Do you know why this might be happening?

Thanks,
John

IndexOutput FilenameFOCALLENRADESYSRADECSYSEQUINOXInput FilenameFOCALLENEQUINOXRADECSYS
001w89_M81_M82_Light_120_secs_001_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_001_c_cc_d_B_a_r.xisf6752000FK5
002w98_M81_M82_Light_120_secs_002_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_002_c_cc_d_B_a_r.xisf6752000?
003w98_M81_M82_Light_120_secs_003_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_003_c_cc_d_B_a_r.xisf6752000?
004w99_M81_M82_Light_120_secs_004_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_004_c_cc_d_B_a_r.xisf6752000?
005w99_M81_M82_Light_120_secs_005_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_005_c_cc_d_B_a_r.xisf6752000?
006w98_M81_M82_Light_120_secs_006_c_cc_d_B_a_r_nsg.xisf675??2000M81_M82_Light_120_secs_006_c_cc_d_B_a_r.xisf6752000?
007w101_M81_M82_Light_120_secs_007_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_007_c_cc_d_B_a_r.xisf6752000FK5
008w99_M81_M82_Light_120_secs_008_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_008_c_cc_d_B_a_r.xisf6752000FK5
009w98_M81_M82_Light_120_secs_009_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_009_c_cc_d_B_a_r.xisf6752000FK5
010w101_M81_M82_Light_120_secs_010_c_cc_d_B_a_r_nsg.xisf674.85486ICRSFK5?M81_M82_Light_120_secs_010_c_cc_d_B_a_r.xisf6752000FK5
Click to expand...
No idea!
If you could provide a link to the reference file, and two target files, I will take a look.
 
jmurphy said:
Provided the scale factor (determined from the stellar photometry) is accurate, and a single scale factor is valid for the whole image, then it is actually desirable to have the sample squares cover the nebula or galaxy. Remember, we are not trying to find the background. We are only trying to measure the difference between the reference and target image. Hence samples over the nebula or galaxy help contribute to the relative gradient model.
Click to expand...
John,
Thank you for your detailed answer. The important point for me was that it tracks the relative difference even in the galaxy area.

jmurphy said:
However, if you have slow moving, uneven clouds, then it is likely that a single scale factor will not be valid. I think this describes the challenging conditions that are typical for your location. If the gradient graph tracks the brightness of the galaxy or nebula, this usually means there is an error in the scale factor. In this situation, it may well be worth adding one or more manual rejection circle over the region that causes the gradient graph anomaly.
Click to expand...
The particular data of this post did not have large gradients, was from a dark site with very few clouds. I blocked the galaxy region and compared the gradient path vs the automatically blocked stars. It was only slightly different. I examined only one subframe, and probably should have done more. The nsg subs backgrounds everywhere was quite consistent, as expected. The resulting galaxy is looking good, but still lots of post processing to do.

Thanks,
Roger
 
First, let me update you what I have discovered so far because there may be clues here and further investigation by you may be unnecessary. @Juan Conejero, you may be interested in some of this.

In this test, I use subframes 001 and 002, which are also listed in the table in post #241.

1. The difference in the FITS keywords present originates with my image acquisition software, INDI. Some subframes (e.g. 001) have the following 13 keywords that are absent in other subframes (e.g. 002). This may depend on whether a plate-solve was performed since the last slew.
Code: 
CRVAL1  =     1.4886813095E+02 / CRVAL1                                         
CRVAL2  =     6.9310812380E+01 / CRVAL1                                         
RADECSYS= 'FK5     '           / RADECSYS                                       
CTYPE1  = 'RA---TAN'           / CTYPE1                                         
CTYPE2  = 'DEC--TAN'           / CTYPE2                                         
CRPIX1  =     4.7880000000E+03 / CRPIX1                                         
CRPIX2  =     3.1940000000E+03 / CRPIX2                                         
SECPIX1 =     1.1492184809E+00 / SECPIX1                                       
SECPIX2 =     1.1492184809E+00 / SECPIX2                                       
CDELT1  =     3.1922735580E-04 / CDELT1                                         
CDELT2  =     3.1922735580E-04 / CDELT2                                         
CROTA1  =     8.9256800000E+01 / CROTA1                                         
CROTA2  =     8.9256800000E+01 / CROTA2

2. When PixInsight reads in a subframe with these extra keywords (e.g. 001), it deletes the following keywords:
Code: 
SITELAT
SITELONG
EQUINOX
...and adds the following keywords
Code: 
COMMENT Astrometric solution by PixInsight 1.1.1-8
TIMESYS 
DATE-END
OBSGEO-L
LONG-OBS
OBSGEO-B
LAT-OBS    
RADESYS 
PV1_1     
PV1_2     
CD1_1     
CD1_2     
CD2_1     
CD2_2

This is a net addition of 11 keywords, increasing the difference in the number of keywords to 24. It is curious that PixInsight claims to have performed the Astrometric solution, which is false.

3. For RGB images, there is a new Debayer process coming out that enables separate alignment of the R, G, and B channels. This generates monochrome subframes, and the NOISE evaluation parameters are stored in NOISExx (not always NOISE00) -- for example, NOISE02 for the B subframes. Currently, NSG is looking for NOISE00 for monochrome subframes, doesn't find it, complains, and calculates its own noise parameters on the registered subframes. It reports this, and tells the user to turn on noise evaluation in ImageCalibration OR WBPP (in this case they should turn it on in Debayer).

4. The difference in the number of keywords persists at 24 until StarAlignment is run. In the subframe that lacked the RADESYS keyword, it is added in this step.

5. At the same step where RADESYS is added (step #2 for subframe 001 with more keywords, and step #4 for subframe 002), FOCALLEN is changed from 675.0 to 674.855.

6. Therefore, by the time NSG is run, the two subframes are consistent with respect to keywords RADESYS and FOCALLEN. I cannot explain why this is different than what I reported in post #241. In that dataset, all three of these keywords differed when NSG was run.

7. However, EQUINOX was removed from subframe 001 in step 2 and remains in 002 at this point. Ideally, it would seem, EQUINOX should be consistent between the two as well.

8. As for why FOCALLEN changes from 675 to 674.855, the latter may be a more accurate estimate based on other data present. Some other information from the reference frame (055), including OBJECTRA and OBJECTDEC were copied to frame 002 by StarAligment. The information transferred and/or addition of the RADESYS keyword may have been sufficient to prompt recalculation of FOCALLEN.

In conclusion, John, it seems like the keywords probably should be consistent when NSG runs and any change to FOCALLEN during NSG is likely due to recalculation based on data present, maybe during aperture photometry. This is probably not a bug/issue with NSG. You may not need the image files you requested, but let me know if you still want them.

(Point #3 above may be an issue for NSG, though).
 
johnpane said:
3. For RGB images, there is a new Debayer process coming out that enables separate alignment of the R, G, and B channels. This generates monochrome subframes, and the NOISE evaluation parameters are stored in NOISExx (not always NOISE00) -- for example, NOISE02 for the B subframes. Currently, NSG is looking for NOISE00 for monochrome subframes, doesn't find it, complains, and calculates its own noise parameters on the registered subframes. It reports this, and tells the user to turn on noise evaluation in ImageCalibration OR WBPP (in this case they should turn it on in Debayer).
Click to expand...
Thanks for reporting this. I will get a fix out as soon as possible.
 
johnpane said:
3. For RGB images, there is a new Debayer process coming out that enables separate alignment of the R, G, and B channels. This generates monochrome subframes, and the NOISE evaluation parameters are stored in NOISExx (not always NOISE00) -- for example, NOISE02 for the B subframes. Currently, NSG is looking for NOISE00 for monochrome subframes, doesn't find it, complains, and calculates its own noise parameters on the registered subframes. It reports this, and tells the user to turn on noise evaluation in ImageCalibration OR WBPP (in this case they should turn it on in Debayer).
Click to expand...
I have update NSG for the new Debayer 'Separate RGB channels' option. Please let me know if this works.
Thanks, John Murphy
 
johnpane said:
2. When PixInsight reads in a subframe with these extra keywords (e.g. 001), it deletes the following keywords:
Code: 
SITELAT
SITELONG
EQUINOX
Click to expand...

SITELAT and SITELONG are nonstandard and obsolete. The latest FITS standard version 4 enforces the use of OBSGEO-B and OBSGEO-L, respectively. EQUINOX is obsolete and should not be used; the standard RADESYS keyword makes it unnecessary and misleading. SECPIXn keywords are redundant, nonstandard and obsolete.

...and adds the following keywords
Code: 
COMMENT Astrometric solution by PixInsight 1.1.1-8
TIMESYS
DATE-END
OBSGEO-L
LONG-OBS
OBSGEO-B
LAT-OBS  
RADESYS
PV1_1   
PV1_2   
CD1_1   
CD1_2   
CD2_1   
CD2_2
Click to expand...

These keywords are the recommended standard ones, except LONG-OBS and LAT-OBS, which we generate for compatibility with legacy software.

It is curious that PixInsight claims to have performed the Astrometric solution, which is false.
Click to expand...

At this point PixInsight has recalculated and regenerated the entire astrometric metadata in the image (with more accurate and fully standards compliant items by the way), so it is not false at all.

5. At the same step where RADESYS is added (step #2 for subframe 001 with more keywords, and step #4 for subframe 002), FOCALLEN is changed from 675.0 to 674.855.
Click to expand...

As noted, we regenerate the astrometric metadata with reasonably accurate values.

Please note that the separate CFA channel feature is still under development and has not been released officially.
 
Just to be more precise about the RADESYS and EQUINOX keywords, this is the logic of our implementation, which is fully standards compliant:

- If the RADESYS keyword exists, then EQUINOX is deleted and not generated if RADESYS is one of 'ICRS', 'GCRS', or 'GAPPT'. Otherwise EQUINOX is preserved if it exists or otherwise generated with the value 2000.0.

- If neither RADESYS nor EQUINOX exist, then RADESYS is generated with the value 'ICRS' and EQUINOX is not generated.

- If RADESYS does not exist and EQUINOX exists, then RADESYS is generated with the value 'FK4' if EQUINOX < 1984.0 and 'FK5' if EQUINOX >= 1984.0.

Please note that RADECSYS is nonstandard and hence ignored.
 
Juan Conejero said:
- If the RADESYS keyword exists, then EQUINOX is deleted and not generated if RADESYS is one of 'ICRS', 'GCRS', or 'GAPPT'. Otherwise EQUINOX is preserved if it exists or otherwise generated with the value 2000.0.
Click to expand...

Note that in my exploration yesterday, StarAlignment added the RADESYS keyword when missing (explicitly, or by copying astrometric data from the reference frame) but did not delete EQUINOX.
 
jmurphy said:
I have update NSG for the new Debayer 'Separate RGB channels' option. Please let me know if this works.
Click to expand...

Looks like it works. I just tested this on B frames generated by the new Debayer separate channels option, and NSG used the NOISE02 weights.

Thank you!
 
NormalizeScaleGradient v1.4.2

This is the version of NSG that is included in the PixInsight 1.8.8-9. I have also updated the NSG documentation for the 1.8.8-9 release.
 

Attachments

  • NormalizeScaleGradient.zip
    105.1 KB · Views: 122
Last edited:
I find writing documentation to be hard and very time consuming, but I struggle through it in the hope that it will be useful. I would recommend starting by reading the 'Quick Start Guide' section. Then use the rest of the document as a reference guide. There are lots of hyper links in the document (use the browser back button to return to what you were reading) to help with this.
 
Hi John, loving the script- Thank you!

....probably a dumb question, but have the NSG weights essentially replaced using the SFS weights? Is there anytime you would use one process over the other?

Thanks
Carlos
 
cdavid said:
... have the NSG weights essentially replaced using the SFS weights? Is there anytime you would use one process over the other?
Click to expand...
For star clusters / globular clusters, I would try to optimize the star resolution. I think SFS weights will often be the best way of doing this. Stars are very sensitive to 'seeing' and tracking. This is because they are bright point sources. Even a tiny percentage of the star light can cause a very noticeable star bloat.

For galaxies / nebulae, the resolution of the faint areas of an image are almost always limited by the shot noise. The tracking errors and atmospheric distortions are usually insignificant in comparison. I would therefore always use NSG to calculate the weights for these objects, and fix star profiles in post processing. ImageIntegration usually does a good job of rejecting the star bloat from the frames that had poor star profiles.

That said, I would probably still weed out frames with extremely bad star profiles.
 
Last edited:
Good day @jmurphy !

I would like to ask you if the plan for the "drizzle feature" is quite far away.

In addition to that, one small feature request: Is it possible to add a column on the Target Images board, just to count the images? # 1 # 2 etc.

Keep up the good work!


Best regards

Euripides
 
Hi John,

I'm trying to understand the effect of your comment:

"I have update NSG for the new Debayer 'Separate RGB channels' option."

Do you have to run NSG on each RGB channel separately or, if all the debayered images are added, will NSG process them as appropriate? I assume the former, as there is only provision for one reference file.

If the former, then is it appropriate to run the auto post-NSG Integration on each channel?

Finally, presumably you would then continue to process the separate channels until combining them before the first stretch?

Thanks, Jim
 
JST200 said:
I'm trying to understand the effect of your comment:
"I have update NSG for the new Debayer 'Separate RGB channels' option."
Click to expand...
This was an internal change. It does not affect usage. Use NSG as before.

This internal change was needed for input files created by the Debayer 'Separate RGB channels' option. This creates 3 mono files. The previous version of NSG would expect these mono files to store the noise estimate in the FITS header NOISE00. However, these mono files 'remember' that they were originally from a color image, so the 'red' mono file uses NOISE00, the 'green' mono file uses NOISE01 and the 'blue' mono file uses NOISE02. I therefore needed to update NSG so that it could find the noise estimate.

JST200 said:
Do you have to run NSG on each RGB channel separately or, if all the debayered images are added, will NSG process them as appropriate? I assume the former, as there is only provision for one reference file.
Click to expand...
Yes, it is the former. If the reference is a mono image, the reference and all the target images must be from the same filter.

JST200 said:
If the former, then is it appropriate to run the auto post-NSG Integration on each channel?
Click to expand...
Yes.

JST200 said:
Finally, presumably you would then continue to process the separate channels until combining them before the first stretch?
Click to expand...
There is no right or wrong answer here. It depends on your work flow.
 
Status
Not open for further replies.
Back
Top