PixInsight New Version Problem

[ajbarr]

Several weeks ago I posted that the new version of PI would not run on my older iMac. I decided to bite the bullet and buy a brand new one with a 2 TB SSD drive and 16 megs of memory. The upgrade runs fine on the new computer however I ran into a problem today when processing some data. After calibration and registration I ran Image Integration and the results were terrible. The first thing I noticed was the pixel rejection screen was inverted, white background and black nebula. I have never seen that happen before. Needless to say, the stack image was horrific. After looking at all the subs several times and changing some settings, I still got the same results.

Then I renamed the new app file and loaded the older version. I processed all the data again and there was no problem. My only assumption is there may be some kind of bug in the new software that still makes it incompatible with Apple.

Does anyone know anything about this yet?

Thanks

Albert

[Juan Conejero]

Hi Albert,

The latest version 1.8.7 of PixInsight is indeed compatible with macOS. We have thousands of users who use it on macOS on a regular basis, with excellent results.

The problem you are describing looks like the result of wrong pixel rejection caused by an image registration issue. The new version of the StarAlignment tool performs PSF fits for all detected stars by default. If some frames in the data set are out of focus, registration may fail because flat star profiles lead to inaccurate PSF fittings, which can be rejected. If you have integrated unregistered frames, pixel rejection can cause this type of strange effects. We need the calibrated frames to further help you.

[ajbarr]

Thanks Juan. What I don't understand is when I ran the older version of PI everything seemed to look fine.

[Greg Schwimer]

Hi Juan,

Albert and I are working with the same data. I can reproduce this on Linux. It's related to the new Subtract Pedestals option in ImageIntegration and the default of Clip Low being enabled. The data in this case has a pedestal keyword value of -100 (yes, negative). Is it possible that Image Integration is using the absolute value of 100 and subtracting that rather than adding (i.e. -(-100) means add 100). I just changed the pedestal value in all of these files to 100 (not negative) and I get the same result so I suspect this may be the case. The workaround for this data set is to disable Subtract Pedestals.

I believe the negative pedestal may be related to SBIG driver settings. I don't have the camera so I can't confirm. If this is the case it may cause problems for some users working with low signal data like narrowband, as is the case for us.

BPP does not have these knobs in the intergration settings so there is no workaround for this there.

Greg

[Juan Conejero]

Hi Greg,

Quoted from the official version 1.8.7 announcement post:

* New subtract pedestals option. When this option is enabled, existing (nonstandard) PEDESTAL keyword values are subtracted from input images. Some applications add small positive values (typically about 100 DN) systematically to calibrated light frames. These small pedestals should be subtracted from source integration pixels to refer all input data to the same zero point consistently. This option should be enabled under normal working conditions, and hence is enabled by default. PEDESTAL keywords generated by PixInsight always have positive values. Other applications may write negative values (another 'nice' example of FITS interoperability issue). For improved compatibility, when ImageIntegration finds a negative pedestal, it issues a warning message and assumes that the value is intended to be added.

So negative PEDESTAL keyword values are being added, that is, actual pedestals are being subtracted as they should. If the new subtract pedestals option is causing these problems, then for some reason the PEDESTAL keywords are incorrect. This probably means that the pedestals in question have already been subtracted.

Do you still have the original raw data? If this isn't asking for too much, can you repeat the entire preprocessing from scratch with version 1.8.7?

[Greg Schwimer]

Ah - yes, I see that. So the behavior is as expected, so something else must be at play. Also a factor (now that I think of it) is some workarounds I had to put in place to get the data to calibrate properly via BPP in 1.8.6. For thoroughness I'll share this post:

https://pixinsight.com/forum/index.php?topic=13887.msg83846#msg83846

I'll admit I'm not used to using pedestals so part of the problem may be related to that.

In the case of our data, all calibration and light frames have a PEDESTAL keyword with a value of -100. When running through BPP, I found that some dust artifacts were not being removed correctly (overcorrection) with the application of flats (via BPP). I managed to resolve the problem simply by taking the dark and bias masters, re-adding the PEDESTAL -100 value to the files, and using them to create master flats (with BPP). I then had to add the same keyword values to the flats. From there, the lights calibrated properly, or at least they appeared to come out with a clean result. I do, in fact, still get a clean result using these modified masters *if* I uncheck the Subtract Pedestal option in the Image Integration process. No luck with BPP as that option is not there yet.

This is a set of data specific to Albert's original problem stated at the start of this thread.

<>

I put a expiration on this download as I need to be considerate of those on the team I'm working with for this project.

I just re-ran BPP with the data from above from scratch (no calibration masters just subs) and it seems I still get the same initial result of overcorrection as with version 1.8.6. I then added the -100 pedestal keyword to the flat, bias, and dark masters and re-ran. I now have mild undercorrection. If I go back and use the masters I made in 1.8.6 using the process I mentioned above, everything is fine. This is possibly because BPP using Image Integration is subtracting the keywords when creating the masters, whereas the masters I made in 1.8.6 did not have the pedestal subtracted and I added the keyword.

EDIT: added additional test results.

[Greg Schwimer]

Here's an example of three methods of BPP processing of data with PEDESTAL -100 set by the imaging software in all calibration and light frames. I ran these tests in 1.8.7 only. The light frames are of a different object in this case, but the source calibration data is the same.

 - from_subs - BPP in 1.8.7 using subs only, no masters - overcorrection
 - set_PEDESTAL_minus100_187 - took the master calibration frames, added keyword PEDESTAL -100 to the fits header, re-ran through BPP using them - undercorrected
 - from_186_masters - same as above, except masters were created in 1.8.6 - corrected, fewer hot pixels too

The same BPP parameters were used for all of these. The last one appears to be a correct result. I'm guessing this is because in 1.8.7 BPP is subtracting the pedestal from the calibration frames.

[Juan Conejero]

Hi Greg,

The problem seems to be that negative PEDESTAL keyword values (e.g. -100) are not being applied correctly by ImageCalibration. I have fixed this problem in version 1.8.8, which I am preparing for release right now. However, an additional check with your data would be great for confirmation. If possible, can you please upload (a possibly reduced subset of) raw frames from this set, including bias, darks, flats and lights? I need the original raw frames, just as produced by your acquisition application. Thank you!

[Greg Schwimer]

Juan,

Thanks again for digging in on this with us. I PMed you a link with data you can use to test the behavior out. I can also beta test 1.8.8 if you'd like. I run linux over here which I think you do as well so if that's helpful I'm up to the task.

- Greg

[dbrahmbhatt]

Hi Juan,
I am working with the same data as well. What I am finding out is that some of the other channels seem to calibrating fine (right now, I am working with NGC55, not sure if Greg shared that particular data with you or not) - but NGC55 data that Greg and I have - I can calibrate Blue fine, but not Lum (I haven't tested other channels, but will do so in a bit and post the results). I know there was an issue getting Blue and Green initially, but that was fixed.

For some background - I am using the normal ImageCalibration function as compared to BPP that Greg seems to be using.

I don't mind testing a beta 1.8.8 as well if it is truly a Pedestal issue.

Thanks,
Dhaval

[Juan Conejero]

Hi Greg and Dhaval,

Thank you so much. You have a PM.

[Greg Schwimer]

Tested in 1.8.8. Seems to solve the problems.