Posible bug in Image registration and Image integration

[gvanhau]

Hello
Since the last update, where image registration and image integration are now using multiple cpu cores, I am getting random PI crashes.
Sometimes it chashes wit few procesed images, other times it crashes after processing lots of images.
After each crash, PI is aborted and has to be started again.

I could reproduce this issue in two different computers using windows 10.

I like the performace increase of both proceses, so hopefully this bug is solved asp. 

Regards
Geert

[avastro]

Hi Geert,
Using last Registration and Integration update process intensively in a Win 10 PC without any crash here till now.

Antoine

[Juan Conejero]

Hi Geert,

I cannot reproduce this. I've just conducted a battery of tests on Windows 10 without any issue with StarAlignment and ImageIntegration. These tests use 300 images and are quite intensive. The machine is an iMac 5K running Windows 10 on Bootcamp.

These problems can be caused by a variety of factors, including running background applications (virus protection, firewall or utility software) and hardware related issues. Bear in mind that these processes are now much more intensive in their memory usage. Since all available processor cores are now used intensively, there can be also thermal issues, especially on laptops and insufficiently cooled desktop machines.

Questions:

- Can you describe the machines where this has happened? (brand, processor, memory available, disks, etc.)
- Which virus protection software are you using?
- How are the images you are processing? (DSLR? FITS? XISF? image dimensions?)

[gvanhau]

Hello Juan
Thank you for your answer.
I know this is a difficult issue to analyze.

I also know, there are/were other processes in PI that were running on multiple cores long before the last upgrade, an they did never crash on my computers.

Im currently working on several sets of 100 to 200 fits images of a Kaf 8300 mono camera (8M pix); the same sets before the upgrade did never crash.

Not that I'm willing to discard a HW issues a priory, but I think it is not likely thay 2 computer suffer from the same issue at once.

Both computers are desktops; one using a Phenom 4 core AMD processor and 8GB ram the other using a FX AMD processor and 32GB ram the last one has a liquid cooler - radiator; I monitored the temp CPU and several components in M.Bo and none ever rised above 48° C, so I dont think this is a temp issue.

The FX processor is running at its rated 4 GHz speed, but even limiting (downgrading) the processor speed and /or ram speed prevented the crash. Only limiting PI to use 1 core on global settings does its job ok. 

On the Phenom computer I am using Kaspersky antivirus, on the FX computer I'm using Norton.

I dont think that running this on a virtual machine is a throughout test, since the VM introduces a lot of other stuff between OS and HW and/or Bios which can alter timing.

Regards.
Geert

[Juan Conejero]

I dont think that running this on a virtual machine is a throughout test, since the VM introduces a lot of other stuff between OS and HW and/or Bios which can alter timing.

Note that the machine I have used to test these processes is physical, not virtual. It is an Apple iMac Retina 5K computer with an Intel Core i7 6700K @ 4 GHz and 32 GB of RAM running Windows 10 on a separate partition.

Definitely, I cannot reproduce this problem on any platform. We only have Intel processors, none of our working and testing machines has an AMD processor, although I see no reason why this might cause problems. I'll try to investigate this further.

You can disable high-level parallel processing in SA and II by setting the useFileThreads parameter to false. Click the Edit instance source code button, change the parameter value, and click the commit button.

[msmythers]

Geert

I've been running some test on my Windows 7 computer with StarAlignment and ImageIntegration without any issues. I'm running all the latest PI updates. My computer is a AMD Phenom X6 1090T with 16GB ram. I'm running Avast Antivirus but I have PI excluded from the anti-virus. During the testing I ran tests with 180 and 350 subs. I'm not overclocking the CPU and not liquid cooling. The machine is now 6 years old. I also ran video processing software during some testing to stress the system a bit more. Just to be complete I am running 2 SSD's, one for the OS, programs and first set of swap directories and the second for a second set of swap directories. All data is kept on a standard hard drive.

I did have a problem with ImageIntegration crashing after the December 20th update but that was a DSLR_Raw issue which Juan promptly corrected. http://pixinsight.com/forum/index.php?topic=10581.0



Mike

[gvanhau]

Hello Mike

I'm not overclocking my machine, I always used it at the rated values. At contrary, I'm currently downgrading it . My RAM is rated at 1.866GHZ and Im now using it at 1.333 GHZ
My CPU is rated at 4GHz and Im now using it at 3.6 GHz. 

The only APP that crashes is PI, all other concurrent tasks still work. I can immediatelly restart PI and test again, without rebooting. If there was a HW issue most likely it should also affect other running app's  but it doesnt.

Obviously, I can not completelly discard HW issues on both machines, and therefore,  I am still testing every thing again and again. 

Regards

[IanL]

ImageIntegration is broken for me too since the latest update to multi-core, whereas I have never had a problem previously.  I am running Windows 10 Home, Core i7 920 at the stock 2.67GHz (not overclocked), 18GB Ram.  See the attachment for the basic ImageIntegration settings which will consistently crash PI.

I can successfully integrate (say) 10 bias images, but if I increase the number to 100 images PI will always crash.  With a "Buffer Size" of 16 and a "Stack Size" of 4 and 100 images, the process will run for a few seconds (images loaded messages appearing on the console) and then crash.  If I increase "Buffer Size" to 64 and "Stack Size" to 12 and 100 images, the process will crash as soon as it is run with no messages on the console.

This is consistent repeatable behaviour.  I have successfully Integrated up to 300 bias frames on the same machine with both the default buffer/stack settings and increased ones prior to the update with no issues (slightly larger DSLR frames than the ASI1600 frames I am trying now).

I have another Windows 10 machine with PI installed that I haven't installed the latest updates on so I'll give things a try on that machine with the same process and report back.

In the meantime is there any way to roll back the updates to (just) the two new modules?

Thanks

[IanL]

The work-around is described by Juan here for anyone having this issue:

http://pixinsight.com/forum/index.php?topic=10603.msg66454#msg66454

1. Set up your process as normal.
2. Drag blue triangle to desktop to create a process instance icon.
3. Right-click the icon and choose "Edit Instance Source Code..."
4. Change the highlighted line in the screenshot below so that it says "false" instead of "true".
5. Click the green tick a the upper right of the editor window to save the change.
6. Close the editor window.
7. Double-click the icon to launch the process again.
8. Run as normal and it will revert to the old single-core behaviour and should work fine.

Hope this helps until the issue is fixed.

[sbabino]

Señores

A mi también se me "cuelga" el equipo integrando imágenes. La cámara es una Stf8300m y Px corriendo en un Lenovo Y50 Core I7 con 6gb de ram y corriendo Bitdefender con Windows 10 como sistema operativo
Nunca tuve problemas con esto , hasta esta actualización.
Trabajo con Fits e integrando 30 lights ,30 darks ,30 flats y 60 bias se cuelga siempre.
Sepan disculpar que escribo en español ya que soy de Uruguay y se me complica con el inglés , aparte creo que este post lo va a leer el señor Juan Conejero.
 

Muchas gracias

Sergio Babino

[Juan Conejero]

I have just released an update to the FITS format support module that should fix these problems on Windows.

The issue was being caused by the Windows implementation of the CFITSIO library, which seems to have thread safety problems. The update prevents concurrent calls to CFITSIO routines on the Windows version of PixInsight. This will decrease performance a bit with fast I/O hardware, but there's no better solution. Hopefully the performance penalty will be negligible in most cases. Of course, PixInsight's native XISF format implementation is guaranteed to be fully thread safe on all platforms, so these problems cannot happen.

Please let me know if this update fixes all of these problems with the new ImageIntegration module on Windows.

[msmythers]

Juan

Could one do a batch format conversion of fits to xisf?


Mike

[Juan Conejero]

Hi Mike,

Sure, you can use the BatchFormatConversion script.

[gvanhau]

Thank You very much Juan.

I installed the update and tested it several times on the FX PC. Could not krash PI anymore.
Also reverted the clocking of my PC to normal and tested again wo issues.

The load on the machine is a litle less than before (98-99% CPU vs 100%), but speed is very, very high compared to the mono thread version.

For me this issue is solved.

Regards
Geert

[Juan Conejero]

Hi Geert,

Glad to know it's working well. As you've noted, the performance loss is marginal.

As for why this has caused problems on some machines and not in others, even with the same versions of Windows... IMO, the reason should be looked for in different versions of critical system components. Some applications, especially old, poorly maintained software (abandonware), can cause these problems easily on Windows. This does not happen on UNIX/Linux.