who tests these update/fixes that break things!

[yock1960]

I updated a day or two ago and it seemed to work until I integrated 6 images, cropped and the PI crashes on DBE. I saw there were a couple of more updates, so I applied those. Same deal. Frustrating and I hope that no one tells me that I need to upgrade my 32bit Windows 7 to 64 bit!

 I'm going to say that the update to the ImageIntegration module (1.8.1.124) did it and .125 hasn't fixed it. This is not the first issue I've had with this module, but setting buffer size to 8MB and stack size to 512MB had tamed it. This is different though, before it would crash when integrating. I've never experienced a crash with DBE before.

Win7 32B
4 GB


How do I go back to what works? I keep forgetting that I shouldn't apply fixes/updates to things that aren't affecting me.

Steve

[yock1960]

FWIW, I replaced ImageIntegration-pxm.dll with the version (1.8.0.117) from my laptop and the DBE crash stopped. Time will tell as I go through my normal routine, whether this is really 'fixed'.

Does there need to be some type of extra info and the ability to opt out of some updates? Something like...."if you are having a problem with xxx, this update may help" or splitting releases into 'stable releases' and 'beta releases'?

Steve

[Juan Conejero]

Hi Steve,

In first place, I want to state clearly that we do not release beta or unstable versions through the official PixInsight update repository (update.pixinsight.com). All updates are tested and we release them to fix confirmed bugs and to deploy improved versions of standard and new tools and scripts.

The latest updates (FITS support module and ImageIntegration module) have nothing to do with the BackgroundModelization module, which is where the DBE process is implemented. It is unlikely that the new versions of these modules change the performance or behavior of the DBE process, mainly because all of these processes are unrelated and don't communicate in any way. As you know, PixInsight is a modular system.

I updated a day or two ago and it seemed to work until I integrated 6 images, cropped and the PI crashes on DBE.

I hope that no one tells me that I need to upgrade my 32bit Windows 7 to 64 bit

setting buffer size to 8MB and stack size to 512MB had tamed it.

It seems that what you're experiencing is out of memory issues, which are typical of 32-bit operating systems with the ImageIntegration tool. As you have said, lowering the buffer size and stack size parameters will fix these problems. You can set buffer size to zero, which will force II to integrate row by row.

[pfile]

there might be a relationship between ImageIntegration and DBE - i know that DBE's algorithms can get fooled by black borders left by II, for instance. of course, i've never seen DBE crash, but i'm just putting that out there.

[Carlos Milovic]

It may be that RAM memory got too fragmented, and allocation of memory for DBE's calculations just failed. It could have happened with any memory intensive process.

[pfile]

could be, but when malloc returns null, do you then crash the application?

[Juan Conejero]

when malloc returns null, do you then crash the application?

Of course not. All bad_alloc exceptions thrown by failed new operators are catched by internal PCL code, if they are not already catched by the module where they happen. That been said, an out of memory error is always a risky situation because we cannot control completely all third party libraries and routines involved in a complex platform like PixInsight.

A very important factor in these discussions is how the word 'crash' is being defined and used:

(a) The application terminates abruptly and control is returned to the operating system.

(b) A critical system error occurs, which is catched by the application. For example, a Win32 system exception on Windows or a critical signal on Linux/UNIX.

(c) A specific part of the application (a PixInsight process for example) shows some erratic or buggy behavior.

(d) Everything works correctly but a process fails for some controlled reason; for example, an out-of-memory error, insufficient disk space, insufficient access rights, invalid parameters, ill-posed data, etc.

The standard definition is (a). Unfortunately, the most often used version is (d). We developers are trained to perceive the underlying definition of 'crash' in each context

[pfile]

fair enough, but to me crash means a), i was not aware of any other semantics for that word

[Andres.Pozo]

For many of my clients "crash" has the meaning d) of Juan's list 

Also, recovering from "out of memory" errors in complex GUI applications is not always possible. Most of the actions that you have to do to recover from the error (Message Boxes, writing in a log, even closing windows) need to reserve new memory. Also, in many cases, "out of memory" errors come from the fragmentation of the memory and the only real solution is restarting the application.

[pfile]

yes i know, tell me about it... about 1/2 the time i try to back PI out of a hang with gdb, it ends up segfaulting somewhere. thankfully these days PI is much more stable and i have not had to do this in a long while.

[yock1960]

Test #1

Exit all programs, rename file ImageIntegration-pxm.dll dated 4/7/2012 to old_ImageIntegration-pxm.dll.

Rename file new_ImageIntegration-pxm.dll dated 6/21/2012 to ImageIntegration-pxm.dll.

Restart Windows.

Start PI

Start ImageIntegration, set buffer size to 8MB, stack size to 512MB, rejection to Linear Fit.

Integrate 6 previously aligned images sized 2,833KB each, setting image #3 to the reference image.

Close the rejection & slope windows.

Perform STF, perform DynamicCrop.

Open DBE, select image, generate samples at default settings, change Minimum Sample Weight to .500, change Tolerance to 1.00, re-generate samples, set Image Correction to Subtract, discard background model....apply to image.

Program terminated....Windows Error message attached.

Test #2

Rename file ImageIntegration-pxm.dll dated 6/21/2012 to new_ImageIntegration-pxm.dll.

rename file old_ImageIntegration-pxm.dll dated 4/7/2012 to ImageIntegration-pxm.dll.

Restart Windows.

Start PI

Start ImageIntegration, set buffer size to 8MB, stack size to 512MB, rejection to Linear Fit.

Integrate 6 previously aligned images sized 2,833KB each, setting image #3 to the reference image.

Close the rejection & slope windows.

Perform STF, perform DynamicCrop.

Open DBE, select image, generate samples at default settings, change Minimum Sample Weight to .500, change Tolerance to 1.00, re-generate samples, set Image Correction to Subtract, discard background model....apply to image.

DBE executes normally...PI user is happy!

I never had any problems with ImageIntegration until version 1.7 if I recall correctly. It was about that time I saw other posts...all Win32 users I believe, also have problems of a similar nature. One of those threads is where I picked up changing the default settings in II, which worked well until the last two updates of II. Back then, I felt that something had changed in II...that it was....if not broken...then it certainly acted differently in that I had to change the default settings. I was content that it worked with the reduced memory setting however.

I was happy with my 'decrepit' 32 bit Win7 Home Premium with 4GB of memory. Now II is updated further, along with some other files. It builds the image, but soon PI terminates without my permission (a crash) in a repeatable fashion on my machine, when working with only 6...shall we say medium size images.

I go back to an older version of II and all is well.

What is the critical X?

[Juan Conejero]

Steve,

Nothing in the ImageIntegration module has changed in any way as to generate a crash as you describe. These are the three updates to the ImageIntegration module that we have released for PI version 1.7:

2012 March 28 (version 1.8.0)

This update installs a new version of the standard ImageIntegration tool. Version 1.8.0 provides direct integration of file formats with and without incremental file read support. This means that all supported file formats can now be integrated, including DSLR raw files such as CR2, NEF, etc. Previous versions only allowed integration of file formats with incremental read capabilities, which in practice restricted this tool to the FITS format.

2012 June 15 (version 1.8.1)

This update installs a new version of the standard ImageIntegration tool. Version 1.8.1 is an update/bugfix release:

* Fixed a bug that was leading to a crash on Windows when the percentile clipping rejection algorithm was selected for images with black pixels. For more information see the corresponding bug report:
http://pixinsight.com/forum/index.php?topic=4348.0

* Fixed wrong cache invalidations on Windows due to unreliable millisecond resolution in disk file times. Bug report:
http://pixinsight.com/forum/index.php?topic=4223.0

* Improved generation of FITS header keywords with exhaustive information on image integration parameters, pixel rejection parameters, and SNR increments.

2012 June 21 (version 1.8.2)

This update installs a new version of the standard ImageIntegration tool. Version 1.8.2 is a bugfix release:

* The cache duration and enabled state were not being serialized, and hence were not being saved and restored across sessions. This bug has been fixed. For more information refer to the corresponding bug report:
http://pixinsight.com/forum/index.php?topic=4223.msg30768#msg30768


To understand what happens, I need to reproduce the same out of memory error you're experiencing on a 32-bit Windows machine. I need some more information:

- The geometry of these six images: width and height in pixels, number of channels.

- The data type of the images (I guess they must be 32-bit floating point images).

- Are you running other applications while you get this crash in PI? What antivirus and/or firewall application(s) are you using? Other relevant background tasks running?

- Can you check if the same problem happens on another 32-bit Windows machine?

- Are there other Win32 users who are experiencing a similar crash in DBE after the ImageIntegration updates?

[gvanhau]

Hello
Only a hint.
It can be that the new integration module uses a litle more memory or leaves the memory a litle more fragmented. Or may be it doesn't release all the memory used?.
May be yock1960 could use windows task manager or resource monitor to see if memory used in each alternative is the same.

Regards
Geert

[yock1960]

Juan,

I guess that since all 32bit versions are obsolete, it doesn't matter, but to answer your questions:

- The geometry of these six images: width and height in pixels, number of channels.
 
  1392 x 1040, 1 channel

- The data type of the images (I guess they must be 32-bit floating point images).

  FITS 16 bit integers

- Are you running other applications while you get this crash in PI? What antivirus and/or firewall application(s) are you using? Other relevant background tasks running?

  Typically, I may have Firefox (13.0.1)  running, Nebulosity2 (2.3.6) almost certainly; background apps showing up in the tray are avast! antivirus (free version),  OpenOffice 3.3 Quickstarter, Norton Ghost, LightScribe Control Panel, Spyder3Express, APC Powerchute, Catalyst Control Center, Windows Sidebar. I just use Windows' firewall.

When I ran the tests, PI was the only 'foreground' app running.

- Can you check if the same problem happens on another 32-bit Windows machine?

Yes. I'll get back to you on that. It will be Vista Home.....Premium I think.

- Are there other Win32 users who are experiencing a similar crash in DBE after the ImageIntegration updates?

Not that I'm aware of. How many 32bit users are left....am I the last one? At least I now have a concrete reason to upgrade....sometime in the next month or two or three...when I feel like dedicating a couple of hours to it.

I'm working fine with the II from April, so I guess this is all a 'tempest in a teapot'. I just won't apply any upgrades until I go to 64bits.

Steve