Author Topic: Release 1.8.6 Repeating Crashes  (Read 4568 times)

Offline Palmito

  • Newcomer
  • Posts: 8
Re: Release 1.8.6 Repeating Crashes
« Reply #15 on: 2019 January 25 08:46:37 »
Hello!

I am running a Win10 64-bit, Intel i7 6700K, 64Gb RAM, GTX-1080-6GB, Samsung evo 960 SSD with a 256Gb pagefile. I have been running PixInsight 1.8.5 with no issues whatsoever apart occasional crash (once or twice in one year).

I am very sorry to say but I am encountering many issues with build 1448.

I am still using old subframeselector script and it crashes very often with the same error.
There are two types of crashes, one were PI just closes and one with the same message as others:
PCL Win32 System Exception
At address 00007FFF63495470 with exception code C0000005:
Access violation: invalid memory read operation at address 0000000000000008

Also after some time it seems all js files fail to run. Typical errors are like:
*** Error [000]: C:/Program Files/PixInsight/include/pjsr/NumericControl.jsh, line 153: Error: At address 00007FFAFD1A5470 with exception code C0000005 :
Access violation: invalid memory read operation at address 0000000000000008

I have uninstalled and reinstalled both pixinsight and my nvidia drivers (including clean install) and it didn't help.
I have tried running PI with QT rendering in both software and rasterized, unfortunately with no success. So I reverted back to opengl.
I noticed subframeselector started working good again after I freed some space on my ssd (had 100gb free and freed an additional 300gb). But it only worked for a day or two. When I reached 250gb free space issues occurred again. I freed another 150gb but it still fails.
PI temp folder is currently 135Gb, which seems a bit higher than running 1.8.5.

I have also often encountered the following issue: If I switch to anther window and let PI sit for a few minutes in the background doing nothing it very often crashes when switching back to it.

After using PI for sometime it slows down at some point. I especially notice it in DrizzleIntegration as it slows down by a factor of ~50 but I believe it is generalized. Also when this happens my process icons starts flickering. Most of the time restarting PI is sufficient to fix, but sometimes a reboot is required.
When a reboot is required I notice GUI refresh is very slow upon reopening.

Build 1457 crashes far more often for me, and no matter what I am doing, it is very painful as when it starts failing I cannot save my progress.

Message is now only printed in console (At address 00007FFF63495470 with exception code C0000005) rather than a message box. But the issue iust that after a minute or two PI becomes unresponsive and crashes.

I will conduct more testing on build 1457, I am sorry if I am reporting fixed issues.

Those issues seems quite random and I haven't been able to reproduce on demand. Though switching between windows and desktops while PI is processing seems to make it worse. And as being a developer myself I know this is the worst situation because you cannot reproduce and while you cant reproduce you cant fix...
My personal feeling (because of the flickering, the free space and the js script crashing when trying to set GUI field) is that it might have something to do with QT/GUI and the hard disk access. But its only a gut feeling, I got no strong evidence for that ;-)

Let me know if I can provide anything, I will be glad to help!


I still noticed a very good overhall perfomance improvement and I believe it is an excellent release. Looking forward for a new build :-)

Offline Palmito

  • Newcomer
  • Posts: 8
Re: Release 1.8.6 Repeating Crashes
« Reply #16 on: 2019 January 31 06:44:16 »
Hello,


So after more thorough testing, as promised, here are my results:

1 - Subframe selector script crashes as often as for build 1448, however errors are now only printed to the console and not in a modal interrupting message box. I have this issue no matter the files I use (even ones that worked fine either with v1.8.5 or v1.8.6).
Sometimes it happens after 3 files have been measured, sometimes after hundreds. Sometimes it is only a single file, sometimes it is a bunch of them in a row or close to each other. I fail to find any pattern other than the more files you add to the list the more probable it will occur.
I am convinced it is not linked to the data, as sometimes a particular file will work sometimes it won't, but if you wish let me know and I'll be happy to provide you my files.
My tests were conducted prior the module update I noticed today. But as I am using the script I believe my tests should still be valid, unless there are any dependencies with the module. Please let me know, and I will re-conduct tests if needed.

2 - About general use it seems crashes happen especially when I use Star mask module several times (on a small image, details attached).
I have a difficult picture with nebulosity that gets picked up and I am doing trial and error with star mask parameters. So I end up generating a lot of new views (I close them as I generate new ones tho).
After sometime it systematically crashes on my setup with the same error I reported in my previous post (after maybe 20 or 30 star masks have been generated).
I notice it crashes when I try to move new star mask window and it doesn't (instead error is printed in console). If I try closing it, each time I try the same error gets printed on the console after about 10 tries PI shuts down non gracefully.
I found a workaround for this! I believe this might help you pinpoint the exact issue, and help others experiencing the issue...
Interestingly, if instead of left click, I right click on the close button it works. The faulty view is closed with no errors and I can resume work without having to restart PI.


3 - For the performance sudden drop, It is still present but I have measured it a bit more precisely. I would say the slow down factor is closer to 25 times rather than 50.
With build 1457 restarting PI is not sufficient as a workaround but a reboot works all the time.

4 - The bug about opening PI and letting it sit in the background leading to a crash has never occurred on build 1457.



I also wanted to share more information about my system/usage of PI:
A - In PI I make use of 3 workspaces and have tens of process icons on each of them.
Upon opening a project with only process icons and no images, each process dialog appears and disappears. Looking like flickering and is slow to open even-though I have it set on workspace 4 containing no process icons.
This might be normal or already happening in v1.8.5, sorry I don't remember. But I didn't want to discard any information that might help you.

B - In my last post I didn't precise that my SSD is an M.2 2TB drive.
It is partioned in two disks, one containing OS, and one PI data, temp folder and pagefile.
Maybe somehow it could have an impact.

C - I also forgot to precise that all my drives have been trimmed, error checked and have windows indexing disabled.

D - PI swap storage folder is not a system temporary folder.

E - My motherboard is an ASUS Z170 PRO GAMING with latest BIOS installed.




Please kindly let me know if I can provide further help.


Looking forward for your response.

Cheers.
« Last Edit: 2019 January 31 08:33:18 by Palmito »

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Release 1.8.6 Repeating Crashes
« Reply #17 on: 2019 February 01 00:55:03 »
Hi,

I am very sorry to know that PixInsight 1.8.6 is performing so badly for you. However, none of the issues you are reporting can be reproduced under normal working conditions on a variety of Linux distributions and desktops, macOS 10.11, 10.12 and 10.14, and Windows 10. In all of our tests, which include severe stress tests with many gigabytes of data, version 1.8.6.1457 is stable under normal working conditions on all supported platforms.

Unfortunately, If I can't reproduce a nontrivial problem, I am unable to understand and fix it. The kind of issues you are reporting are most likely being caused by hardware-specific conditions, faulty device drivers, wrong operating system components, or conflicts with installed software utilities running as background processes. Windows is intrinsically a problematic platform, mainly because of bad design principles, the vast variety of uncontrolled interactions between software and hardware components, the proliferation of malware and bloatware, and also, unfortunately, because of problematic and even buggy system updates.

Assuming the obvious fact that you want to continue using Windows, my only recommendation in such a situation is a complete, clean reinstall of Windows 10. Please don't install a vendor provided copy of Windows, but an original copy downloaded directly from Microsoft servers. Make sure you install up-to-date device drivers. Do not install any bloatware. Perform a minimal software configuration that you always should keep as pure as possible. This is more or less what we call normal working conditions.

Sorry for not having more constructive advice to offer.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline nytewind1

  • Newcomer
  • Posts: 1
Re: Release 1.8.6 Repeating Crashes
« Reply #18 on: 2019 February 26 19:09:18 »
Palmito, you're not alone on this.  I too have been running into memory issues on 1.8.6 that were not present with 1.8.5.  Same type of error messages that you posted.  I'm using a high-end Windows 10 Pro system also.  Notably the issues occur when I end up using a process like PCC, StarMask or repeated process instances to a preview.  This may not be considered normal operation as I'm trying to perfect the usage of some of these processes and will repeat iterations for over thirty minutes or more.  At some point (and not repeatable) PI will report the crash error.  As there seems to be no apparent fix as this is difficult to pinpoint, my only workaround is to save the PI project often so I can recover after I shutdown the PI project instance that's failing.  This is not a show-stopper for me as I really enjoy the product but just an added nuisance.  Maybe it will work itself out with future updates.

Offline CCDer

  • Newcomer
  • Posts: 7
Re: Release 1.8.6 Repeating Crashes
« Reply #19 on: 2019 March 14 12:11:59 »
I too get repeated crashes on W7 with 01.08.06.1457 after never seeing them on previous releases. My crashes always occur when I go to expand an options box and this can happen immediately after starting PixInsight. I most often see it with Image Integration when I expand Pixel Rejection (1). But if I change Scale Estimater from the default of "Iterative k-sigma..." to "Average absolute deviation..." then it does not happen.  Then if I change it back to "Iterative k-sigma..." it's still okay to open and close the Pixel Rejection(1).  The details are:

 Problem Event Name:   APPCRASH
  Application Name:   PixInsight.exe
  Application Version:   1.8.6.1457
  Application Timestamp:   5c472890
  Fault Module Name:   StackHash_34cf
  Fault Module Version:   6.1.7601.24358
  Fault Module Timestamp:   5c5e1dff
  Exception Code:   c0000374
  Exception Offset:   00000000000bf302
  OS Version:   6.1.7601.2.1.0.256.1
  Locale ID:   1033
  Additional Information 1:   34cf
  Additional Information 2:   34cf7b3675121a09fd72c5e6172c6f07
  Additional Information 3:   9e0c
  Additional Information 4:   9e0c6839d118692b0cb17798b34a9e74


Also the same crash happens with Pixmath when I expand the Destination options. The details are:
 Problem Event Name:   APPCRASH
  Application Name:   PixInsight.exe
  Application Version:   1.8.6.1457
  Application Timestamp:   5c472890
  Fault Module Name:   StackHash_ce1e
  Fault Module Version:   6.1.7601.24358
  Fault Module Timestamp:   5c5e1dff
  Exception Code:   c0000374
  Exception Offset:   00000000000bf302
  OS Version:   6.1.7601.2.1.0.256.1
  Locale ID:   1033
  Additional Information 1:   ce1e
  Additional Information 2:   ce1e0dbc49f315cc4a1a3875da369dd0
  Additional Information 3:   2bb2
  Additional Information 4:   2bb2b08292fcfa513ad09e3b24ef60e3

I'm running with W7 on a i7 6700 CPU with 16GB memory. It's a Shuttle DH110 computer.





Offline CCDer

  • Newcomer
  • Posts: 7
Re: Release 1.8.6 Repeating Crashes
« Reply #20 on: 2019 March 14 12:29:46 »
I will also add that these crashes occur when running PI as the first process immediately after a clean reboot.

Offline CCDer

  • Newcomer
  • Posts: 7
Re: Release 1.8.6 Repeating Crashes
« Reply #21 on: 2019 March 14 12:45:54 »
And in addition, Windows Memory Diagnostic shows no errors and I can successfully run image integrations and drizzle integrations of up to 660 images without issue if I work around these option window expansion issues.

Offline tmatherne

  • Newcomer
  • Posts: 3
Re: Release 1.8.6 Repeating Crashes
« Reply #22 on: 2019 November 22 16:33:53 »
Just wanted to chime in and say I too have issues where the longer PI is open and the more I work in it, it continuously slows down.

This is particularly evident with the flashing process windows as noted above, something I've never had happen until the update. I thought it was some weird one off thing but it has been progressively getting on my nerves having to constantly restart PI or my computer to bring PI back to life. Figured I would google it, and sure enough, others were having this issue.

Weird. I would add more to this topic, but I feel as though it's already been explained in greater detail by others. Just wanted to chime in and say it IS an issue.

Connor

Offline greendaleobs

  • Newcomer
  • Posts: 11
Re: Release 1.8.6 Repeating Crashes
« Reply #23 on: 2019 November 23 20:48:47 »
This cannot be reproduced on any of our working and testing machines.

On Windows, these exceptions can be caused sometimes as a result of severe exhaustion of RAM resources; for example if you are trying to integrate a huge set of images that cannot be loaded incrementally (for example, digital camera raw images) and you don't have enough memory to load them at once.

Another possible cause is a defective OpenGL implementation, but if you have been using version 1.8.5 before this is unlikely.

Anyway, can you please upload a data set where this can be reproduced? If you can use something like Dropbox to upload a project, that would be of great help.

Juan,

with respect ... I would like to help... but simply blaming these issues on windows 10, drivers  or memory are not going to help

both my machines have far more ram than needed and my server in fact as 512GB of ram so its not the issue

both machines are opengl 4.5 and all up to date ... all have the latest drivers for their cards and until 1.8.8 NEVER had an issue with pix

Now you have multiple people reporting the same memory errors across a variety of PC's - the common issue ... 1.8.6 and beyond upgrades

I have also suffered locked and crashed pixinsight during processing apart from image integration in the latest weeks

there is no pattern - its happened during WBPP at differing times/steps .. it leaves no trace in the log but windows throws the memory error

my server can handle anything pix can ever offer it ... dual cpu 48 core 3ghz xeon and 512GB ram so its not related to loading of images into memory and running out ...

this is something that went in with 1.8.8 - where these memory errors started - last crash for me was star aligning in WBPPP - previous crashed at debayer ... other times its at actually writing a file

no pattern - only root cause 1.8.8 - I know its clearly not everyone ... and prob an edge case but theres enough it now makes 1.8.8 unusuable for me so much I have now reverted on machine back to 1.8.5

and its not just on here there are issues - people on the pixinsight for beginners FB also report it the same memory errors

there is a problem - now how can we help find the cause it we can't find the log info - windows does not report the error and it appears in diff ways

just because your machine does not have the issue doesn't mean there isnt one - theres enough of us who do seems to now have issues since 1.8.8 to suggest something a little more than 'bad drivers'

I say that from experience as I am both an Agile Certified Practioner and software development project manager.

So .. how we can help - but i know i paid a lot of money for pix and I'd like to keep using it and helping create a better product - what can we do to help find the error???

thanks

Simon
« Last Edit: 2019 November 23 21:03:13 by greendaleobs »

Offline greendaleobs

  • Newcomer
  • Posts: 11
Re: Release 1.8.6 Repeating Crashes
« Reply #24 on: 2019 November 23 22:56:52 »
Hi,

I am very sorry to know that PixInsight 1.8.6 is performing so badly for you. However, none of the issues you are reporting can be reproduced under normal working conditions on a variety of Linux distributions and desktops, macOS 10.11, 10.12 and 10.14, and Windows 10. In all of our tests, which include severe stress tests with many gigabytes of data, version 1.8.6.1457 is stable under normal working conditions on all supported platforms.

Unfortunately, If I can't reproduce a nontrivial problem, I am unable to understand and fix it. The kind of issues you are reporting are most likely being caused by hardware-specific conditions, faulty device drivers, wrong operating system components, or conflicts with installed software utilities running as background processes. Windows is intrinsically a problematic platform, mainly because of bad design principles, the vast variety of uncontrolled interactions between software and hardware components, the proliferation of malware and bloatware, and also, unfortunately, because of problematic and even buggy system updates.

Assuming the obvious fact that you want to continue using Windows, my only recommendation in such a situation is a complete, clean reinstall of Windows 10. Please don't install a vendor provided copy of Windows, but an original copy downloaded directly from Microsoft servers. Make sure you install up-to-date device drivers. Do not install any bloatware. Perform a minimal software configuration that you always should keep as pure as possible. This is more or less what we call normal working conditions.

Sorry for not having more constructive advice to offer.

today after some extensive testing I am narrowing this down

I dropped back to 1.8.7  and WBPP 1.4.0

it crashed 3 times in a row everytime at  image integration


* Integrating channel 1 of 3:
Integrating pixel rows:     0 ->  2821:  36%
*** PCL Win32 System Exception: At address 00007FFBB01BA839 with exception code C0000005 :
Access violation: invalid memory read operation at address 0000000000000000
<* failed *>


same issue ...

now today I did capture this

Windows successfully diagnosed a low virtual memory condition. The following programs consumed the most virtual memory: PixInsight.exe (16320) consumed 13120978944 bytes, chrome.exe (15684) consumed 825663488 bytes, and chrome.exe (2564) consumed 794980352 bytes.

so ... is it something that is allowing pix to simply eat all the memory to a point that it simply crashed windows?

i will try again now without chrome running and only pix running





Offline greendaleobs

  • Newcomer
  • Posts: 11
Re: Release 1.8.6 Repeating Crashes
« Reply #25 on: 2019 November 24 01:18:53 »
so I can reproduce this again again ......

WBPP - 100 x 300s lights - 50 flats 2sec 50 flat darks 2 sec and 50 darks 300s

runs all processes until it gets to integration

it then crashes out within the first few % .... eats all the memory it can and the PC crashes

same on my server eventually

it causes the whole PC to screw up and requires a reboot

now this is with 1.8.7 and WBPP 1.4.0

later in the week I will install 1.8.5 and do the same


Offline stevek

  • PixInsight Addict
  • ***
  • Posts: 148
Re: Release 1.8.6 Repeating Crashes
« Reply #26 on: 2019 November 24 12:42:34 »
I get very similar issues with the Win32 error and C00000005.  In my case nearly always it is the use of scripts that causes it. Image Integration also can cause it. I also see lots of API errors in the console randomly when I click on execute.  I have done extensive memory and CPU torture tests and the HW is fine.  I also use this machine for Adobe CC apps such as Premiere, Photoshop and After Effects and all those work perfectly.  I run python and Visual C++ IDE and they are all fine.

I never had any issues on this same machine prior to 1.8.6.  Since 1.8.6 I have these issues and I no longer trust PI like I did and have to do a save project every 5 minutes or else risk losing my work.

I am persevering with PI nonetheless since it is so good when it works.  But my confidence is shaken a bit.

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: Release 1.8.6 Repeating Crashes
« Reply #27 on: 2019 November 24 19:57:50 »
in another thread juan said that he found a problem with the javascript interpreter's garbage collection algorithm (this is part of the interpreter's memory management system) which was leading to these sporadic crashes on 1.8.8-1. i don't know if the same bug is in the JS interpreter bundled in 1.8.7 or 1.8.6, but if you are having crashes running scripts on 1.8.8-1 then the update to 1.8.8-2 will probably fix the problem.

rob

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Release 1.8.6 Repeating Crashes
« Reply #28 on: 2019 November 24 23:23:15 »
As Rob has pointed out, we have found a complex bug that has been causing problems during script execution in version 1.8.8-1. I'll release version 1.8.8-2 later today, which fixes these problems. Note that this bug was not present in 1.8.8 and older versions.

However, the problems caused by this bug are clearly identifiable, very specific and, although with some difficulty, reproducible:

https://pixinsight.com/forum/index.php?topic=14283.0

Besides this, there are other Windows-specific issues that we cannot reproduce. Some graphics drivers on Windows are causing problems that we cannot solve because are not under our control. This mainly applies to some Intel HD graphics drivers, which provide poor OpenGL implementations incompatible with recent versions of PixInsight. Outdated versions of NVIDIA drivers can also cause problems on Windows.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Release 1.8.6 Repeating Crashes
« Reply #29 on: 2019 November 25 10:12:48 »
Quote
There are two types of crashes, one were PI just closes and one with the same message as others:
PCL Win32 System Exception
At address 00007FFF63495470 with exception code C0000005:
Access violation: invalid memory read operation at address 0000000000000008

Also after some time it seems all js files fail to run. Typical errors are like:
*** Error [000]: C:/Program Files/PixInsight/include/pjsr/NumericControl.jsh, line 153: Error: At address 00007FFAFD1A5470 with exception code C0000005 :
Access violation: invalid memory read operation at address 0000000000000008

Today we have been able to reproduce this same problem on a Dell laptop with an NVIDIA graphics card running Windows 10 Pro (see attached screenshot). The access violation error has happened during execution of a long WBPP task with 120 CFA light frames.

I am very sorry to say that we currently have no solution to this problem. The access violation does not happen in our code. We have to investigate this problem further, but it seems to be an incompatibility of recent NVIDIA graphics drivers for Windows with the versions of Qt used by PixInsight 1.8.6 and later (Qt 5.12.2, 5.12.5 and 5.13.2). Other than this problem PixInsight runs very well on the same machine. We have repeated the same procedure with no errors, so this issue seems to be sporadic and, to our knowledge, completely unpredictable.

This does not happen on other Windows 10 machines with AMD graphics cards, where the latest version 1.8.8-2 of PixInsight runs without any problems during extremely intensive tests. Of course, this does not happen with the FreeBSD, Linux and macOS versions of PixInsight.

I apologize for this problem, even if we are not directly responsible. I'll try to do my best to find a solution.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/