PixInsight and Large files/NEF files from Nikon D810A

[astroccd1962]

Good afternoon,

Recently I have moved from using a Nikon D80 to a Nikon D810A which has a full frame sensor (7360x4912).  These raw NEF files have gone from about 3-6MB per image to about 73MB. 

I am able to create a Master Dark and Bias Frame with no issues.  (about 20+ files each)

After processing the Light frames using the Dark/Bias frames I get a full set of calibrated images.  Again no issues.

The problem comes about when trying to do Star Alignment.

There are multiple approaches...  Views vs Files.

I first tried to use the File approach and would select about 60 Light Frames (which are now about 426MB each after calibration and registration)

During the loading of these files the computer just shuts down!  I believe it ran out of memory.  I am running on an Alienware i7 with 16GB memory.

*** I have a theory that temporary swap space is being completely eaten up and I will test this theory later by pointing the TEMP/TMP environment variables to a large drive with a lot of free space.

When I try to load all of the 60 Light frames as a view they all show up within the workspace just fine!

Below is the size of each file during the processing:

73MB DARK's  --> 142MB Master DARK
73MB BIAS's   --> 142MB Master BIAS
73MB LIGHTS --> 142MB Calibrated (_c) LIGHTS  --> Debayered (c_d) LIGHTS 426MB -->  Registered (c_d_r)  LIGHTS 426MB


It is only when I start running the Star Alignment (via View) that I see memory issues towards the last 3/4 of the processing and then the computer shuts down again.

Observation indicates that
1. Loading LARGE images into PI via FILE could end up crashing the system but yet loading them via VIEW does not

Questions:
1. Is PI using the system variables called TEMP or TMP ?

2. What is the difference between View vs File process wise? 

3. Is there any known limitations publicly posted for large frame images I could be directed to?

Thank you very much,

Keith

[Juan Conejero]

1. Is PI using the system variables called TEMP or TMP ?

Not in this case. StarAlignment loads disk images completely in memory. Target files are loaded one at a time. The star detection phase creates several single-channel temporary working images of the same dimensions as the target image. Once stars have been detected, the target image and all working data are deallocated.

If your SA task consumes a significant amount of RAM, the operating system may start to use virtual memory allocated on disk as a side effect. However, with 16 GiB of RAM this doesn't seem reasonable in your case.

2. What is the difference between View vs File process wise?

There is no functional difference, other than the fact that a file item requires loading a new image from a disk file, while a view item uses the image that is already stored in memory.

3. Is there any known limitations publicly posted for large frame images I could be directed to?

There are no theoretical limits. The only practical limits are the amount of available RAM and disk swap space, in case your system decides to use virtual memory.

During the loading of these files the computer just shuts down!

Your machine should *never* shut down spontaneously. If this happens, there is a hardware problem with very high probability IMO. This could be a failing memory chip or a thermal problem, among other things.

[pfile]

along those lines, perhaps a stress test like prime95 or memtest86 is in order - if your machine dies while running those workloads then you definitely have some kind of hardware issue.

rob

[astroccd1962]

Thank you both for the feedback.

This machine only has 16GB of memory (Windows version limitation) and does not have any issues processing any other non-full frame data set.

The issue began with using integrated images (Nikon D810A) being 500MB each after calibration and registration...and having over 60 of them (roughly 30GB)

I agree Virtual memory is being used for loading all 60 as a View but I do not believe Virtual memory is being used for a direct disk load.  I say this because before the computer shuts down I do not see a BIG hit on the memory side of the PI process.  This leads me to believe there is an issue with the Star Alignment module when loading from Files.  When I was using Views I got to approximately 35 before out of memory issues showed up

 BTW, using normal DSLR sized images (between 3-6MB) each does not seem to be an issue via File.... just loading a set of 500MB/file does.

I ran Prime95 with no issues and no errors were detected.

I wish I could load one of these files in for someone to try it themselves.

Thanks a bunch,

- Keith

[pfile]

prime95 is going to tax the CPU more than the memory, and anyway you need to run the "torture test" for several hours at least... running it once doesn't count

memtest86+ will test the heck out of the DRAM and again it could take many hours to make one pass thru the memory. for best results you should boot from a USB key or CDROM with memtest86+ on there so that most of the memory is available for test. if you run it while windows is booted lots of memory will escape test.

anyway i suppose this could also be a disk issue, since you're really beating the VM system up. not sure how to test that.

rob

[astroccd1962]

There is a Dell Support Assistant Utility that will test all of the components thoroughly too.

As soon as I can shut everything down for the day I will try it.

I also noticed that the pagesys file was only 16GB... that is not enough I am sure.... so I am moving content off and will defrag the drive then enlarge the pagesys file.

Actually hoping its a hardware issue... that I can fix myself !!! 

Thank you everyone for the responses!

[astroccd1962]

After finding the app that examines the event history I discovered that an app called Dell Data Vault was to blame.  Research showed that recent upgrades to this app caused surges in the temperature within the CPU which caused the computer to shutdown.  With this app disabled I was able to at least stack the registered images.

I still have to process the image but with just a little tweaking using the Histogram Transformation and STF this is what I have so far.

Now I need to figure out how to safe the image with all of the modifications made!   Practice and Persistence will make perfect.

Thank you everyone!

- Keith

[pfile]

cripes - thermal shutdown... that's awful. then again now that you mention it i have heard that this can be a problem on windows laptops, especially if the cooling system has a bunch of dust in it...

rob

[jerryyyyy]

I use a Dell M3800 PrecisionLaptop and no question that PI pushes the temps.... fans are always off until I run something intense on PI.... but never had a thermal shut down.

BTW, I have a Nikon D800 and would be interested in your workflow with the D810A.