Very SLOW Integration of Nikon NEF/RAW Bias and Dark Field Images (Mac OS X)

[james7]

I've been working on a few new sessions that I captured over the last week or two and I've noticed that the integration of my Bias and Dark Fields runs VERY slowly during the calibration phase of my batch processing. This seems to happen once I have more than a few Bias/DFs (more than four?) and the process seems to start at normal speed but then once the integration reaches about seven percent completion on the master Bias/DFs it slows dramatically. In fact, the jobs never complete because from that point on the processing is so slow that it might take an hour (or more) for even an additional one percent to get done.

This happens under both PixInsight v1.7 and v1.8 (Ripley) and I've tried running on both Mac OS X 10.6.8 and 10.7.5 (with different installations of PixInsight on each OS). Interestingly, this only happens on the Bias and DFs and I think it might have something to do with low signal levels OR the standard integration settings for Bias/DFs. My light fields integrate just fine (but without calibration) and I've been able to duplicate this problem even when not using the batch processing script (i.e. it also happens when using the standard imageintegration process, as long as you select the typical/recommended normalization and weight settings for Bias/DFs).

I've used Mac OS X's Activity Monitor to sample PixInsight when it is in this slowdown and it appears that PixInsight is in some kind of deadlock condition where it is just waiting on a series of semaphores. It's probably a "race" condition as some work gets done but it's done VERY slowly. The OS has plenty of free memory and PixInsight is spinning at 100% on one core of my four core Mac Pro system. This slowdown can happen with as few as sixteen files and my system has 10GB DRAM with PixInsight being the only active user process.

The "kicker" is that this seems to only happen with Nikon NEF/RAW files, since I've been successfully using a Sony APS-C camera for many months now without similar issues. Also, these very same jobs/files run fine under MS Windows 7, so it seems to be a Mac OS X related problem.

As yet I haven't done enough testing to determine whether this issue is related to the integration settings for Bias/DFs OR the differences between the signal levels in Bias/DF and LFs. It may be a combination of both, that and the Nikon NEF/RAW files themselves since I'm pretty sure that my Sony/RAW files are still working (I hope, since I've been working to try and finish my latest captures which were done exclusively with my Nikon DSLR).

Tonight I'm going to try a fairly long job with just my Sony RAW files (Bias/DF/LF) and if that works then it will point directly to some issue with the Nikon Bias and DFs. My Nikon is a D5100 and my Sony is an NEX-5N (they both have about the same sensor resolution, in fact I think they use the same sensor -- made by Sony).

If I can't resolve this issue within the next few days (by myself) then I may need to send you some copies of my Nikon Bias files so that you can try to duplicate this problem yourself. It doesn't appear to be a Ripley issue, since I went back to v1.7 and the same problem happens there.

[georg.viehoever]

Can you check what happens if you first convert your images to FITS (with the BatchConversionScript), and then integrate them. With my Canon images, I had the impression this is *much* faster than integrating the .CR2 images directly.
Georg

[edd]

I use a Nikon D3100 myself and haven't noticed this when running in OS X. What are you doing in terms of debayering the NEFs, if anything?

I expect to be taking data tonight with it again - I'll pay particular attention to the making of the master bias and see if I notice anything amiss.

[james7]

Okay, the Sony RAW files (ARW) worked fine, so it seems to be limited to the Nikon NEF files. Also, I'm pretty sure I tried pre-converting the Nikon NEF files to FITS RAW and that didn't change the behavior (but I'll check on that once more). I also need to reconfirm that this happens on my installation of PixInsight v1.7, but I really don't know whether an installation of v1.8 Ripley precludes going back to v1.7 (under Mac OS X can you have interactions between two, concurrent installations?). In any case, I have a "clean" install of Ripley on my Mac OS X 10.7.5 system and that shows exactly the same problem.

I also want to backtrack on one of my previous statements (that SOME work was getting done even after the integration enters this slowdown phase). It's possible that may happen under some trials for a short time, but it's just as likely that the system is entering a complete deadlock shortly after it reaches a small percentage of completion. So, it may not be a simple race condition, it could be a symptom of a memory or stack corruption where the process is just jumping somewhere it shouldn't be (and thus causing a deadlock rather than an outright crash).

Finally, after looking at some of the other "bug" reports on Ripley I'm beginning to wonder whether I'm just seeing a different symptom of a more common problem that is affecting other users and installations. In particular, it MAY be related to this report by user papaf: "Crash during master bias creation in BatchPreprocessing." Perhaps this is something that is being triggered by the way the code is being loaded (location) on each individual configuration. If so, and I can reproduce this under v1.7, then it may be a problem that was introduced in v1.7 (or earlier) and that has just sat dormant waiting for the correct set of conditions (that may now be more prevalent under Ripley). In order to confirm the latter, however, I need to know whether it is safe to have installations of both v1.7 and v1.8 on the same Mac OS X boot drive (stored in different locations and run at different times).

[papaf]

Try setting normal sigma instead of windsorized and see if it helps.
In fact, even in my case, the process is slowed down enormously before crashing out, so it may be related.

[james7]

papaf, I think we may be seeing the same problem. If I integrate my bias files with any other rejection method (i.e. other than Winsorized Sigma Clipping) then the process runs quickly and completes in just a few minutes. Furthermore, if you use a small set of bias files then the integration is slow but it eventually finishes even in the batch script. By small I mean eight files or less and with somewhere between eight and sixteen files the integration times for the bias and dark fields become so long that you either have to wait for hours or abort the process.

However, this only happens on bias and dark fields, as I can integrate large numbers of light fields when using Winsorized Sigma Clipping even when using the same integration parameters that are used for processing bias and dark fields.

So, the problem seems to be exactly as I described in my original post. It depends upon the number of files you have for your bias and dark field calibration and it depends upon the signal levels within those files (i.e. if you have small sets of files then you may not notice the slowdown and I suspect that at some signal level the problem disappears). The reason I may be seeing this problem right now is that I was working with some files that probably had very low signal/noise levels. Low ISO settings (ISO 400 to 800) and short exposures (1 to 2 seconds). I was using my Nikon D5100 with a 50mm prime lens to capture untracked/unguided, widefield images of the constellations.

Lastly, it should be noted (again) that this problem doesn't seem to happen under MS Windows, as I've been able to process these very same files with PixInsight v1.8 on my Windows 7 system. Of course, I don't know it the files are being processed as quickly as they should/can be, maybe the slowdown under windows happens at a different level or at a different number of files.

For the time being I may just change the batch preprocessing script to use a different rejection method. I wonder if Linear Fit Clipping would be a good choice, since I generally have 16 or more samples for my bias, dark fields, and light fields.

[edd]

Well I just integrated 20 bias NEFs from my D3100 in 1.8RC7 on OS X in just under 70s, 128M buffer size and 8000M file cache, using ImageIntegration rather than the batch script, and the settings from http://pixinsight.com/tutorials/master-frames/en.html.

[pfile]

no problem having multiple version of PI installed on a mac, that works fine.

[james7]

Well I just integrated 20 bias NEFs from my D3100 in 1.8RC7 on OS X in just under 70s, 128M buffer size and 8000M file cache, using ImageIntegration rather than the batch script, and the settings from http://pixinsight.com/tutorials/master-frames/en.html.

Thanks for checking. What ISO setting were you using? I suspect that if you were using a high setting you may not see the problem, or the slowdown might not happen after the same number of files. In any case, the D3100 has a slightly smaller sensor and slightly higher noise levels than the D5100, so that could also change things. Then again, temperature could play a role, I was shooting at somewhere around the mid-50s F (not that cool, but not that hot either, given that summer is approaching in the northern hemisphere). It would be interesting if this turns out to be an ISO/noise related issue. I wonder whether the Winsorized Sigma Clipping uses any recursive algorithms. By the way, I ran the same file sets thought the batch processing script on my Mac OS X system using Linear Fit Clipping and that worked fine. Regular Sigma Clipping also seems to work, but I haven't use the latter to complete a run in the batch processing script.

I think I'm going to do a quick test at ISO 3200 to see if the problem still happens. It may also be a random thing, depending upon your particular configuration (memory, processor speed, hard drive speed, OS version, etc.). And lastly (of course) it could just be something that is specific to my situation (software conflict, corrupted installation). Although in the latter cases I'd be surprised since this problem happens on two different boot drives (one running 10.6.7 and the other a pretty clean install of 10.7.5, each running a separate install of PixInsight).

[edd]

Well I just integrated 20 bias NEFs from my D3100 in 1.8RC7 on OS X in just under 70s, 128M buffer size and 8000M file cache, using ImageIntegration rather than the batch script, and the settings from http://pixinsight.com/tutorials/master-frames/en.html.

Thanks for checking. What ISO setting were you using?

800.

In any case, the D3100 has a slightly smaller sensor and slightly higher noise levels than the D5100, so that could also change things.

That's certainly possible. The biases I integrated were shot at 1/4000S and at probably around the same temperature as you, but the different sensor seems like the most likely candidate right now to me.

[Juan Conejero]

Hi,

I am quite sure that this problem is a variation of a similar issue reported by Fabio in another thread. This problem is now fixed and the fix will be available through updates later today. Please read the information that I have posted in the other thread.

Hopefully all of these issues will get fixed at the same time. However, the presence of many zeros in your calibration frames (assuming that this is also the origin of this bug report) is a different problem that you should detect and fix during acquisition, IMO.

[james7]

Juan, thanks for looking at this issue. However, in my case (Nikon NEF Bias/DF) it may be worth noting that these very same files integrated using Winsorized Sigma Clipping without problems when running the Windows version of PixInsight. It was only when I ran under Mac OS X that these problems happened.

[james7]

This (original) issue still exists under Mac OS X and the latest and fully updated version of Ripley (and the legacy version of v1.7). Basically, when using low-noise NEF files from my Nikon D5100 (low ISO and/or very short exposures) you can't use Winsorized Sigma Clipping with more than a handful of images or the integration slows to a virtual crawl (with more than about 16 files it will "hang" and never complete). Happens only under Mac OS X, same files work under MS Windows.

The workaround is to select a different rejection method during your integration (when creating masters for Bias and DF and for the final integration of the light frames).

[Juan Conejero]

Hi James,

If you are using the latest version of PixInsight with all updates applied, then this is a very strange bug. I need a minimal set of files that reproduce this problem.