WBPP 2.1, file path length issue (>260), despite LongPathsEnabled?

Linwood

Well-known member
I have a new problem and may understand it, or may not.

I'm getting these errors:

[2021-08-02 18:54:35] * Writing output file: T:/Stage/M31_540mm_AP1100_2021-08-01/calibrated/Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10/FLAT_FlatsFullIPower540mm1x1_2021-08-02_04-48-06_Ha-5nm_1.00s_SET-TEMP_-10.0_Gain_100_Offset_50_Bin_1x1_NP101is_FP8833_FT28.00_Temp_-5.60C_0000_c.xisf
[2021-08-02 18:54:35] Writing image: w=9576 h=6388 n=1 Gray Float32
[2021-08-02 18:54:35] 62 FITS keyword(s) embedded.
[2021-08-02 18:54:35] 28 image properties embedded.
[2021-08-02 18:54:35] *** Error: File I/O Error: Unable to create file: Win32 error (3): The system cannot find the path specified.
[2021-08-02 18:54:35]
[2021-08-02 18:54:35] : T:/Stage/M31_540mm_AP1100_2021-08-01/calibrated/Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10/FLAT_FlatsFullIPower540mm1x1_2021-08-02_04-48-06_Ha-5nm_1.00s_SET-TEMP_-10.0_Gain_100_Offset_50_Bin_1x1_NP101is_FP8833_FT28.00_Temp_-5.60C_0000_c.xisf

So I think I figured out it is related to an old Windows 10 file path limit of 260 characters, but that's gone:

https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation?tabs=cmd

Initially it was set to 0 (disabled) but now I definitely have long names enabled. I confirmed it by manually creating that long name

Code:
t:\>type nul > T:/Stage/M31_540mm_AP1100_2021-08-01/calibrated/Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10/FLAT_FlatsFullIPower540mm1x1_2021-08-02_04-48-06_Ha-5nm_1.00s_SET-TEMP_-10.0_Gain_100_Offset_50_Bin_1x1_NP101is_FP8833_FT28.00_Temp_-5.60C_0000_c.xisf

t:\>cd \Stage\M31_540mm_AP1100_2021-08-01\calibrated\Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10

t:\Stage\M31_540mm_AP1100_2021-08-01\calibrated\Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10>dir
 Volume in drive T is VDisk
 Volume Serial Number is 022F-86F4

 Directory of t:\Stage\M31_540mm_AP1100_2021-08-01\calibrated\Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10

08/02/2021  03:02 PM    <DIR>          .
08/02/2021  03:02 PM    <DIR>          ..
08/02/2021  03:02 PM                 0 FLAT_FlatsFullIPower540mm1x1_2021-08-02_04-48-06_Ha-5nm_1.00s_SET-TEMP_-10.0_Gain_100_Offset_50_Bin_1x1_NP101is_FP8833_FT28.00_Temp_-5.60C_0000_c.xisf
               1 File(s)              0 bytes
               2 Dir(s)  1,992,697,749,504 bytes free

t:\Stage\M31_540mm_AP1100_2021-08-01\calibrated\Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10>

But it appears that Pixinsight can't do it. And yes I rebooted first.

I see Juan commented on this here:

https://pixinsight.com/forum/index....-working-directory-problem.16947/#post-102750

Who indicated you could change the limitation. I did, but it doesn't seem to work.

Is it somehow baked into WBPP?

I can try to work on file names to reduce the length, but should this fail in this way?

Linwood

PS. Showing it is set:

Code:
t:\Stage\M31_540mm_AP1100_2021-08-01\calibrated\Flat_BIN-1_FILTER-Ha-5nm_Mono_GAIN-100_OFFSET-50_SET-TEMP--10>reg query HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
    DisableDeleteNotification    REG_DWORD    0x0
    FilterSupportedFeaturesMode    REG_DWORD    0x1
    LongPathsEnabled    REG_DWORD    0x1
    NtfsAllowExtendedCharacter8dot3Rename    REG_DWORD    0x0
 
Should not --10 instead be _-10?

Perhaps but that is WBPP's doing not mine. The original file is using dash only as a negative (or in the dates) and using underscore as a separator.

Also, after I dropped unused fields from the file (like names and focus position and temperature) to shorten the names, then WBPP worked fine.

I just don't understand how windows can honor the registry setting to allow long file names and WBPP does not, unless it is somehow baked in. Yet... the error message looks like it is arising from windows.

Does anyone else's paths generate more than 260 characters and work (in Windows 10)?

Also, I think I understand why this is new behavior -- if my memory serves, there was not a sub-folder set created under calibrated (and worse deeper under cosmetized)? Weren't they all dumped into one, not separated by filter (etc)? This increased the path length and is likely why my file naming convention which worked before, fails now?

But... it should still work. Any ideas? Anyone from PI know?
 
So ... no one else has this problem? I just ran WPBB and had a LOT of files just disappear, because in the middle of the huge log file I get errors like this:

Code:
[2021-09-21 00:06:15] Writing output file: T:/Stage/M31_AP/M31_08-01/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C/LIGHT_M31_2021-08-02_00-00-33_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_NP101is_FP8925_FT29.30_Temp_-6.80C_0008_c_cc.xisf
[2021-09-21 00:06:15] Create file: T:/Stage/M31_AP/M31_08-01/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C/LIGHT_M31_2021-08-02_00-00-33_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_NP101is_FP8925_FT29.30_Temp_-6.80C_0008_c_cc.xisf
[2021-09-21 00:06:15] Writing image: w=9576 h=6388 n=1 Gray Float32
[2021-09-21 00:06:15] 73 FITS keyword(s) embedded.
[2021-09-21 00:06:15] 28 image properties embedded.
[2021-09-21 00:06:15] *** Error: File I/O Error: Unable to create file: Win32 error (3): The system cannot find the path specified.

I keep trying to shorten file names, but a part of the issue is WBPP is creating folders with very long names, so that the total file name is too long. For example, one subfolder name is:

Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C

I'm happy to sue the file name with the "_c" and "_cc" attached.

but... I HAVE LONG FILE NAMES ENABLED.

Here is the exact same file name created from the command line. So this seems specific to Pixinsight in some fashion.

Code:
C:\Users\ferguson>copy nul: "T:/Stage/M31_AP/M31_08-01/cosmetized/Light_BIN-1_EXPOSURE-120.00s_FILTER-Luminance_Mono_GAIN-0_OFFSET-50_SET-TEMP--10.00C/LIGHT_M31_2021-08-02_00-00-33_Luminance_120.00s_SET-TEMP_-10.00C_Gain_0_Offset_50_Bin_1x1_NP101is_FP8925_FT29.30_Temp_-6.80C_0008_c_cc.xisf"
        1 file(s) copied.

Or at least Pixinsight uses some service(s) that are limited in file name length.

I would also argue that WPBB needs to say something like "Errors were encountered" or some such at the end of the run, it's too hard to just notice even if you take time to look at the logs.

Should I post this in the bug area?

and/or if I'm the only one -- any idea why?
 
Note: according to MS Windows 10 documentation, to use long file names it is neccessary that:
  • The registry key Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem\LongPathsEnabled (Type: REG_DWORD) must exist and be set to 1.
  • The application manifest (for the application) must also include the longPathAware element.
The PixInsight application manifest is embedded in pixinsight.exe:
Code:
<assembly xmlns='urn:schemas-microsoft-com:asm.v1' manifestVersion='1.0'>
  <trustInfo xmlns="urn:schemas-microsoft-com:asm.v3">
    <security>
      <requestedPrivileges>
        <requestedExecutionLevel level='asInvoker' uiAccess='false' />
      </requestedPrivileges>
    </security>
  </trustInfo>
</assembly>
It does not include longPathAware, so PI will not use long filenames in Windows 10.
Clearly, the PI developers will not add this support unless it has been thoroughly tested (e.g. to ensure all file system calls can accept suitably sized string arguments). This testing may not be a high priority (e.g. it only applies to some versions of just one of the supported operating systems).
 
Last edited:
Ah, shoot... I didn't see that aspect of .net.

So with PI supporting linux (and presumably much longer file paths), does this mean it's likely an oversight in PI they haven't enabled it?

The manifest is to (I presume) permit programs without adequate storage internally to disallow usage, but that seems an unlikely case for a multi-OS program like this?

Anyone from the PI team that might comment?
 
The manifest is to (I presume) permit programs without adequate storage internally to disallow usage
I think it is more a matter of ensuring that the huge body of legacy Windows programs aren't suddenly confronted with undesigned, untested long file names; applications have to opt in to the new capability.
 
Well yes, a different way of saying it.

But what I'm saying is that PI is not one of those, it's a multi-platform program that, it would seem, already supports long file paths?

if they enable it in the manifest it only comes alive if one has already enabled it in windows, so it's a no-change for people who have not or are not aware of it.

Unless you think PI can't support long file paths?

My complaint is I have less control over this (while maintaining file names with specific attributes for grouping), since it is WBPP that is creating a great deal of the length of the name by using all these long sub folders,
 
Actually, this is a bug because the manifest should include the longPathAware key. I will fix this as soon as possible, sorry for the inconvenience.

By the way, it seems that this limitation is also present on Windows 11... ?
 
Hi,
I often see the same problem with the very long subfolder names generated by WBPP.
As a work around for my DLSR images without binning, filters etc I modified some definitions in the
WeightedBatchPreprocessing-engine.js to reduce the subfolder names e.g. on line 1075
let filter = "" ; //"FILTER-" + ( isEmptyString( this.filter ) ? "NoFilter" : cleanFilterName( this.filter ) );
as you see I just put an empty string for the Filtername etc.
With this modifications I never had problems with long file/path names in Windows.
I know this will be overwritten at the next update - but as I have to change only 3 lines it is acceptable for me.

It might be a philosophical question having all informations as binning, filter, CFA, RGB, exposure etc in the filenames - as I do,
or distributed over the different subfolder names as others do.

Unfortunately I am not familiar with Java ( I am a fosile FORTRAN ) a little tikmark for short subfolder names in the global options
of the WBPP script would be helpfull for all of us having similar filename problems.

Regards
Klaus
 
It might be a philosophical question having all informations as binning, filter, CFA, RGB, exposure etc in the filenames - as I do,
or distributed over the different subfolder names as others do
Whether it is in the file name or the folder names it all ends up in the full file path - which is what is restricted.
 
That is totaly correct - but not exactly what I wanted to say.
Having long subfolder/file names in the source directories is one thing, but I am talking about the subfolder names generated by the
WBPP script in the working directories as calibrated, Flat_...., Light_.... or debayered, Light_.... or registered, Light_.....
As I use always a separate project folder for processing, long subfolder names on the image source folders does not matter but often in the
project folder.
A simple Flat, Light for the subfolder names would do it without _BIN_Exposure_Filter_CFA... etc.
The full file path problem occures only on that part of the processing, writing the preprocessed files into the calibrated, registered, debayered
directories with the long Flat- and Light- subfolders as "Light_BIN-1_EXPOSURE-300.00s_FILTER-NoFilter_CFA" which can easily reduced
to "Light" saving 43 characters in my example.

With image file names as: M12_300sec_ISO400_001_2021-05-31_224010_amb_13,0C_fpos_20285_sop-west_c_d.xisf
it is easy to come to the file path restriction with long subfolder names in the project folders repeating informations already in the file names.

Klaus
 
Back
Top