PixInsight Forum (historical)
Software Development => PCL and PJSR Development => Topic started by: pfile on 2011 October 14 16:55:56
-
looks like ImageWindow.open(); does not understand format hints, is this correct?
it would be nice to fix up BatchFormatConversion so that it can take a format hint. when preparing master darks/flats etc. i need to first convert CR2 to .fits, and unless i can hack the script to add a format hint, i have to remember to go into the format explorer and change up the DSLR_RAW configuration. usually i forget to do this...
-
Hi Rob,
Format hints are fully supported by the FileFormat and FileFormatInstance JavaScript objects, which were introduced in PixInsight 1.7.0. I decided not to comment on these new objects until now because the implementation is still slightly incomplete, but I'll do briefly now.
These objects are the JavaScript counterparts of the pcl::FileFormat (http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1FileFormat.html) and pcl::FileFormatInstance (http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1FileFormatInstance.html) C++ classes. On the PCL reference documentation links given you'll find complete information.
FileFormat gives you access to an installed file format such as FITS, TIFF, JPEG, etc. You can create a FileFormat object with the constructor:
new FileFormat( String nameExtOrMime[, Boolean toRead=false[, Boolean toWrite=false]] )
For example, this line gives you access to the DSLR_RAW module:
var F = new FileFormat( ".cr2", true/*toRead*/ );
Once you have access to an installed file format you can instantiate it with FileFormatInstance. A file format instance is an abstraction that allows you to work with existing or newly created image files of a particular format. The constructor is:
new FileFormatInstance( FileFormat format )
and the relevant functions to open and create/write a file:
Array FileFormatInstance.open( String filePath[, String hints] )
Boolean FileFormatInstance.readImage( Image )
Boolean FileFormatInstance.create( String filePath[, String hints[, int numberOfImages=1]] )
Boolean FileFormatInstance.createImage( ImageDescription d )
Boolean FileFormatInstance.writeImage( Image )
Boolean FileFormatInstance.close()
For example, you can open a Canon CR2 file in this way:
var f = new FileFormatInstance( F );
var a = f.open( "/path/to/foo.cr2" );
FileFormatInstance.open() returns an array of ImageDescription objects. Each object in the array describes an image stored in the file. Some formats (FITS) support multiple images stored in a single file but this is not the case of DSLR_RAW, so the returned array in this case will have only an element. This function supports file format hints. For example:
var f = new FileFormatInstance( F );
var a = f.open( "/path/to/foo.cr2", "raw" );
if ( a.length == 0 )
throw new Error( "File access error" );
would open the foo.cr2 file as a Bayer RGB image without interpolation and no black point correction, irrespective of the current DSLR_RAW settings. Once you have opened the file you can read an image with code such as:
Image img;
if ( !f.readImage( img ) )
throw new Error( "File read error" );
f.close();
File creation and storage is also rather simple. Following the same example, suppose you want to write the CR2 image as a floating point FITS file with top-to-bottom/left-to-right orientation, irrespective of the current FITS settings:
var F = new FileFormat( ".fit", false/*toRead*/, true/*toWrite*/ );
var f = new FileFormatInstance( F );
if ( !f.create( "/path/to/bar.fit", "up-bottom" ) )
throw new Error( "File creation error" );
var d = new ImageDescription;
d.bitsPerSample = 32;
d.ieeefpSampleFormat = true;
if ( !f.createImage( d ) )
throw new Error( "Image creation error" );
if ( !f.writeImage( img ) )
throw new Error( "File write error" );
f.close();
Let me know if this helps.
-
wow, thanks for that explanation. i'll try and hack this into BatchFormatConversion.js.
-
started hacking on this; got the gui fixed and was working on this new method of opening/reading/writing files and... i started wondering about the loop over the variable w when writing the file and finally realized that's there to handle the case of fits or other input formats that could have multiple images in a single file. unfortunately it seems that it's currently broken since it determines an output filename and then goes ahead and writes over that file multiple times when the output format only handles one image per file, like .jpg.
i guess i'll try to fix that. easiest way would be to evaluate the filename each time thru that loop, but maybe i can get a handle on the fits image identifiers to make nicer filenames. what other file formats support multiple images per file?
-
For now the only format supporting multiple images per file is FITS. This will change when we implement PixInsight's native format. It will be an XML-based format supporting... well, everything :)
-
okay, just checking in:
1) seems like maybe .jpg and .tif are not implemented yet - i get errors if i try to convert to either format:
*** Error: TIFFFileFormatImplementation: CreateImage() must be reimplemented in descendant class
2) it looks like changing the sample format/type is not working right. thru printfs i can see that i'm fixing up the output ImageDescription properly, but the changes don't seem to take effect. could it be the case that f.writeImage ( img ) is simply using all the attributes of the input image when writing the file without regards to the call to createImage()? or maybe d in your example below is incomplete and is ignored by f.writeImage? do i have do to "d = a" before fixing up the fields of d?
even if i fix this all up, the lack of tiff and jpg support seems like it can not replace the existing BatchFormatConversion script.
other than those two problems i'm successfully able to convert CR2 to fits and the script obeys the input hint. i still need to work on output hints and multi-image FITs files as the input file, though. what single-image file format targets are currently supported by FileFormatInstance.createImage()?
-
bueller? bueller?
-
OK, here I am! :)
1) seems like maybe .jpg and .tif are not implemented yet - i get errors if i try to convert to either format:
All file formats are fully implemented -- otherwise you would be unable to write a TIFF or a JPEG file...
What happens is that I made a mistake in the code I put in my previous post: the call to createImage() is not necessary unless you want to write a file incrementally (row by row, or by successive row strips). See the corresponding PCL documentation entry (http://pixinsight.com/developer/pcl/doc/html/classpcl_1_1FileFormatImplementation.html#ae70f51ff3c9c548457f046c1aa7b9b7d).
Since only FITS supports incremental file writing, only the FITS format implements the createImage() function. So please try the following:
var F = new FileFormat( ".fit", false/*toRead*/, true/*toWrite*/ );
var f = new FileFormatInstance( F );
if ( !f.create( "/path/to/bar.fit", "up-bottom" ) )
throw new Error( "File creation error" );
var d = new ImageDescription;
d.bitsPerSample = 32;
d.ieeefpSampleFormat = true;
if ( !f.setOptions( d ) || !f.writeImage( img ) )
throw new Error( "File write error" );
f.close();
using the format of your choice (TIFF, JPEG, etc) and the corresponding hints, if necessary.
Let me know if it works this way, and sorry for the inconvenience (I hope there are no mistakes now!)
-
thanks. i didn't know that documentation existed, or rather, i did not know what stuff here was new and undocumented vs. what is older.
any ideas about the problem with sample depth and type?
edit - sorry, missed the new call to setOptions()
-
if the script takes the code path where f.setOptions( d ) needs to be called, invariably i get:
*** Error [000]: /Applications/Pixinsight64.app/Contents/src/scripts/BFC.js, line 243: Error: FITS: Invalid image identifier
where line 243 is the call to f.setOptions. FITS, TIFF, JPEG, does not matter.
strangely no matter what i do with a newly created ImageDescription (set some fields, don't set some fields, assign the input file description to it), FileFormatInstance::setOptions() just does not like it.
for fun i passed in the input file ImageDescription to setOptions() and although it parses and runs, the output file sample depth always seems to be uint16 even if the input file is a 32-bit ieee float fits, for instance.
strangely, the DSLR_RAW module seems to take "raw" properly as a input format hint, but other hints (tried "bilinear" and "vng") don't seem to do anything... in those cases, DSLR_RAW is defaulting to what's configured in it's settings panel.
i still have to hack in a couple of things and when i'm done with that i'll just post the script here and you can see what's going wrong.
-
Rob,
I need to see your code to know what happens. Maybe you've discovered a bug in the PJSR bridge to file formats (not a big surprise since this functionality is new in version 1.7).
-
well, i didn't get around to fixing the following:
1) output format hints
2) moving the code to generate unique filenames into the output loop, to deal with multi-image fits input files
3) trying to get a handle on the fits image names in a multi-image file when computing output filenames
there are currently 2 problems with this script:
1) as i mentioned earlier, changing the bit depth and sample format does not work
2) the script has some kind of memory leak - converting 80 CR2 files to raw fits resulted in a 13GB VM image for PI when the script was done. when the script exits, the memory is freed. obviously this script is a non-starter for anyone using a 32-bit version of PI.
on #2, i thought maybe i needed explicit destructors for everything that i had new'd. but adding them did not change anything. i'm not sure if the original script bloats up like this as well. i've never written or hacked on a javascript program in my life, so it's probably the case that i am doing something stupid here.
juan if you can look at problems #1 and perhaps #2, that would be great.
-
Try using Garage Collection (some Call like gc(), i don't have Pi Access right now) After Each iteration. In my scripts that released Most of the Memory.
Georg :)
-
i'll give it a try but will all the leaked memory end up in my garage? because it is already full of crap >:D
these languages with "automatic" memory management never seem to work right. malloc() and free() or die! 8)
seriously though the call to gc() is probably going to slow the script down even more. here's hoping i just don't know how to program in javascript!
-
gc(); it is and in fact it worked great. i put calls to gc() immediately after i had called the destructors for all the stuff allocated during each loop.
and my garage seems to be no more or less full of junk as well, so that's good.
thanks for the tip.
-
bump....
i know this is several sigmas less interesting than new multiscale modules and stuff but it sure would be useful to get it sorted out for us DSLR users...
problem #2 was solved with gc(), but problem #1 still stands.
-
Downloaded and working on it right now ;)
-
thanks, sorry to be such a pest
-
Here is BatchFormatConversion 1.2.1. It works fine except the sample format parameter, which is ignored. In this version each output file is written with the sample format of its corresponding input image. This is due to a bug in the PI Core application (see the comment at line 225 of the script). I have identified this bug and will fix it in the next version. Despite this bug the script is perfectly usable for DSLR_RAW to FITS conversion, since for this task we don't have to change the original 16-bit integer format.
For completeness, I have added also an output hints field. Let me know if this version of the script works as expected.
-
thanks, when I get home I'll read the source and try it out. I still wanted to fix the output filename uniquification when the input file is a multi-image fits. or maybe you fixed that too, I'll have to read the script.
-
I simply disabled multiple image files completely (the script throws an exception if it finds one). They are rare and not supporting them is not a practical problem. Feel free to add support for them if you want.
-
yeah i managed to view the file with my iphone and could see that it does not support multi-fits. i think i had done that too in the version i posted, can't remember. anyway i'll try it out on a big batch of CR2s and see what happens.
-
okay, i tried it out and it seems to be working great, but i didn't push it that hard. looking forward to the bug fix and final release. thanks.