New File Format: XISF - Extensible Image Serialization Format

[Juan Conejero]

This has been a small bug in the CosmeticCorrection tool. I have just released two updates that fix this problem on all platforms.

Sorry for the inconvenience.

[Juan Conejero]

Although I haven't yet tried it myself (and will resist as long as possible after reading here), clearly there are compatibility problems in spite of all assurances that that would not be the case.

there should probably be a beta program for changes that affect the whole platform such as this one.

The XISF module is a production stable release. It has been tested for stability and compatibility with the platform. The problem that has arisen here with CosmeticCorrection is of marginal importance and has nothing to do with the new format. It is just a regression caused by some recent changes in the BatchPreprocessing script, mixed with a small mistake I made when I packed the latest update.

It has taken less than 13 hours to get this problem fixed with the corresponding updates, which I think is not so terrible, especially taking into account the time zone differences. And that's because I have taken the liberty to sleep more than usual today, as I was tired after the effort put in the latest releases.

Bugs happen! They happen everywhere. Frankly, I think that the assertions you have made here are unfair with the work we are carrying out day to day to deliver high quality, innovative software to our users.

As everybody knows we are absolutely perfect. Obviously we have to simulate bugs from time to time, just to look normal, so bear with us...

[Juan Conejero]

Jay,

Just as I feared would happen after downloading and installing the updates relating to the new XISF file format and related batch pre-preprocessing script, I now have a system that will not process my .fits images. Thank you very much for this gratuitous solution to a problem that I at least did not know that I had. As I said, for me it is a "solution" in search of a problem; and all you have done for me is create a previously non-existent problem, which I will now describe.

My answer to this paragraph is the two updates that I have released a while ago. Let me know if the updates fix your problem, and if they don't, we'll try to do our best to fix them as quickly as possible.

Please accept my apologies for having caused you so much trouble. It was not my intention.

[dayers]

All,

I work with PI extensively virtually every day. I have several projects going, and the format change for me so far has been seamless. I think there have been some overreactions in this thread that I think are unfair to Juan and his team. Not everyone has had problems with the change from FITS to XISF format.

Dave

[pfile]

there should probably be a beta program for changes that affect the whole platform such as this one.

Bugs happen! They happen everywhere. Frankly, I think that the assertions you have made here are unfair with the work we are carrying out day to day to deliver high quality, innovative software to our users.

exactly. bugs happen. which is why our design team has a > 1:1 ratio of verification engineers to rtl designers. and we have a giant QA department. and we run nightly hardware and software regressions. and on and on and on. obviously the cost of a bug in silicon is orders of magnitude more costly than a software bug, but when you have a big system, even the software needs to be hammered on for quite a long time before it is ready for release.

since you don't have those resources, a logical alternative is to "hide" big releases from the bulk of your users. you will always have a subset of advanced users that want to try new things, and they can help you find bugs before those bugs hit the mainstream users who (as evidenced by this thread) don't have the patience or desire to deal with these bugs.

no one is saying you don't write good software, or you don't support your software. i'm just saying that the blow-up in this thread could have been avoided by letting those users who are willing to take risks do that "behind the scenes" to flush out the bugs that you didn't find. case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

rob

[Juan Conejero]

Hi Geoff,

3.  I would have expected both integrated images to be identical, but they were not.  Subtracting one from the other with PixelMath gave an image that was non-zero.

There are very small differences caused by the way each format module loads pixel data. The FITS module has to communicate with CFITSIO (the software library that provides FITS format support), and this involves an "adaptation layer". In particular, pixel samples are always read as 64-bit floating point values from FITS files in order to support all possible data types (from 8-bit integers to 64-bit floating point), and then the values are converted to the data type of the image being read. For a 32-bit floating point image, this involves a conversion from double precision to single precision. In our XISF implementation pixel data are read and written directly without any additional conversions.

This conversion is not perfect. For example, 1.234567 in double precision gives 1.2345670461654663 when converted to single precision and then back to double precision. The tiny difference is caused by the different  binary representations of floating point types. Basically, this difference can be understood as a form of noise. However, this noise is limited to the least significant binary digit of the single precision type, which cannot be larger than 1.2x10^-7 (the machine epsilon for the single precision type), so it is insignificant for all practical purposes.

Median noise reduction    : 1.3839
Median noise reduction    : 1.0701

Effective noise reduction (ENR) estimates given by the ImageIntegration tool are not comparable between different data sets. They are relative estimates that you should try to minimize for a particular set of images by tweaking II parameters. Since they are not absolute estimates, a comparison as above is in general meaningless. The ENR function is extremely sensitive to small differences in the distribution of pixel values over low-signal areas of the image, which is a very good property in this case.

[Juan Conejero]

case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

Do you think this problem would have arisen had I released XISF as a beta version? Based on my experience after more than 10 years in this market, I bet it wouldn't. Unfortunately, this does not work that way. We perform exhaustive stability tests for each new release, but we cannot, as you say, verify every possible scenario where a specific tool or new feature can fail. We release when we are sure that what we are releasing is stable and works with our test data sets and stress projects. Then we make mistakes, overlook things, and have limitations, so bugs happen. But I believe that our current release policy is the best option to keep moving forward. IMHO, the improvements that we achieve are beneficial for all of our users, even for those that experience some annoyances.

[pfile]

case in point - a bug that just popped up this morning where saving XISF with japanese characters in the filename did not work. are you going to test every localization on your own? impossible.

Do you think this problem would have arisen had I released XISF as a beta version? Based on my experience after more than 10 years in this market, I bet it wouldn't. Unfortunately, this does not work that way. We perform exhaustive stability tests for each new release, but we cannot, as you say, verify every possible scenario where a specific tool or new feature can fail. We release when we are sure that what we are releasing is stable and works with our test data sets and stress projects. Then we make mistakes, overlook things, and have limitations, so bugs happen. But I believe that our current release policy is the best option to keep moving forward. IMHO, the improvements that we achieve are beneficial for all of our users, even for those that experience some annoyances.

who knows? if that japanese user had tried it, then yes. if you had put out a beta and said, "i didn't have time to test X, Y and Z" then someone might find the time to test those things. for instance, i speak a little japanese and i have 3 japanese input methods turned on on my mac. in theory i could have tested the localization for japanese even though i don't normally have those input methods turned on.

i'm just telling you you have a base of somewhat fanatic users that would be happy to test things out in advance of wide release. if you don't want to take advantage of that, it's your choice. if no one takes you up on it, then you were no worse off. if they do take you up and find some bugs, then less sophisticated users are spared some grief.

rob

[jkmorse]

Rob,

I respect what you are suggesting but tend to disagree with its application here.  We are already a relatively small community of users and, as this post has shown, even us non experts can help identify problems (though maybe with a bit less venom at times).  I think that is particularly true with sophisticated software like PI which has users with highly scientific/technical backgrounds and us liberal arts types who love the hobby but couldn't write code if our lives depended on it (that said, I love reading the discussions between the experts like yourself and Juan as you chart solutions for the uninitiated like me).

Second, I fear the impact of beta-testing time dilation.  From what I have seen, running that system can substantially slow the roll out of new developments.  True, when the update does arrive, it may be "cleaner" but I would rather gets the updates weeks or months earlier even if it means a few days of frustration as we hammer out the kinks.

For what it's worth, but as I have said before and as I am sure others feel as well, one of the best aspects of PI is its sophistication.  But the other is the amazing dedication of Juan and his team to keep making it better.  I have to say, i am thrilled every time I see a new announcement from Juan about a major new initiative even if it takes work to understand it and time for its full potential to be realized.  Never once have I thought, "oh god, what if this doesn't work right out of the box?"  If I wanted safe, I would have just kept painting my images with Photoshop.

Best,

Jim

[Jay W. Butler]

Juan,

I have as of late today (Tuesday, Wednesday in Europe) downloaded the debugged versions of PI and the batch pre-processing script you posted earlier today and have now used them successfully to create and process an LRGB image using bias, dark, flat and light frames generated in FITS format using TheSkyX. I have been able successfully to generate both XISF and FITS outputs by making the appropriate alternative selections in Global Preferences and in the BPP script window and to further process that output using a variety of PI processes. The problem I experienced earlier has disappeared.

Many thanks to you, Juan, for making the necessary bug fix in a timely way. I continue to believe, as I said from the outset, that PI is by far the best astronomical image processing tool on the market and, for my money, makes anything else unnecessary.

Jay

[jkmorse]

Jay,

Sorry for the venom comment.  It was a cheap shot.  But your note does verify the point I was making in my response to Rob.  Beta testing would likely have denied your ability and that of the Japanese user to help get the kinks out quickly.  And rest assured, you can count on Juan to work issues as fast as he can (if he would just give up that whole sleep thing).

Best,

Jim

[pfile]

sorry you had to waste so many electrons, but i never said it should be a closed beta.

[cdesselles]

Boy oh boy!  What a thread!  Even though I have sat out the all of the debate (pro or con) as I haven't had the chance to check out the new update yet, I applaud ALL of the posters here for one whale of an entertaining discussion. 

To Juan and all of the PI team:
 
Keep up the fantastic work!  Please... continue to drag the rest of us, kicking and screaming, into that unknown realm of advancement.   All some of us ask is that you remember that the vast majority of your user base are not PHD mathematicians.  Try to be a little less harsh in your descriptions that invariably go way over most of our heads.

To the user base:

Even as only a semi-technically trained professional, I could recognize the statement that XISF didn't replace FITS.  It enhances it with all of FITS functionality included within it.  As for file conversion, who among us has not made extensive use of converting our results from floating point FITS to integer Tiff or JPEG so we could share our results with other via Astrobin, Photobucket, or even Facebook.  This is a quick simple step that is of such little consequence I'm amazed it comes up at all.  On the other hand, we as a customer base should exercise our right to voice our opinions on features and user friendliness of such an amazing and powerful software package.  After all, we have paid for that right when we purchased a license.

Speaking for myself:

Part of what I paid for is not just the software itself, but the dedicated and very innovative team that keeps PI on the cutting edge.  There is no possible way that innovation can continue without allowing the innovators the leeway to innovate.  This is where a bit of trust comes to the fore.  I trust the PI team to keep this application at the forefront of astronomical digital image processing.  The PI team can trust in me to give them the benefit of the doubt when it comes to new updates, to let them know as fully and factually as I can when those updates don't quite meet expectations, and to support them as best I can to assist them in resolving such "bugs" when they inevitably arise.  Can either side ask for more?

     

[Buzz]

As a comparative newcomer I welcome the forsight that the PI development team have to ensure that the image processing capabilities of PI (and other) applications are not limited by the file format.  It certainly is pushing the boundaries. My images have improved greatly since taking the blue pill and I take every opportunity to encourage others to use PI.

At the same time, many of us have personal boundaries, influenced by the fact that the detailed operation of several key tools remains undocumented and requires extensive browsing and research to move forward.

My own helicopter view (and I am know of others who think similarly) is the effective use of PI is hampered by limited documentation and the advancement of new tools needs equal attention to supporting information to be at its most effective.

[Geoff]

Totally agree Chris. I'm sure the lack of documentation deters a lot of potential buyers, but more important than documentation is a "How to get started in PI". Yes I know there is plenty of web and video info, but a pull down menu item in PI itself with a link to an official PI document outlining the basics would be a great asset to people using the trial program.
Geoff