RAW file corrupted on load every time

[mdantonio]

Hi folks...

I have a sony A7Sii camera and it has a raw format of ARW which PixInsight says it can load. Excitedly I downloaded the trial to see if we might be able to use it for our observatory for stacking and processing duties...

My FIRST attempt to use it was a showstopper. The ARW file from the Sony camera was corrupted upon loading every time. PixInsight apparently CANT load these files even though listed as doable.

Using Irfanview a FREE program, it loads just fine and is perfectly viewable .. Photoshop loads them just fine with no issues whatsoever. PixInsight cannot. For that matter neither can DeepSkyStacker...

It turns out the connection between DSS and Pix is that they take their ARW codefrom the SAME faulty library which is DCRAW. That lib says that ARW is supported but it is NOT functional by all manner of attempts at testing it.. I tried to email the DCRAW author Dave Coffin a week ago or so and never heard anything back...

So are there any solutions before I just start going down the Photoshop only path at the observatory? We are going to be an automated facility where people

Marc Dantonio
John Zack Memorial Observatory
FX Models, LLC

[msmythers]

Marc

I have a couple of original Nex-5's which I have no problem loading those older ARW's with PI. Of course that camera can only store image files to a memory card, no computer support at all. Have you tried a raw file directly from the memory card? Might be worth trying.


Mike

[pfile]

just keep in mind that ARW (and CR2, and NEF) are not 'static' file formats. as new cameras come out, the manufacturers modify the file format to suit new sensor dimensions and features.

DSS has an even more downrev version of DCRAW than PI. Juan is going to release a new version of PI soon which should hopefully have the latest version of DCRAW. that may or may not fix the problem. if you have the skills, you can download the very latest version of dcraw and compile it, and see if it can successfully convert your ARWs to TIFFs. if it can, then the next version of PI will be able to do it.

rob

[Juan Conejero]

Hi Marc,

The current version of our DSLR_RAW module (which integrates the latest dcraw version 9.27) fully supports the Sony A7Sii. Can you upload one of these frames for us to check?

On the other hand, keep in mind that image retouching applications and viewers normally apply nonlinear transformations when you open raw frames. This is not the case with PixInsight, which loads and shows the linear raw data (with or without demosaicing). Can you describe what you understand by "corrupted upon loading"?

[mdantonio]

Hi Marc,

The current version of our DSLR_RAW module (which integrates the latest dcraw version 9.27) fully supports the Sony A7Sii. Can you upload one of these frames for us to check?

On the other hand, keep in mind that image retouching applications and viewers normally apply nonlinear transformations when you open raw frames. This is not the case with PixInsight, which loads and shows the linear raw data (with or without demosaicing). Can you describe what you understand by "corrupted upon loading"?

Hi Juan,
Thanks for getting with me on this. By Corrupted I mean the file loads as a purple set of finely spaced lines visually. That is what you see... There is no photographic data that is in any normal format. I attached ARW images directly from the card... One is completely corrupted and one is completely correct!

The first one (1062) is supposed to be of the ring nebula, a 13 second photo at ASA 32000with the A7Sii...

The second is a photo (1044) of Jupiter taken at 1/50 sec at ASA 50. THAT raw file comes in fine! But the other taken just minutes later doesnt ...

So you have two to compare ...Why does one load and display right and the other doesnt... That is the question... Does the ASA cause problems? Exp time? It seems to be a problem with DCRAW ..

Thanks, I would live to buy PixInsight for the observatory but being unable to stack raws is a huge problem... Love to know what you come up with.
Thanks Juan,
Marc Dantonio

Oh, I tried to submit the post and it said I cannot upload more than 512 k... Can I have an email to send the ARW files to?
Thanks
Marc

[pfile]

just put them on google drive or dropbox or something and post the link here

rob

[mdantonio]

Rob thanks... And that is why you are on the Jedi Council...  8)

[mdantonio]

Hi Marc,

The current version of our DSLR_RAW module (which integrates the latest dcraw version 9.27) fully supports the Sony A7Sii. Can you upload one of these frames for us to check?

On the other hand, keep in mind that image retouching applications and viewers normally apply nonlinear transformations when you open raw frames. This is not the case with PixInsight, which loads and shows the linear raw data (with or without demosaicing). Can you describe what you understand by "corrupted upon loading"?

Hi Juan,

Here is the link to the two ARW files I listed in my earlier post. Again one works, the shorter exposure low ASA one of Jupiter and the other doesnt, which is the ring nebula 13 seconds at asa32000.. Interestingly Google drive shows the ARW files fine...

Link is below...

https://drive.google.com/open?id=0B9q0xkdeK4b1ekUxa0dzYlg0YUk

[Juan Conejero]

Hi Marc,

Problem confirmed. For some reason the file DSC01062.ARW is not being decoded correctly by dcraw. The other file (DSC01044.ARW, Jupiter with satellites) can be loaded without problems.

This looks like a weird issue with pixel coordinates. I think the data is actually decoded and loaded, but pixel values are not being distributed correctly. Go figure.

Sorry for this problem. Unfortunately I have no solution, mainly because I don't know what happens with this file. Does this happen consistently with other raw files from the same camera, or is this an isolated case?

[msmythers]

Marc

I was also able to open both files in the original Image Data Converter program from Sony. That software is from 2011. Using the latest version of DCRAW that I could find, ver.9.27 I get the one good and one bad image no matter what settings I tried. The only difference I see between your 2 images is ISO which I would think would not be a problem for DCRAW. Here is the screen shots showing the info and images in Image Data Converter.

By the way, that's impressive for 13 seconds.


Mike

[mdantonio]

Hi Marc,

Problem confirmed. For some reason the file DSC01062.ARW is not being decoded correctly by dcraw. The other file (DSC01044.ARW, Jupiter with satellites) can be loaded without problems.

This looks like a weird issue with pixel coordinates. I think the data is actually decoded and loaded, but pixel values are not being distributed correctly. Go figure.

Sorry for this problem. Unfortunately I have no solution, mainly because I don't know what happens with this file. Does this happen consistently with other raw files from the same camera, or is this an isolated case?

Hi Juan... Sorry to hear its confirmed... I was really hoping that the problem was traceable to my camera. The Jupiter image and the ring nebula image differ both in exp time, 1/50 for Jupiter, 13 seconds for ring, and ASA, 50 for jupiter, and 32000 for ring... I left Dave Coffin a message about this but he didnt respond...

I can tell you that from my perspective I dont know where the cutoff lies for photos that fail versus those that do not. I can do an experiment to see because other products use DCRAW and it is universally failing across all those that do. Why Irfanview and photoshop can open these is a mystery. The one thing that is liikely true is that they are not usind DCRAW.

[mdantonio]

Marc

I was also able to open both files in the original Image Data Converter program from Sony. That software is from 2011. Using the latest version of DCRAW that I could find, ver.9.27 I get the one good and one bad image no matter what settings I tried. The only difference I see between your 2 images is ISO which I would think would not be a problem for DCRAW. Here is the screen shots showing the info and images in Image Data Converter.

By the way, that's impressive for 13 seconds.

It was a pretty cool shoot but I was disappointed that I could not then VIEW my images to stack them... Stacking Jpgs is meaningless.... I wonder if converting them to a tolerable format that isnt lossy first might be workable?

Mike

[Koten]

Hi there!
I have the same problem with my A7Rii.
No problems in daylight photos
Here is some example

1st is a light frame of pleiades and california nebula wide field (180secs of exposure @ iso 1600)
2nd is the same raw preprocessed (import then export in .dng) by Adobe Lightroom
3rd is an integration ho 40 dark frames of 900secs exposure @ iso 1000
4th image of the process console where I found something very strange during the opening of the ARW file.

As you can see in the 1st photo attached, the photo seems to be divided in 3 sectors. In each sector there is Pleiades (the extreme right part of the real photo)

Actually, to bypass the problem, I’m importing all arw files in Adobe Lightroom and exporting as .dng

You should really solve this issue!

[sharkmelley]

Hi there!
I have the same problem with my A7Rii.

[SNIP]

Actually, to bypass the problem, I’m importing all arw files in Adobe Lightroom and exporting as .dng

You should really solve this issue!

Something is being done.  It's called libraw.

Try this:  https://pixinsight.com/forum/index.php?topic=12126

Mark