Batch-Preprocessing Sony .ARW files (not getting anywhere)

[zillion42]

Hello again,

I followed the tutorials

http://troypiggo.blogspot.com.au/2012/06/pixinsight-tutorial-batchpreprocessing.html
http://www.harrysastroshed.com/preprocessing.html

which were both straight forward, and not much to do wrong there. Nevertheless I run into many different problems.

I have taken adequate amounts of darks, bias and flat frames for each iso setting for which I have lightframes. I have previously posted some information I found on the net on how to create good dark, bias and flat frames, asked questions whether those instructions are satisfactory for PI, so I guess my dark, flat and bias frames should be OK. It seems like creating the masters is not really a problem. I will upload them here in case anyone is willing to check them out.

My concrete problems start with calibrating the lightframes. As previously mentioned my images are sony ARW files and of course they are CFA images. I could try, of course, to export the the individual channels in grayscale out of photoshop but I never heard of such procedures. It seems RGGB is the order of the color matrix and VNG was the default setting for debayering, so I left it at that. However the debayered and calibrated files look horrid, there is way more noise and lots of individual Red Green and Blue Pixels all over the image. I will also upload one of the calibrated frames here together with one of the originals (limited webspace), just in case again anyone is willing to take a look.

Now my next problem is registering the images. Usually it takes at least a good hour for all my lightframes (Minimum specs core duo 4GB ram) because about 50% of them fail. If you look at the image I uploaded (they all look about the same), it seems to me, there is not a horrible amount of distorsion nor star trailing.
Now I'm not really sure if I should mention the next thing that comes to my mind, but then again, no harm done if anyone doubts the credibility of my statement: Because I currently study space and aerospace computer science I have recently written my own star matching algorithm based on the angles in the triangle composed of each star and its two nearest neighbors in the Hipparcos catalogue. Using my own program I am able to match several triangles in each of my images to the HIP catalogue, so I understand what should be happening, I just don't understand why it fails in PI, or better (I understand there can't be anything wrong with the SA module), why do I fail using SA properly ?

The initial star matching routine is based on triangle similarity. As a result, the whole image registration process is rather intolerant of global distortion.

I set the matcherTolerance parameter to 0.01, instead of its default value of 0.003. This parameter is not accessible from StarAlignment's GUI. It controls the maximum difference in triangle similarity tolerated by the star matching algorithm. A higher value makes the algorithm more tolerant of distortion, although less robust too, which may be causing these problems. This value worked very well with all of the images I used to test the script.

Would it be possible to put that value in the SA GUI in future versions ?

I usually have around 25 tries and get the

*** 0 star pair matches found - need at least six matched stars.

and finally

*** Error: Unable to find an initial set of putative star pair matches
<* failed *>

errors.

So all in all I sincerely hope anyone is able to help me out...

Big Thanks in advance.
Tobi

[pfile]

something has gone wrong in your calibration. the calibrated image has lots of hot pixels and noise. the uncalibrated image does not look that bad.

so in the calibrated image, all that noise is fooling the star detection routine, which is finding 10s of thousands of "stars" which are really just hot pixels. increasing the noise scale and hot pixel removal in StarAlignment helps, but you need to attack the real problem. if you do star detection on the uncalibrated image you'll see that the defaults in StarAlignment find only actual stars.

with a DSLR you have to be very careful to match your dark temperatures to your lights. you should make a master dark from 10s of frames and do pixel rejection on the stack. if you want to scale the darks, make a master bias out of 100s of frames (yes 100s) and calibrate the darks with that master bias before creating the calibrated master dark.

rob

[Carlos Milovic]

Just downloaded your files, and the first thing I noticed is that your flats are way too dark. They should be near half the full depth of the sensor. So, in your case, near 0.03 in the normalized range (since your camera appears to have a 12bit ADC). They read just "Mean 0.0028833". So, your flats have a lot of noise. Increase exposure times by a 10 factor.
Also, how many are "adequate"? I would not use less than 20 flats, and 60 bias... darks more than the lights, hopefully twice as many.

[zillion42]

Hello again,

and thank you very much pfile and Carlos Milovic

@Carlos Milovic
my individual flats are by no means as dark as the master flat came out in PI. I was thinking its dark because its linear 32 bit, but then maybe something went wrong there aswell. I can show you how they looked before they were binned.
Converted to jpeg:


P.S. In my case I took 15 flats for each iso setting I use. So that's what I considered adequate. So You think I should take ten more for each 400,800,1600,3200 iso ?

[Carlos Milovic]

Is this a jpeg that came out from the camera. If so, then it is not linear. This data has being stretched by a gamma transform (it is an automatic process). Neither the histogram shown by the camera is the true histogram. It also uses the jpeg visualization.
Increase the flat exposure time by a factor of 10. Also use more frames.
If you are using the "AV" mode in your camera, increase the exposures by setting the exposure compensation to +2 steps.

[zillion42]

No

Converted to jpeg:

this is a RAW converted to jpeg without any non linear stretch.

if I increase the exposure or F-Stop of the flats I took by factor 10, They will be completely white.

PS: since I used a white baby cloth in front of my scope looking at a afternoon sky for this, I didn't use AV mode, rather I took a frame and used custom white balance, which was about 7k° Kelvin

[pfile]

here's how to check the flat.

1) overexpose a flat, like completely overexpose it... 30 seconds or something.
2) set DSLR_RAW to raw mode (hover mouse over format explorer on left, then double click on DSLR_RAW in the menu that appears)
3) open the overexposed flat, debayer using the Debayer process. Superpixel is fine
4) open the histogramtransformation process and take note of where the spike is. chances are due to 14-bit vs. 16-bit issues it will be 25% of the way over. the purpose of the foregoing is to figure out where full saturation is.

then try to expose your flats so that the histogram is midway between where that spike was in #4 and 0. if you look at this flat with any terrestrial processing program it's going to look over exposed, but in fact it is not.

custom white balances can be deceiving - all they are is multiplicative factors and do not affect the actual sensor data. postprocessing programs just multiply 2 of the channels by those factors before presenting you the image.

[Carlos Milovic]

White balance has nothing to do here. RAW files are not affected by it. They are the pure ***raw*** data. It is linear, and in your camera, seems to be of 12bits.
Your flats are definetely under-exposed. And this flaw is making them to have that insane amount of noise. And, since flats are applied by division, noise is amplified. There is nothing to do, but improve your flats. This is not a fault of software.

BTW: Binning is a whole different thing.

[zillion42]

Hi,

No

Quote from: zillion42
Converted to jpeg:
this is a RAW converted to jpeg without any non linear stretch.

so being a PI Padawan I have come to the conclusion that it is very unwise to disagree with PixInsight Jedi Masters or Gurus because surely the force will come around in one way or another to strike your Harddisk with bad sectors.

I just had a major hardware breakdown here, so now all is back up and running, I will take a look at the flats.

So to get this right:
1. I open DSLR_RAW settings and chose create RAW Bayer CFA Image
2. I open the Debayer process and chose Superpixel (for testing)
3. I check the spike in the Histogramm Transformation process

Ok, looking at an overcast sky at 2pm with a white cloth in front of the scope, the camera set to 400iso the spike for a totally overexposed (1s) debayered RAW image is at x=0.0628. There is no gradient in that image, a screenTransfer leaves the image completely black.
Hmm, Ok, doing a flat with various exposure settings I have three spikes, RBG (in that order), unfortunately each channel is exposed slightly differently.
Here are the values:
1/25s   R=0.016 B=0.031 G=0.043
1/30s   R=0.012 B=0.024 G=0.031
1/40s   R=0.012 B=0.020 G=0.027
1/50s   R=0.008 B=0.016 G=0.023
1/60s   R=0.008 B=0.012 G=0.016
1/80s   R=0.007 B=0.012 G=0.016
1/100s R=0.004 B=0.008 G=0.012

Which one would you think is okay ? I would probably choose the 1/25s or 1/30s.

Thanks a lot in advance,
Tobi

[Carlos Milovic]

Both seem to be a safe choise. I would go for 1/30s.
Remember to take a lot of flats. No less than 20.

[zillion42]

Ok,

I'm having some progress now. Although altogether I'm not sure whether my calibrated, debayered and registered master light frame has really improved its SNR at all. The problem is really, I'm not used at looking at raw data in all its "raw-ness", everything I normally see is already been through loads of sony smoothing, levels and ICC adjustments.

If I open the raw with photoshop, it is as if I never had any noise, like, automatically, the other application I normally see my RAW's in is Sony Image Data Converter.

I can tell that compared to the individual Light frames, the  master/light-BINNING_1.fit is much improved, especially the vignetting is almost completely gone. Altogether, the colors just seem wrong however. The attached screenshot is loaded and STF autostretched. I can the tweak the colors back in the histogram transformation, if I do the individual channels.

I went ahead and read the dcraw documentation to see what

Code: [Select]

dcraw -a -w -D -k 0 -t 0 -o 0 -4 actually does (standard in the batch preprocessing for opening RAW's)

• -a
  Calculate the white balance by averaging the entire image.
• -w
  Use the white balance specified by the camera. If this is not found, print a warning and use another method.
• -d
  Show the raw data as a grayscale image with no interpolation. Good for photographing black-and-white documents.
• -D
  Same as -d, but with the original unscaled pixel values.
• -k darkness
  When shadows appear foggy, you need to raise the darkness level. To measure this, apply pamsumm -mean to the dark frame generated above.
• -t [0-7,90,180,270]
  Flip the output image. By default, dcraw applies the flip specified by the camera. -t 0 disables all flipping.
• -o [0-5]
  Select the output colorspace when the -p option is not used:
  0   Raw color (unique to each camera)
  1   sRGB D65 (default)
  2   Adobe RGB (1998) D65
  3   Wide Gamut RGB D65
  4   Kodak ProPhoto RGB D65
  5   XYZ

Is there any way I can correct the colors by loading an ICC profile maybe ?

Thx in advance,
Tobi

PS: that is
5 x iso 1600, 30s lightrames
61 x iso 1600, 1/4000s bias Frames
31 x iso 1600, 1/5s flat frames (properly exposed)
and
9 x iso 1600, 30s darkrames (-1°C, taken on the same day as the lightframes, few hours later)

[zillion42]

Hello,

... ind end... and I'm rather sad to state this, considering honest efforts of people trying to help me, and honest efforts of myself trying to apply PI DSLR raw workflows, I found that using Long Exposure Noise Reduction built into my camera delivers far less noise than any Bias and/or Dark Masters that PI offered to me, were able to do. Again that is sad, especially since I kind of feel like wasting money in your (still honored) software.

The only reason, I still use PI so far is DBE (Dynamic Background Extraction) and ACDNR (Adaptive Contrast-Driven Noise Reduction), which seem to offer two methods to 1) get rid of light pollution to some extend and 2) reduce Noise a little more in balance between contrast and noise.

PI, yet is waiting for, but seems like unable to explain to DSLR users, its very secrets, and unless that's not changed, in an easy to grasp yet essential Tutorial (for the masses) I can not ( and will not ) recommend PI to anyone in that user segment.



Regards,
Tobi

P.S. Please prove me wrong, and if so, make tutorials

[pfile]

uncouple the channels in STF and reapply it, you will see something that's pretty close to correct.

or after DBE with 'normalize' unchecked, a linked-channel STF will probably give you something close to correct.

the bottom line is that after raw processing you have to do color calibration one way or another to fix your image. photoshop, etc. always stretch the image and undo the green weighting that the raw images have. PI won't do that (and it shouldn't)

rob

[georg.viehoever]

I am quite happy with the results that the BatchPreProcessing script gives me for my DSLR images. Results are comparable or better to the results of DeepSkyStacker. Sometimes I need to deactivate the adptive weighting (i do not really know why), but otherwise there are only positve surprises...

Georg