Author Topic: DSLR preprocessing: How to apply findings to BPP script  (Read 4746 times)

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Hi,

we recently had two very interesting discussions regarding the preprocessing of DSLR data: "Preprocessing Canon DSLR frames - a different approach" http://pixinsight.com/forum/index.php?topic=7006.15 and "DSLR Dark Investigation - Puzzling Result" http://pixinsight.com/forum/index.php?topic=6998.30 . I am not sure what the take-aways from the discussion are when thinking about the BatchPreProcessing script. Here is my interpretation:
- BPP should be extended allow to a pedestal parameter (which Tab would that be?).
- BPP should be able to work without darks (for cooled DSLR, I believe I always get errors when I dont have the complete set of calibration frames) -maybe with warning.
- there was some discussion about flat darks. BPP does not have suppport for these. Should these be added as well?
Anything I am missing?

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #1 on: 2014 May 08 02:10:32 »
BPP is a pretty complex but well written script.
Some people pretends that you should never use it because you lose control (but then what do they not do all their preprocessing with PixelMath?), other use it blindly with various rule of thumbs without checking the log or understanding the operations.
IMHO, like many thinks in PI, it is applicable in some cases and not in others.  I see two main use cases or type of users:
  • Experienced users usingit it to make fast calibration, as it handles the operations by filter automatically (I suppose that they have a library of master and undestand what happens).
  • Relatively novice or casual users that want to get something reasonable out quickly.
The experienced users may need more functionality. The more casual users would prefer simplicity but, above all, robustness. The BPP may not support all manual capabilities, but the user should be informed clearly (as much as possible) when there is a problem. BPP does already a good job with its warnings, but more could be done. Currently it is difficult to recommend it to beginners because if it is not adapted to their needs they may get bad results and not be aware of the problem.

Functionality:
  • PEDESTAL: I am not sure that PEDESTAL is that useful if the darks are calibrated only during integration of the lights (I would need to check the source). 
  • Flat darks: BPP already handles dark flats if they are present.  The problem is that it does not handle correctly the more common case of not having dark flat.  It also do useless bias calibration of darks when we do not use optimise, but this has no functional impact.
  • Master library: Maybe a simpler way to handle a 'master library', providing the root directory of such a library and letting BPP use the best dark/bias without having to load them explicitly would be useful and not too difficult (most of the logic is already in the script).

Robustness:
It is not always obvious that something did not work as expected. It would be nice if BPP could summarize the processing done and the warning at the end (in the log or popup), as examining a very long log is not very practical. The kind of unusual values or results that could be checked are:
  • Very small/large factors if optimization is used.
  • Unusually high rejection values when building a master dark/bias
  • Negative values when calibrating or integrating (light darker than the dark....)
  • Abnormal median of the bias,dark,flat (for example if the flat is less than 0.1 - it is probably the wrong file).
  • Abnormal relationship between those medians (bias higher than dark)
  • Files ignored (for example when a file add a zero size)
  • Others ?

Some of these capabilities may need the collaboration of the ImageCalibration and ImageIntegration tools (that are open source as well).

So you know what to do when it is raining  ;)

-- bitli

astropixel

  • Guest
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #2 on: 2014 May 08 05:48:29 »
Gentlemen. You are way ahead of me with this stuff. From a very basic user opinion, BPP is not too bad for beginners. The problem is that DSLR data does not always produce the results that one might get with linear data - which is very frustrating at times. I don't think pedestal is an issue. It's more the  non-linearity of the DSLR data that seems to be the problem - but, I am always happy to be corrected about these things.
« Last Edit: 2014 May 08 05:56:51 by Rowland »

Offline bitli

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 513
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #3 on: 2014 May 08 07:23:42 »
Quote
the  non-linearity of the DSLR data that seems to be the problem

Just a question, did you do measure some non  linearity? I have made very crude measurements (images of a flat with different exposure time) and I did not see significant non-linearity between the offset level and mid intensity (there is certainly non linearity near saturation)  My 'measures' are very crude (may be within 10%).  The problem of 'rebiased' dark levels had a much more significant impact. The effect may look the same at the end - badly corrected hot pixels if you scale the darks - but the cause and counter measure are different.

This was just one measure with one DSLR, so I would be very interested of what other people measure, which may or may not match what we suppose is happening.
-- bitli

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #4 on: 2014 May 08 08:54:42 »
i think he's talking about dark signal. it does not behave properly in canon cameras since they are preprocessing the "raw" data in the camera firmware.

rob

Offline Phil Leigh

  • PixInsight Addict
  • ***
  • Posts: 220
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #5 on: 2014 May 08 08:55:00 »
Lights and flats are highly linear from a modern DSLR... Darks... Not so much!

astropixel

  • Guest
Re: DSLR preprocessing: How to apply findings to BPP script
« Reply #6 on: 2014 May 08 15:23:02 »
OK guys, you've lost me - a rhetorical question - are you saying that DSLR non-linearity is a not an issue, but may be an issue with darks. Again, just inquiring, it would seem that darks is where problems with DSLR calibration begin and end. Phil's camera is capable of -25C, which makes darks a bit redundant, so lets leave that out. However, many DSLR cameras do not operate in those regions and darks are required.

I have not measured the linearity of flats or bias frames as I do not see that as much of an issue - no obvious calibration problems. I am not sure how reliable lights are because of camera firmware corrections - the same with darks. However, I find that the PI standard methodology invariably truncates darks and lights - but, bias_in_the_dark does not truncate data in light frames - to any observable extent.

That's it in a nutshell, based on experimentation. I think George is wanting to get to the bottom of this and see how it can be applied to BPP, empirically. I would also like a better understanding of the DSLR data issues that crop up frequently.