Author Topic: New Version of BatchPreprocessing Script with Bayer Drizzle Support  (Read 54638 times)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
For example: BatchPreprocessing calibrate MasterFlat which already calibrated . Impossible to disable calibrate it.

We haven't seen this problem, and I am unable to reproduce it with any of our test data sets. Can you put a repeatable example of this problem?
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline vicent_peris

  • PTeam Member
  • PixInsight Padawan
  • ****
  • Posts: 988
    • http://www.astrofoto.es/
We plan on releasing new tool versions to support Bayer drizzle outside BatchPreprocessing. They will be released as updates to the upcoming version 1.8.3 of PixInsight.
Juan, please do it faster.
Because BatchPreprocessing don't provide full of power manual calibration and do it wrong. For example: BatchPreprocessing calibrate MasterFlat which already calibrated . Impossible to disable calibrate it.

Hi Nikolay,

You mean that, when using a master flat frame (checking the "Use master flat" option), the master flat frame is again bias and dark subtracted?


Best regards,
Vicent.

Offline NKV

  • PTeam Member
  • PixInsight Guru
  • ****
  • Posts: 677
We haven't seen this problem, and I am unable to reproduce it with any of our test data sets. Can you put a repeatable example of this problem?
OK, I prepare set of files.

You mean that, when using a master flat frame (checking the "Use master flat" option), the master flat frame is again bias and dark subtracted?
Yes.

We plan on releasing new tool versions to support Bayer drizzle outside BatchPreprocessing. They will be released as updates to the upcoming version 1.8.3 of PixInsight.
Hi, Juan.
Independent Bayer drizzle possible right now.
Please check my new module CFA2RGB

Offline Rod771

  • Newcomer
  • Posts: 22
Guys, I keep coming back to this when process my under sampled data.

I must have missed something? When generating a bayer drizzle image, when do you use the files in <output-dir>/calibrated/light/bayer. ? It says the Drizzle integration tool will use thm , however Step 7 in the OP says "After ImageIntegration, open the DrizzleIntegration tool. Select the same .drz files on the <output-dir>/registered/bayer directory" These are the registered file that Image Intergration used. Just confused a bit.

Cheers

Rod

Offline jkmorse

  • PixInsight Padawan
  • ****
  • Posts: 931
  • Two questions, Mitch . .
    • Jim Morse Astronomy
Rod,

Yes, they are the same .drz files that you added in the Image Integration step.  Image Integration updates the ones that were produced as a part of Star Alignment so after Image Integration you should have an entirely new set that are in the same folder that you used to populate the Image Integration tool.  Load them into Drizzle and watch the magic happen  :D

Best,

Jim
Really, are clear skies, low wind and no moon that much to ask for? 

New Mexico Skies Observatory
Apogee Aspen 16803
Planewave CDK17 - Paramount MEII
Planewave IFR90 - Astrodon LRGB & NB filters
SkyX - MaximDL - ACP

http://www.jimmorse-astronomy.com
http://www.astrobin.com/users/JimMorse

Offline Rod771

  • Newcomer
  • Posts: 22
Thanks Jim

So to confirm , the files exported to <output-dir>/calibrated/light/bayer  during BPP are never selected in Image Integration or Drizzle Integration. Only use files in <output-dir>/calibrated/registered/bayer.

Offline Alejandro Tombolini

  • PTeam Member
  • PixInsight Jedi
  • *****
  • Posts: 1267
    • Próxima Sur
Hi Rod, here there is a summary of the steps that may help
Saludos, Alejandro

Offline jkmorse

  • PixInsight Padawan
  • ****
  • Posts: 931
  • Two questions, Mitch . .
    • Jim Morse Astronomy
Rod,

That sounds right but I have to confess I have never used BPP.  I want total control so do all the steps manually.

Sorry,

Jim
Really, are clear skies, low wind and no moon that much to ask for? 

New Mexico Skies Observatory
Apogee Aspen 16803
Planewave CDK17 - Paramount MEII
Planewave IFR90 - Astrodon LRGB & NB filters
SkyX - MaximDL - ACP

http://www.jimmorse-astronomy.com
http://www.astrobin.com/users/JimMorse

Offline whwang

  • Member
  • *
  • Posts: 62
Hi Juan,

In this thread:
http://pixinsight.com/forum/index.php?topic=7151.0
you instructed that we should select the "generate drizzle data" option in the ImageIntegration tool.  On the other hand, for Bayer Drizzle here, you did not instruct to do so.  May I ask why is there such a difference?  I guess what I am trying to understand is what this "generate drizzle data" actually does.

Cheers,
Wei-Hao

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Hi Wei-Hao,

Quote
On the other hand, for Bayer Drizzle here, you did not instruct to do so.

I forgot to mention it. Yes, the ImageIntegration tool must always update the .drz files with rejection and normalization data, so the generate drizzle data option has to be enabled also in the case of Bayer drizzle. Note that ImageIntegration enables this option automatically when you select drizzle files (by clicking the Add Drizzle Files button), so in general one doesn't need to care about this.

Quote
what this "generate drizzle data" actually does

A drizzle data file (with the .drz suffix) stores all the data required to perform a drizzle integration process:

1. The full file path of the source drizzle image (calibrated, de-Bayered, but not registered). This is the image that will be integrated with DrizzleIntegration.

2. The full file path of the registration target image (calibrated, de-Bayered and registered). This is the image that should be integrated with ImageIntegration to compute normalization parameters, image weights, and pixel rejection data. This image will not be used in the final drizzle integration process.

3. The geometry of the reference registration image. This defines the geometry of the final drizzle-integrated image (multiplied by the drizzle output scale).

4. The geometrical transformations necessary to register the image. This may include a projective transformation matrix and/or surface spline control points and parameters, if local distortion correction is being used. These transformations are applied by DrizzleIntegration to compute the geometry of each 'drop' that 'rains' over the drizzle-integrated image

5. The coordinates of each rejected pixel. In the next version of the ImageIntegration module this will be more complex, since we'll implement a fuzzy-logic rejection feature. For now, rejection is a purely binary process (rejected/not_rejected).

6. Image normalization parameters, including estimates of statistical location and scale.

7. Image weighting parameters. These are normally based on noise estimates computed during the image calibration phase (and stored as XISF properties and FITS keywords in the image), but can also be generated from statistical parameters or image properties/keywords by ImageIntegration.

Drizzle data files are created by the StarAlignment process with items 1, 2, 3 and 4. The ImageIntegration process updates drizzle files with items 5, 6 and 7.

In the case of Bayer drizzle, the trick consists of replacing item 1 with the path to the calibrated Bayer image, that is, the image after calibration but *before* applying the de-Bayering process. However, drizzle cannot use the monochrome CFA data directly (mainly because color planes would get intermixed without sense), so the CFA must be split into three separate R, G and B planes, with black pixels replacing the lacking data in each plane, and conveniently packed as an RGB image. This is what we call a separate RGB Bayer image. Then the drizzle process can be applied unmodified, as if it was working with a normal RGB image. The BatchPreprocessing script does this automatically, including generation of separate RGB Bayer data under the hood. The same task can also be implemented manually, with the help of Nikolay's SplitCFA tool to automate part of the process, but it is somewhat tedious.

Right now the Bayer drizzle method can only be implemented as an automatic process using the BatchPreprocessing script. An optimal implementation of Bayer drizzle requires a tight integration with a preprocessing pipeline, such as BPP. This will also be implemented outside BPP, probably as a standalone tool, but we have other priorities at present.

Hope this clarifies what this drizze data stuff is all about.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline whwang

  • Member
  • *
  • Posts: 62
Thanks for the clarification and explanation.  Now I understand the whole thing better.

Cheers,
Wei-Hao

Offline IanL

  • PixInsight Addict
  • ***
  • Posts: 116
    • The Imaging Toolbox
Re: New Version of BatchPreprocessing Script with Bayer Drizzle Support
« Reply #41 on: 2015 November 06 11:44:12 »
Note that there is now a new BayerDrizzlePrep script available for testing here:

http://pixinsight.com/forum/index.php?topic=9218.0

This should make it possible to perform stand-alone Bayer drizzling more easily, without the need for the BPP script.

Offline whwang

  • Member
  • *
  • Posts: 62
Re: New Version of BatchPreprocessing Script with Bayer Drizzle Support
« Reply #42 on: 2017 September 19 02:08:19 »
Hi,

I see that in the latest Batch Processing Script (1.46) there is no longer a Bayer drizzle check box.
How can Bayer drizzle be applied now?

Cheers,
Wei-Hao

Offline mcgillca

  • PixInsight Enthusiast
  • **
  • Posts: 77
    • Astrobin Website
Re: New Version of BatchPreprocessing Script with Bayer Drizzle Support
« Reply #43 on: 2017 September 19 04:49:58 »
Hi,

I see that in the latest Batch Processing Script (1.46) there is no longer a Bayer drizzle check box.
How can Bayer drizzle be applied now?

Cheers,
Wei-Hao

Dear Wei-Hao,

In the release post for 1.8.5, there is this comment:

* DrizzleIntegration can now work with monochrome CFA frames directly. That is, Bayer drizzle is now supported directly without needing to create RGB Bayer images anymore. The BatchPreprocessing script has also been updated accordingly.

I'm guessing that this is the answer - however, I think you need to create the RGB bayer images still so you can carry out the standard integration to allow pixel rejection BEFORE you do the drizzle integration.

I hope Juan can give us the definitive answer, since the above is just a guess - it would be useful to find out the complete workflow. I think it is:

1) BPP for most of the work, with the drizzle option selected.
2) LN to set up the LN files.
3) Image integration with the LN files and drizzle files (taken from the registered images folder, rather than the registered debayer folder?)
4) Drizzle integration with the LN files and the drizzle files again.

Can anyone confirm this is correct?

Thank you,

Colin

Colin
Paramount MX
Ikharos 8" Carbon Truss RC
Atik 460ex
Atik EFW2 with Baader LRGB and HA SII OIII
ONAG with Atik 314L+
http://www.astrobin.com/users/mcgillca/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New Version of BatchPreprocessing Script with Bayer Drizzle Support
« Reply #44 on: 2017 September 19 06:55:36 »
Hi Wei-Hao / Colin,

As Colin has pointed out, drizzle works now for CFA frames just as it has always worked for monochrome CCD frames. Enable generation of drizzle files in BPP (it is now enabled by default), then proceed with calibration, debayering, and registration in BPP, as usual. Update the .drz files with ImageIntegration, also as usual. Then use DrizzleIntegration with the CFA option enabled. Since version 1.8.5 of PixInsight, Bayer drizzle is applied in a completely automatic way directly on the monochrome CFA frames, so RGB Bayer images are no longer needed.

Quote
1) BPP for most of the work, with the drizzle option selected.
2) LN to set up the LN files.
3) Image integration with the LN files and drizzle files (taken from the registered images folder, rather than the registered debayer folder?)
4) Drizzle integration with the LN files and the drizzle files again.

LocalNormalization *must not* be used 'blindly' on a regular basis, and definitely *must not* be used without a correct understanding of the LN algorithm, its limitations and potential issues. As most inverse problems, local normalization is powerful but inherently ill-posed, so it must be applied along with a critical analysis of its performance for each particular data set. Please wait until I complete a new tutorial on the LN tool.
Juan Conejero
PixInsight Development Team
http://pixinsight.com/