Guide to Preprocessing of Raw Data with PixInsight

[CosmicCaptures]

The current version was revised on December 10, 2023.

This is the structure of the text:

Preface

A Camera Hardware and Control Software
A1 Setup and Properties of Digital Cameras used for Astrophotography
A2 Quality Settings, Camera Drivers, Image Acquisition Software and File Formats
A3 Metadata

B Calibration Frames, Master Calibration Files and Image Calibration
B1 Why Do We Perform Image Calibration?
B2 Types of Calibration Frames
B3 Settings and Conditions for Image Acquisition of Calibration Frames
B4 Generation of Master Calibration Files
B5 Light Frame Calibration with PixInsight's ImageCalibration Process
B6 Inspection and Judgement of Calibration Results

C Post-Calibration Preprocessing Steps
C1 Linear defect correction (LDC) [Only if linear defects are detected]
C2 Cosmetic correction
C3 Debayer [Only for OSC Cameras]
C4 Subframe Weighting with SubframeSelector
C5 Registration
C6 Local Normalization
C7 Integration
C8 Drizzle Integration [Recommended]
C9 Cropping

D Automation of Preprocessing: WeightedBatchPreprocessing (WBPP) Script

References and Endnotes

List of Abbreviations


Bernd

Thank you so very much for not only sharing this but periodically updating it as well. This is exactly the breakdown I needed to try and understand the calibration process and better utilize WBPP.

 

[Riccardo A. Ballerini]

For the ASI533MC (and its big brother, ASI2600MC), I've have had great success with darks, dark flats, flats but no bias. I now use WBPP with minimal PP. I started out just testing it with no flats and darks and the images were truly amazing, due to low noise and no amp glow, especially considering nothing was done with the lights. Of course, if your optics are dirty, the image will look nasty. So, not a good practice.

I do the opposite,
I bias instead of darkflat
I measured practically no differences in noise values between them

 

[bulrichl]

The following changes were made today in my guide "Preprocessing of Raw Image Data with PixInsight":
• Paragraphs about PixInsight Documentation and the list of abbreviations are moved into the new section "E Appendix".
• Endnotes and references are separated.
• The FastIntegration process is mentioned under post-calibration preprocessing steps.
• Sections about documentation are revised.
• Installation and use of local databases for ImageSolver, PCC, SPCC and annotation of solar system bodies is described.

Bernd

 

[AstroShed]

For the ASI533MC (and its big brother, ASI2600MC), I've have had great success with darks, dark flats, flats but no bias. I now use WBPP with minimal PP. I started out just testing it with no flats and darks and the images were truly amazing, due to low noise and no amp glow, especially considering nothing was done with the lights. Of course, if your optics are dirty, the image will look nasty. So, not a good practice.

+1 for the above
Darks, flats and dark flats for me with the QHY268c and mono versions works perfectly every time

 

[cloudbait]

+1 for the above
Darks, flats and dark flats for me with the QHY268c and mono versions works perfectly every time

There is no such thing as a "dark flat"... as Bernd observes in his guide. There are only three kinds of frames: lights, darks, and flats.

 

[AstroShed]

There is no such thing as a "dark flat"... as Bernd observes in his guide. There are only three kinds of frames: lights, darks, and flats.

I was just quoting the post i was replying too, so chill out…they are darks you use to calibrate your flats, so in my eyes a dark flat…

 

[cloudbait]

I was just quoting the post i was replying too, so chill out…they are darks you use to calibrate your flats, so in my eyes a dark flat…

Just keeping up a little pressure to use better, more consistent terms here in order to avoid confusion. "Dark flat" is simply wrong (even more wrong than "flat dark"). There is no such thing, and PI doesn't recognize such a thing, and camera control apps don't write such things.

Do you also use the term "dark light"? You know... the darks you use to calibrate your lights?

 

[AstroShed]

Just keeping up a little pressure to use better, more consistent terms here in order to avoid confusion. "Dark flat" is simply wrong (even more wrong than "flat dark"). There is no such thing, and PI doesn't recognize such a thing, and camera control apps don't write such things.

Do you also use the term "dark light"? You know... the darks you use to calibrate your lights?

You can moan and roll your eyes all your like, it’s your opinion…get a life, while I get on with the hobby…

 

[cloudbait]

You can moan and roll your eyes all your like, it’s your opinion…get a life, while I get on with the hobby…

It's increasingly seen as the proper usage. And using words properly and precisely reduces confusion... which is a wise thing to do in an activity like astroimaging where a great deal of confusion is possible!

 

[AstroShed]

It's increasingly seen as the proper usage. And using words properly and precisely reduces confusion... which is a wise thing to do in an activity like astroimaging where a great deal of confusion is possible!

It doesn’t confuse me at all, I know exactly what I am talking about…..

 

[AstroShed]

I have been in the hobby for over 20 years, and I could also send you links to literally thousands of instructional videos and websites for the hobby, which are still widely used in teaching, who all use either the term Dark Flat, or Flat Dark, it’s widely understood what they are what they do, so to try and change that now is a pointless task…
You may think it’s wrong, incorrect or confusing, well good for you, but good luck with trying to change the rest of the imaging community, especially those that don’t use Pixinsight…
And we are off topic so have a good day…

 

[cloudbait]

I have been in the hobby for over 20 years, and I could also send you links to literally thousands of instructional videos and websites for the hobby, which are still widely used in teaching, who all use either the term Dark Flat, or Flat Dark, it’s widely understood what they are what they do, so to try and change that now is a pointless task…
You may think it’s wrong, incorrect or confusing, well good for you, but good luck with trying to change the rest of the imaging community, especially those that don’t use Pixinsight…
And we are off topic so have a good day…

Again, I'm only pointing out that the usage is increasingly seen as obsolete and confusing, including in this forum. If you choose to use inaccurate terminology, nobody is going to stop you. Objectively, there is no such thing as a "dark flat", and the term itself represents poor English usage, because it isn't even a flat. It might sound better in a language that places its modifiers after its nouns...

 

[nico1038]

+1 for the above
Darks, flats and dark flats for me with the QHY268c and mono versions works perfectly every time

Have you tested the results against a more "classic" calibration with Darks, flats and bias? I would be surprised if there is any differences: with my 2600MC and 533MC cameras I couldn't find any significant difference between a bias and a 5s dark.

 

[AstroShed]

Have you tested the results against a more "classic" calibration with Darks, flats and bias? I would be surprised if there is any differences: with my 2600MC and 533MC cameras I couldn't find any significant difference between a bias and a 5s dark.

TBH, the reason I use darks and not bias, is that I don’t dither, and I find the darks pretty much eliminate walking noise and hot pixels, so have stuck with what works for me…but if I dithered then yes I would probably use bias as you say….

 

[cloudbait]

TBH, the reason I use darks and not bias, is that I don’t dither, and I find the darks pretty much eliminate walking noise and hot pixels, so have stuck with what works for me…but if I dithered then yes I would probably use bias as you say….

I think the question was about using bias frames to calibrate your flats, not about using them to calibrate your lights.

 

[AstroShed]

I think the question was about using bias frames to calibrate your flats, not about using them to calibrate your lights.

Either way i don’t use them…if this is ok with you…and the question was if you take the time to read, was
Have you tested the results against a more "classic" calibration with Darks, flats and bias?
Not exclusively using bias for flats

So why don’t you just but out

 

[cloudbait]

Either way i don’t use them…if this is ok with you…and the question was if you take the time to read, was
Have you tested the results against a more "classic" calibration with Darks, flats and bias?
Not exclusively using bias for flats

So why don’t you just but out

Somebody got up on the wrong side of the bed today...

(As the comparison was for a bias compared with a 5-second darks, it's clear he wasn't talking about using bias frames to calibrate lights.)

 

[AstroShed]

Somebody got up on the wrong side of the bed today...

(As the comparison was for a bias compared with a 5-second darks, it's clear he wasn't talking about using bias frames to calibrate lights.)

He was replying to my post to someone else, so why the hell do you have to be involved, your should be called clickbait rather then cloudbait…

 

[nico1038]

What I was trying to say is that, with the IMX571 sensor, calibrating flats with bias or with short exposure darks doesn't make any differences (or at least I couldn't see any in my tests)

 

[AstroShed]

What I was trying to say is that, with the IMX571 sensor, calibrating flats with bias or with short exposure darks doesn't make any differences (or at least I couldn't see any in my tests)

Sorry about the idiot who keeps replying, I have blocked him now.
no i have not tried that, and I would say you are correct, I would not see any reason at all they they would not work just as well….something to try