Making a Synthetic Flat question

[varmint]

Hello,

I am a recent convert to the Pixinsight Way of processing astro images, and I'm taking it in somewhat slowly.  I'm relative inexperienced when it comes to processing astrophotos, and only have logged about 6-8 hours of experience with PixInsight, but have to say that I'm quite pleased with how quickly I'm getting better results.

I am currently working on some images I captured last October of the  Eagle Nebula and unfortunately have a small dilemma.  I never took Flats and am looking to "fake" it with what I call a synthetic flat, not sure if that's the correct term, but basically I'm looking to create a "flat" from my image frames.  But I can't seem to get anything to work and was wondering what you'd recommend I try next (I was never very good at it with Photoshop either).

I couldn't find a tutorial or process flow from a quick search of this forum, but wouldn't be surprised if something is here and I just can't find it.  Please direct me if you can.

I tried a few things, and adjusted some settings, with DBE thinking that maybe I could get DBE to "find" the flat faults but was unsuccessful.

I'm figuring I need to make a combination of a star mask and a partial luminance mask, to protect the information I want, then subtract the rest of the image (the information I don't want).  I'm just not exactly sure how to do this effectively.

One other thing I can't seem to find is a running list of applied actions/processes/scripts.  I know the Processing Console lists the results of all actions, but I'm finding I often "undo" and it doesn't seem to log that the task was "undid" so I can easily see "where" I'm at in the actions I've done or undone.  Does this exist?

Thanks,

Jim

[Nocturnal]

hi Jim,

let me start by saying that if you're a bit inexperienced with processing then trying artificial flats may be a bit much Flats are meant to compensate for illumination differences by dividing the image by the normalized flat. That means that dim areas (edges and under dust motes) get 'amplified' for lack of a better word. DBE is typically used to remove 'additive' effects such as what is caused by light pollution. So you'd first flatten and then use DBE to remove residual gradients.

That said DBE does have a division operation but I have no experience with it. You could give it a try. If there is no light pollution then the background model as constructed by DBE should allow you to flatten accurately. It is unlikely the dust artifacts will be fixed though.

Now your image looks pretty flat already except for the dust motes so perhaps you can simply process it as is? Try stretching the image a bit more, it looks like there's a lot still there.

To see where you are in your processing undo/redo simply right click in the image and select 'load history explorer'.

[varmint]

That said DBE does have a division operation but I have no experience with it. You could give it a try. If there is no light pollution then the background model as constructed by DBE should allow you to flatten accurately. It is unlikely the dust artifacts will be fixed though.

Now your image looks pretty flat already except for the dust motes so perhaps you can simply process it as is? Try stretching the image a bit more, it looks like there's a lot still there.

Thanks Sander, the image did have a DBE subtraction, but as you state it took away the gradients but not the dust artifacts.  I did try the Division setting this evening, but that didn't work either.

I've attempted synthetic flats (not completely successfully) in Photoshop before, but I find the star detection map generation of PixInsight to be much better and easier than what I could do in Photoshop...I just can't seem to figure out how to create a mask that I could then re-apply to the master image to remove the dust motes.

To see where you are in your processing undo/redo simply right click in the image and select 'load history explorer'.

Rookie mistake.  I didn't have an image selected, now I can see it. THANKS!

[varmint]

I think I figured out a process to accomplish what I want, but after seeing the results I need to think through a better process flow.

I took the image posted on my Flickr account (well, the FITs version) and created a Structure Map Overlay star mask, adjusting settings so that I can easily see the "dust motes" in the mask.  Then I used Pixel Math to divide the Original by the Overlay.  This worked in eliminating the dust motes while still protecting the stars and the nebulosity, somewhat (I think I can create a better mask) but there's lots of color artifacts (looks like vertical banding) and the stars are off a bit.

Is this a suitable technique to accomplish what I'm trying or is there a different function I should be using?

Right now the image looks pretty horrible, but it doesn't have the obvious artifacts from a flat, so I feel like I'm on a better track.

Thanks in advance.

[Carlos Milovic]

Hey Jim

Can't you make a super flat from  individual light frames of the whole night?

[varmint]

Hi Carlos,

I don't completely understand your question, so please excuse me, I'll try to re-explain what I'm trying to do.

I captured several light frames, and a bunch of dark frames, but no bias nor flats of M16 (I had thought I captured bias, but I can't find the frames, I didn't learn how to take flats until just recently).  I reduced, aligned and stacked them but still have the flat artifacts.  I took the calibrated/reduced image into PI (still in it's linear form) and performed a DBE and Color Calibration, then applied some Histogram Transformation and noticed the dust motes were quite visible.

So that's when I started seeing if I could create a mask from the light image that would "act" as a flat.  Is this what you mean by a Super Flat?

Thanks again

[Carlos Milovic]

No, a "Super Flat" is a flat made from normal light frames, hopefully from relatively empty sky regions. If you made drizzle between your shots, or fired more than one object, you may try to integrate the light frames rejecting stars and other DSOs leaving only the sky background. This can be used as a flat frame. This technique is also called "night sky" flat.

[varmint]

Carlos,

Thank you for the clarification.  Let me check my files, I may have 2 objects from that night, and the previous night I may have had another (I don't think I moved the camera or took it off between sessions, but I'll inspect the frames to see).

Now I'll have to figure out how to reject the stars in the combination method.

[Carlos Milovic]

A median combine may be enough.

[RBA]

If you made drizzle...

Dither?

[Carlos Milovic]

dithering, yes

[varmint]

Sorry gents I didn't even know what dithering was until two weeks ago, so unfortunately I've not dithered..

Median combine didn't seem to work, I had three images I chose (one of M13, one of M27 and one of the M16), I ended up with a partial composite of all three M's unaligned...

I'm going to try redoing the reduction/calibration and combination, using the tutorials on CR2 workflow.

[RBA]

Jim, FWIW, I find "super flats" to be an "advanced" process. I haven't got into doing superflats yet.

If I didn't take flats on a given night, before getting into superflats, my approach would be:

1) Take them now with a light box, or next morning at dawn, etc. Sure, camera rotation won't be exactly the same, focus might be not the same, etc. but I'll only know how bad the results will be after I've done it. Sometimes it might just do what you wanted. If it didn't, you only lost a few minutes capturing the flats anyway...

2) DBE... DBE is this incredible tool that often does a really good job to solve a particular problem, but yes, sometimes it may take several attempts (or even several DBE applications) to get it done. Sometimes a lot of samples are needed, sometimes just a few is all it takes, sometimes you need to fool around with it for a while. Sometimes it may not do what you want because maybe what you want isn't really what DBE can do for you. It all depends...

Do this: construct a model, apply it, and use STF to "view" the stretched background model to see if it's shaped more or less the way you think it should. If it isn't, place/remove samples to be better defined as you think it should. Adjust some of the parameters, etc.

But of course, do take flats next time!

Sorry gents I didn't even know what dithering was until two weeks ago, so unfortunately I've not dithered..

Median combine didn't seem to work, I had three images I chose (one of M13, one of M27 and one of the M16), I ended up with a partial composite of all three M's unaligned...

I'm going to try redoing the reduction/calibration and combination, using the tutorials on CR2 workflow.

[varmint]

1) Take them now with a light box, or next morning at dawn, etc. Sure, camera rotation won't be exactly the same, focus might be not the same, etc. but I'll only know how bad the results will be after I've done it. Sometimes it might just do what you wanted. If it didn't, you only lost a few minutes capturing the flats anyway...

This is definitely my current approach when I miss taking flats at dusk...but last October I was still in the "don't know how to take these correctly/afraid of messing them up" mode.  

2) DBE... DBE is this incredible tool that often does a really good job to solve a particular problem, but yes, sometimes it may take several attempts (or even several DBE applications) to get it done.

After doing a lot of test runs and trials last night I started coming to this conclusion as well.  I know that DBE does a good job of removing gradients, I didn't know if it could handle a couple of the other artifacts that are present (dust motes and read/color noise).  I'm using this as a learning experience so I can better understand exactly what some of these tools do.  I am impressed with how quickly I feel I'm coming up to speed.

I'm still a "Padawan" if that rank is allowed , but I feel like I'm making bigger strides per attempt than other tools I've tried using.  

But of course, do take flats next time!

Yup, I've learned this lesson.    

[Carlos Milovic]

Yeah. At the end, nothing is better than true flats. Specially for those nasty donuts. Large scale uneven field illumination may be handled quite easily with theoretical models, based on samples, but if you have sharp edges, of high frecuency features, they are just too difficult to handle.