Flats Don't Remove Artifacts

Farzad_k

Well-known member
Oct 15, 2016
178
1
Hi.

I am processing a fresh set of data and have created flats and flat darks as well as darks and biases. The camera is a ZWO ASI294 MC Cooled Pro. The results after BPP indicates a lot of artifacts still remain. I have placed the master files as well as sample flat and data subs >>>HERE<<<.

Thanks for any feedback.

Farzad
 

pfile

PTeam Member
Nov 23, 2009
5,298
63
do you have an example of a calibrated frame? that will help diagnose the image without going thru the exercise of running ImageCalibration on my end.
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Farzad,

Also include screenshots of your settings in BPP/WBPP.
Also...did you have a chance to review this WBPP YouTube series?

-adam
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Farzad,

I looked at your data. Your flats do not characterize your light frame. Your optical system changed between the time(s) you took your flats and your light frame data. My guess, you are reusing flat frames... but fundamentally something is different between the data and the flat.

The files calibrate just fine.

-adam
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
Farzad,

I looked at your data. Your flats do not characterize your light frame. Your optical system changed between the time(s) you took your flats and your light frame data. My guess, you are reusing flat frames... but fundamentally something is different between the data and the flat.

The files calibrate just fine.

-adam

Adam, someone else had the same suspicion. However, the flats are taken fresh after the scope was moved inside. I have noticed that the master flat needs to be rotated 180 degrees.
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
do you have an example of a calibrated frame? that will help diagnose the image without going thru the exercise of running ImageCalibration on my end.
I will work that out for you and send you a link shortly. Thanks.
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
Farzad,

Also include screenshots of your settings in BPP/WBPP.
Also...did you have a chance to review this WBPP YouTube series?

-adam

Adam:

I am getting fresh calibrations. >>>HERE<<< is a short video that shows my BPP settings. I have never used WBPP.
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
do you have an example of a calibrated frame? that will help diagnose the image without going thru the exercise of running ImageCalibration on my end.

>>>HERE<<< is a folder with one calibrated frame and one debayered frame. Let me know if anything else is needed. Thanks
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Adam, someone else had the same suspicion. However, the flats are taken fresh after the scope was moved inside. I have noticed that the master flat needs to be rotated 180 degrees.
Wait...what? There should be no rotation of files at any point! And...flats should be taken *before the telescope is moved* for best results.
You will need to explain that a bit more!
-adam
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Adam:

I am getting fresh calibrations. >>>HERE<<< is a short video that shows my BPP settings. I have never used WBPP.
You will want to learn WBPP going forward.
Something confuses me... the names for what presumably are your Flat Darks and your Flats are identical. That is concerning.
-adam
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
You will want to learn WBPP going forward.
Something confuses me... the names for what presumably are your Flat Darks and your Flats are identical. That is concerning.
-adam
They aren't. One is called Flat and another is Flat Dark.

I just found out about WBPP and I am on video 4. Definitely a lot of improvements to the original BPP.

Farzad
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
Wait...what? There should be no rotation of files at any point! And...flats should be taken *before the telescope is moved* for best results.
You will need to explain that a bit more!
-adam
Explaining the rotation of the integrated flats versus integrated Lights: when I compare the two with each other, and if I rotate the integrated flats 180 degrees then I get a matching artifact. I posted the masters to the shared folder.

Sometimes there isn't an option of doing flats while the scope is still mounted in perfect condition, but it is possible to carefully move the setup 10 feet from it and set it on a sturdy table.

Farzad
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Hmm... I guess I am really stuck on this because it is a red flag. You need to explain why you need to rotate in the first place. How is it you are generating a flat and light frame that are rotated with respect to one another? Is it something about how the software is saving the images?

But regardless- some of your dust shadows were calibrated and some were not- and others are embossed (not quite calibrating due to a shift).
So, the problem is has to do with how you are taking your flats...or what you are doing to your flats.

Make a video and blink your flat with your light and convince yourself that:
1. There are no shifts in dust shadows between the frames.
2. All dust shadows in one are in the other.

When I blinked your original image with your flat...this was not true. (Same thing when I blinked the original image with the calibrated image.)
It is almost as if... you combined two different sets of flats to make your master...so that some stuff calibrates... and some doesn't..this is bizarre.

You need to show all of your work... but the raw files I think.

-adam
 

pfile

PTeam Member
Nov 23, 2009
5,298
63
I will work that out for you and send you a link shortly. Thanks.
you don't have to do it; adam found what i was looking for. if you see embossed dust motes then it means that the flats don't match properly. the dust moved between light and flat acquisition.

rob
 

Farzad_k

Well-known member
Oct 15, 2016
178
1
Hmm... I guess I am really stuck on this because it is a red flag. You need to explain why you need to rotate in the first place. How is it you are generating a flat and light frame that are rotated with respect to one another? Is it something about how the software is saving the images?

But regardless- some of your dust shadows were calibrated and some were not- and others are embossed (not quite calibrating due to a shift).
So, the problem is has to do with how you are taking your flats...or what you are doing to your flats.

Make a video and blink your flat with your light and convince yourself that:
1. There are no shifts in dust shadows between the frames.
2. All dust shadows in one are in the other.

When I blinked your original image with your flat...this was not true. (Same thing when I blinked the original image with the calibrated image.)
It is almost as if... you combined two different sets of flats to make your master...so that some stuff calibrates... and some doesn't..this is bizarre.

You need to show all of your work... but the raw files I think.

-adam
I do not have an answer to the question of why I need to rotate. All that I have said is that in my diagnosing of the mismatch I notice that the integrated flats appear to be 180 degree rotated relative to the integrated lights when I compare the artifacts on the two.

I have actually done the blink now and the artifacts match exactly. Please see >>>THIS<<<, >>>THIS<<< and >>>THIS<<<.

Thanks
 
Last edited:

ngc1535

Well-known member
Feb 1, 2014
503
53
This is why I (sometimes) dislike the forum method... this could be handled in 2 minutes on zoom with a shared screen!

Some things in your flats are calibrating things fine... some things are not. You can be eye see the difference in dust (and how they are illuminated) between your raw light frame and your master flat. And they SAME THINGS that I see by eye that are different are exactly the things not being calibrated in the image (bottom).

Now here is a technique you can always do to prove this to yourself. Take your master flat and divide it by one of your flat subframes using pixel math. You can reverse the order if you want... you just might need to check rescale the image. This should result in a virtually FLAT and EVEN image. When I did this with YOUR subframe and master flat. Nope... no dice. To prove this, I include a screenshot of me taking a sub flat and dividing by a master flat... do you see how uniform that looks? Try that with your own data. Something with your optical system changed between the generation of the data and the generation of the flats. (or you are changing something else... you should not be rotating flats... should not be rotating camera... no changes).
 

Attachments

  • Like
Reactions: Farzad_k

Farzad_k

Well-known member
Oct 15, 2016
178
1
This is why I (sometimes) dislike the forum method... this could be handled in 2 minutes on zoom with a shared screen!

Some things in your flats are calibrating things fine... some things are not. You can be eye see the difference in dust (and how they are illuminated) between your raw light frame and your master flat. And they SAME THINGS that I see by eye that are different are exactly the things not being calibrated in the image (bottom).

Now here is a technique you can always do to prove this to yourself. Take your master flat and divide it by one of your flat subframes using pixel math. You can reverse the order if you want... you just might need to check rescale the image. This should result in a virtually FLAT and EVEN image. When I did this with YOUR subframe and master flat. Nope... no dice. To prove this, I include a screenshot of me taking a sub flat and dividing by a master flat... do you see how uniform that looks? Try that with your own data. Something with your optical system changed between the generation of the data and the generation of the flats. (or you are changing something else... you should not be rotating flats... should not be rotating camera... no changes).

Hey, thanks for taking the time.

So what I am hearing you say is that the flats are able to eliminate some artifacts and they are failing at eliminating some others.What would cause that, I wonder? Could it be that these spots that are not taken out just cannot be taken out?

I don't know much about pixel math but it is something I should be studying. I did a test today to see what could be going on. I removed a filter out of the way of the optics to create less dust factor, pointed the scope indoors to a wall and captured some light data. I know this may sound stupid, but it is the best that can do in a region of rare clear night skies. I then took some dark frames, again stupid, perhaps, given the duration of lights frames was very short (.5s), and then did some flats. I created master calibration files and calibrated the flats, and then calibrated, debayerd and integrated the lights. The master flat definitely shows a bunch of dust spots. The dust spots don't go away when calibrating the lights but in the master light they look like blisters. The below picture is the lower right corner at 1:1, NL Light, NL Flat.

I am not sure what I am doing wrong.


1593987099380.png
 

ngc1535

Well-known member
Feb 1, 2014
503
53
Would you mind if I make a small video featuring this question about flats using your example?

I will need to think about your new observations...but let me answer your first question.
I demonstrated that one of your subflats when divided into your integrated flat does not result in a uniform image.
This means that the flats you used to create the integrated flat had variations. This is my guess. I would need to blink the flats to see if this was indeed true. Without looking at everything in hand... it is very hard to say. But I hope you agree I am giving you fitness tests to work from.
-adam

P.S. The "blister" effect makes sense because this is bright source (your wall) with scattered light illuminating the dust in different ways. I
suspect this avenue isn't going to be helpful. (I am glad to be wrong though.)
 
  • Like
Reactions: Farzad_k

Farzad_k

Well-known member
Oct 15, 2016
178
1
Hi Adam, and thanks very much for making a video of this issue to help me and perhaps others.

I understand from the video that you believe something may have changed during the capture of the flat subs because dividing a flat sub to the flat master (or visa versa) leaves some of the artifacts. It is strange to me also.

I have placed a folder >>>online here<<<that contains the flat subs and the flat darks that were used to calibrate them and turn them into the master flat in case you want to evaluate them. They are in process of being uploaded and will be available in a few minites.

Thanks again, and I have been watching your WBPP videos as well, hoping to learn how to use it.

Farzad