Batch processing help DSLR

[Rod771]

HI all

Complete newbie here.

I've found and run the batch processing script for the first time and have an ugly result.

I have used Pixinsight for a couple of images before which have turned out good. However for these images I only registered and integrated as un-calibrated  DSLR raw images.

Now with a new target (Helix) I wanted to apply some flats, darks, bias frames to get some better results. I watched Harry's batch processing video and tried out the script.

I dont know what I've done. After applying Bais x15,  Darks x 15 (same ISO same exposure length as lights ,different temp) and Flats x 10 (ISO 800) and debayering I end up with this image below.

Lights were not dithered could that be it?

Camera is the 60Da with clip CLS - CCD filter,  I used RGGB as the debayer option. I have done a stack of reading and searched forums only to have ended up with "information over" and my brain has crashed.

Thanks for help

Cheers Rod

[pfile]

what was your sensor temperature like during the lights vs. darks? they have to match or you get stuff like you see in your image... dithering will help reject those hot pixels, but you have to use a pixel rejection method. load the _c_d_r.fit images into ImageIntegration and experiment with pixel rejection.

[Rod771]

Hi pfile

The temps were different and I was concerned about this , but the Pixinsight tutorial said not to worry about matching the temps as it rescales the dark to match the lights. I have run the whole batch processing script again , without darks , just bias and flats but have ended up with same result???

Im out at the moment and will try different pixel rejection setting when I got home, Thanks.

Could it be my flats? I've found these the hardest to acquire. As I use the hyperstar lens and the camera is hanging of the front of the OTA, i cant do white t-shirt flats and I stuff up sky flats all the time. My flats are taken by plugging the laptop into the LCD TV , displaying a blank full screen document (so white screen) and pointing the scope at the TV with the CLS-CCD filter in. I wonder if my method is good enough?

Thanks Rod

[pfile]

ah yes, the CLS, sorry i missed that.

the CLS filter gives a very strong blue cast to all images, flats included. color casts are not really a problem because the flat is applied to the lights on a per-channel basis, that is, you can think of the histogram peaks being normalized (so that the flat looks white) before the flat is applied. however, there can still be an issue with the CLS because the red channel will be very weak. this can lead to low signal in the red channel, which means a noisy red channel in the flats. to combat this you can use a slightly pinkish light source or a slightly pink T-shirt so that the histogram peaks are roughly aligned right out of the camera. also if you let the camera meter the flat it will be underexposed. +2EV is usually a good strategy for DSLR flats.

the dark current in a CMOS sensor is linear in time but exponential in temperature. generally you will find that people make, say 1800s darks and then scale them down for 1200s or 600s lights. but you really should try to match the temperature of the darks to the lights and not rely on scaling. scaling for time is ok due to the time behavior of the dark signal.

since the darks don't actually seem to be your problem, it could be that the lights themselves are. what is the f/ratio of your system and what is the exposure length? the background signal in the R channel might just be so low that it's being destroyed by the read noise of the camera.  is the light ISO also = 800?

rob

[Rod771]

Appreciate your help , Rob 

Looks like I will have to shoot some new flats applying your suggestions. My flats were metered by the camera and will be under exposed.

I agree on the darks, I usually take darks on the same night as my lights but in this case I simply forgot.

The F/ratio of the system is F2.1 and the exposures of the lights are  60 and 90 second subs all @ ISO 800 (see the above BP attachment)

Here is what the image looks like with just the un-calibrated lights stacked and processed. You can see some slight light artifact to the left which is what I'm hoping to clean up.

Cheers  Rod

[pfile]

f/2.1 is indeed very fast but you can probably go longer than 60/90sec. i regularly shoot 20min (1200s) Ha subs thru a 5nm Ha filter with a CCD and in the past i did the same with a modded 50d and 12nm Ha filter, both at f/2.8.

of course the transmission of a CLS filter is much, much broader so 20min is out of the question, but i'll bet you could do 3 minutes without overexposing. of course if you have tracking or guiding issues then shorter is better and at f/2.1 you can get results from short exposures. you may need to experiment with this if in the end you determine that the red channel is attenuated too much in the lights.

you can also just start making darks on cloudy nights and if you have enough overall, you'll probably have enough at the same temperature as the lights you want to calibrate. i used to bin in 5C increments. since you are using BYE it's relatively easy to sort the files since the EXIF temp is in the filename.

one last thing, some canon DSLRs exhibit pretty bad banding noise in the bias frames, and the only way to combat this is to use huge numbers of bias frames. like 100 or 150 (!). also with such short lights it should not be too hard to do master darks with 40-50 frames. seems crazy but it's the only way to get good calibrated results from a DSLR.

rob

[Rod771]

I'm limited to about 90 sec at the moment as I don't have any guiding gear yet.

There is very little red channel data in the flats, so i will do them again with your suggestions.

Here are the histograms for the 60 sec and 90 sec lights and one of the flat frames

Thanks for your help 

[pfile]

i think the lights look okay actually, the histogram is "well detached" in the red channel.

for the flats though, that does look bad. that's DPP right? on one of the panes there is a check box for "linear" - if you hit that you'll see what the linear histogram looks like. the blue channel is probably pretty low too. it's good to get your linear histogram toward the middle of the display in linear mode (assuming a fully overexposed frame shows the histogram at the extreme right - in PI usually you'll see an overexposed DSLR frame at 25% the way over - it's 14 bits of data in a 16-bit word width...)

[Rod771]

Yes it is DPP , well spotted Rob!

Here is the linear flat histogram.

I will definitely re do the flats 

Cheers Rod

[pfile]

oh - a couple of more points about flats.

generally speaking man-made light sources usually have some kind of oscillation frequency associated with them - a fluorescent tube will flicker at 60hz, an EL panel in the 10s of kilohertz range and very likely similar with LED backlights like in your TV.

if your flat exposure is too short - less than a complete cycle of the light source flicker - you can end up with uneven illumination in your flat.

one way to combat this is to lower the ISO for flats, resulting in a longer exposure. in theory all the calibration frames should have the same ISO as your lights, but for flats it probably does not matter as much, due to the way they are used in the calibration flow. don't forget to calibrate your flats with bias frames, or corresponding flat-darks. i'm not sure how this is handled in the BPP script - if there are no matching darks for the flats, it probably tries to scale your main darks rather than using bias frames. further i'm not sure if the BPP script is taking ISO into account - if you shoot the flats at iso100 then the darks or bias should also be at iso100.

since flats are generally short, it's usually enough to calibrate them with a master bias. again with the DSLR you should use a crazy number of bias subs in your master bias.

rob

[Rod771]

I've taken 20 new flats using a pink screen instead of white, to bump the data in the red channel. It has help but I'm still not sure if they're right. The focus should be good as I set up again and slewed to the Helix and focused with BYE (didn't get many new light frames as the clouds were coming in but I just wanted to match focus anyway.)

I have created a master bias consisting of 150 frames and the master dark frame consists of 30 ninety second frames taken with the camera in the fridge to help match the temp to the lights.

I've run the lot through BP again and I'm still "sinking".

I also tried creating a calibrated master flat outside BP following the PI tutorial but that didn't work when stacked with the light frames. The image had red vignetted edges which i couldn't get rid of?

So for now I've been using BP with the master bias , master Dark , individual flats and lights and letting it do its thing.

Below are some shot of the flat statistic and two shots of the image after integration. The image on the right is after integration with the background neutralization applied. The one on the left is after ABE and an Auto stretch with STF. What I'm I doing wrong??? Are the flats still under exposed?

Cheers

Rod

[pfile]

is the screenshot of the flat histogram shown in linear mode? the blue and green channels are kind of hot, if so. you want the peaks at mid histogram (assuming that a completely overexposed frame has all the peaks at the extreme right edge of the histogram display)

[Rod771]

Hi Rob

Yes the histogram graphs are in linear mode

I did take some flats with shorter exposures, here's a shot of one at 1-80 sec. Linear mode.

[MikeOates]

Rod,

I would advise that you do not to use a laptop screen as your light source, as I think this is where your problem lies. The eveness of a screen light is very suspect and will vary greatly from model to model, but in general there is a variation in brightness over the screen, mainly top to bottom. This is not very noticable, but when you stretch the image it becomes more obvious.

I use an EL panel, but at a pinch, shooting an iluminated white card as well as using a teeshirt type screen over the scope may work. Or I suppose putting enough diffuse material over the laptop screen may work, but the easy in which you can take flats with an EL panel makes an imaging session a lot easier and each exposure will be the same from session to session as you don't have to measure the distance from the laptop screen to the scope each time.

A very important point I want to make, if your not already aware of this, is that flats should be taken with the exact same optics, filter, focus, camera etc. as the main (light) images and they should not be moved between taking the lights and the flats. Any such movement can change the position of the dust, one of the very reasons you are taking the flats for to remove. In fact it will then add extra dust marks on your images.

Unless you have a cooled DSLR, there is no harm whatsoever taking darks on another day. This is because the sensor temperature will vary between exposures and thay can vary by about 10degC or more during one session anyway. So I have now stopped taking darks each session and just use one set to span a few weeks, after that the dark current may changed or more likely the ambient temp has changed by a marked amount, so just do another set.

What I do is record the ambient temperature and get a cool box with ice packs in till the air inside gets to the recorded temp and place the DSLR in there with an intervalometer set to replicate an imaging session, i.e. set the time to match the lights and the same interval between exposures and leave it taking photos in the dark cool box while I go to bed or if a long session go to work.

When you do take darks, or even the lights, ensure that the camera is in the dark or well covered as you may find that light gets in via gaps in the case or more likely via the connections. This can lead to patches on your images that may look like amp glow. To test this, with a body cap on the camera, take dark frames with a high ISO of the length you are likely to want to take. Take a few with a torch pointing at the camera from different angles. I was amazed at how much light was getting in via the connectors. You should also have the eyecup covered as well.

My DSLR (500D with full spectrum mod) has lots lost of bits of black gaffer tape all over it, but I no longer get light leaks!

Mike

[Rod771]

Hi Mike

Thanks for the tips  . I appreciate it.

Is this the type of Light panel your referring to, click here

I have tested the 60Da for light leaks as per your procedure and I'm please to say it has none. Great idea, thanks, its good to rule out possible problems .