PI Batch Pre Processing Rejects Galaxies

[timtrice]

In processing 62 300s exposures of NGC 7331 using Batch PreProcessing script, PI rejects my galaxies.

The first go-round rejected 20 images though when trying to copy the message from the Process Explorer I hit Ctrl+X instead of Ctrl+C - PI shut down. The first go-round was using pretty much the default settings.

The second go-round all I did was change the star alignment threshold to 0 which I understand is Auto. This time only 3 images were rejected. But my galaxies were still rejected.

The processed fit file is here: https://www.dropbox.com/s/i665kpf1w2xipjo/light-BINNING_1.fit

Can someone please explain what I am doing wrong here?

[jkmorse]

Tim,

Can you provide a bit more detail.  Where you calibrating and integrating both or just trying to I integrate the frames?  If calibrating, did you use calibration frames made with PI (note it does not like using darks, bias and flats made elsewhere).  Also, what types of frames were you processing, from a monochrome Ccd or from a color dslr?

Also, what exactly do you mean it is "rejecting" the galaxies.  I am not sure I understand since you said only three frames of your 62 frame set were rejected.

Jim

[timtrice]

It's when using the batch script. The images that were calibratred, registered and aligned show the galaxies so my presumption is the rejection is coming during the integration portion of the script. When I say galaxy rejection, I mean the galaxies are being rejected as if they're hot pixels. But in the Process Explorer, there was some RANSAC error insenuating to me anyway that it couldn't align - it said only 39 of the 62 were processed the first time. The second time when I dropped the star threshold to 0 it did 59 of 62 images.

I shifted up my sigma settings and am running the integration tool only this time using the calibrated files. All the calibration files were processed through PI.

Th

[timtrice]

This is my rejection high map w/ a sigma high setting of 4.7 - the highest I've used on three attempts... practically same output.

[naavis]

Well, the rejection algorithm obviously rejects the galaxies if part of the images failed to be registered. In the non-aligned frames the galaxies reside in a completely different location than in the rest of the images, which causes the algorithm to interpret them as outliers. Either register the images properly or leave the unregistered images out of the stack.

[timtrice]

I was finally able to get the galaxies to stick. But on that image and now this one attached I have these nasty horizontal lines. Someone elsewhere pointed to the cosmetic correction to remove vertical banding. However, I did that both times and this banding is still present. How can I remove this? Please don't say I have to rerun the script as it took 14 hours!

[pfile]

what kind of camera are you using?

rob

[Carlos Milovic]

There is something really wrong with this image. The vertical lines are not even vertical! They have an inclination. Were the subs rotated? Are you sure you used the right calibration frames?
I tried to remove the lines with a notch filter, but this pattern is just too wild.

[timtrice]

Canon 60D.

The flats were taken around 7:40 CDT aiming towards the north. Sky flats. Camera on  AV, BYEOS set to AV-Flat. 50 exposures around ISO 200, TV just says 8 so thinking 1/8. I shot my lights afterwards. Darks then 19, same TV, ISO. Then 50 bias, 1/8000 TV 500 ISO.

I get this banding all the time w/ new calibration files. I thought it was my camera; I tried the Canon banding script and flipped the image but that didn't work. My camera is never rotated start to finish. In fact, I use pliers to lock the thumbscrews as they're too small for my fingers to get in and tighten manually. But of course I make sure not to overtighten.

[Carlos Milovic]

Bias frames should have the same ISO of your lights and darks. Since your flats are at a different ISO, take also bias for those flats. 50 may not be enough for this camera, so go for 100.
If you are using an ISO for the lights, be consistant and take the flats, darks and bias at the same value, of things are going to be screwed. Also, double check that you are using bias frames as bias, darks as darks, etc.

Is not the camera that is rotated, it is field rotation due to polar misaligment. So, for normal DSO pictures, you'll have a small amount of rotation between frames. Did you use the first or last frame as the reference?
Anyway, these vertical lines are very wierd, and should not be that consistant unless there is something very wrong with the bias frames. Could you upload your master calibration frames and one uncalibrated light?
(I assume that you always shooted in raw format... if not, there is your answer).

[pfile]

it would be interesting to see one calibrated sub. canon cameras are notorious for banding...

is it a multi-night project? if the camera's position angle is not the same from night to night, then the banding is superposed at odd angles which are no longer perpendicular to either axis of the integrated frame, and thus impossible to remove with the script.

the other possibility is that you're getting some field rotation from inaccurate polar alignment, which could have the same effect even in a single-night project.

personally i push to +2EV from what the camera meters on flats. the camera meters the exposure assuming it will be stretched, but of course a flat is never stretched and there's a lot of well depth left for a longer exposure.

how long were the light subexposures? the darks should match the light sub exposure, unless you are talking about flat-darks. finally what was the ISO for the lights? the bias, dark and light ISO should definitely match. the flat ISO does not need to be matched to the lights but i would calibrate the flats with matching ISO flat-darks or matching ISO bias frames.

rob

edit: carlos beat me to it!

[Carlos Milovic]

Also, there might be a problem with your RAW format preferences. Enable "Create RAW Bayer CFA image" and "No black point correction".

[timtrice]

The settings are correct for my camera; I even double checked those yesterday to make sure.

I have never taken flats at the same ISO as my other calibration/light files. I wasn't aware that was a requirement. For the program I use, I just set the camera to AV and the program (Backyard EOS) to AV-Flat. In this instance I just pointed at the early dusk sky and ran off about 50. I figured that'd be plenty to even out and PI could negate any stars I may catch though I don't think it went that far.

My mount was off alignment a bit. This is a major issue I've had as my mount sits in the grass. Each night it settles into the ground a bit more and I used to adjust but since I got my autoguider, I stopped. But these images were rotated some when stacked.

As it turns out anyway, I'm going to have to toss this data. I discovered my scope was improperly aligned (my fault) and I thought I had collimated it better. I made it worse. I just spent about four hours trying to get it as close to perfection as possible (along w/ recording PEC which failed - another story).

And lastly I am saving in RAW format, no camera processing.

I'll see if I can shoot some flat darks tomorrow for some other images I have with the same issue. The one attached above took 14 hours for PI to process - no joke - and I'm not sitting through that again.

BIAS: https://www.dropbox.com/s/8d0ay690qw5eibh/bias-BINNING_1.fit

FLAT: https://www.dropbox.com/s/ot3qfjoowz4i7o8/flat-BINNING_1.fit

MASTER DARK: https://www.dropbox.com/s/yapdngv35weapil/MASTER%20DARK%20-%2012pos%20-%2019frames%20-%2060s%20-%20ISO%20400%20-%2025c.fit

LIGHT: https://www.dropbox.com/s/jianbrqqgzgkis4/LIGHT_NGC%20225_60_400_%2B24c_00772_20131007-21h06m01s272ms.CR2

[pfile]

if you are using BatchPreProcessing carlos' advice about consistent ISO is good - i don't think BPP is smart enough to match ISO but i could be wrong. i don't use BPP, and generally speaking i calibrate the flat subs with a master bias that matches the ISO, then when calibrating the lights i do not tick "calibrate" on the flat section. i geuss based on your filenames you are using BPP so carlos' advice is spot on.

it may have taken forever because of reading CR2 files directly in ImageIntegration. your best bet is to convert any CR2s that don't need calibration (like your bias files) to FITS using the script called "BatchFormatConversion". i think BFC has forgotten about format hints, so be sure to set your DSLR_RAW reader to "create raw bayer CFA image" and "no black point correction". (this is equivalent to the format hints "raw cfa").

rob

[MikeOates]

I just want to add some more advice about images taken on different nights, this being even more critical for DSLRs than CCDs due to the banding issue.

Each nights images should have their own set of flats, it does not matter so much for bias and darks as they don’t change much over a short period of time, (weeks/months) But the flats change each time you re-focus, change filter, remove camera etc. This is not so much for the effects of vignetting, but dust on filters, sensor, camera window etc. Dust easily moves and if you don’t have flats taken just after or before the lights, you could well ADD more noise to the image instead of reducing it, i.e. dust motes. (I know, I have done it)

So process each nights subs as a separate batch, but don’t integrate them, well you can just to check the image, but don’t use them in the workflow. Once you have all the nights calibrated and aligned subs, this is the point when you integrate them all in one go. This will avoid that angled banding you have.

I found that the vertical banding is removed only after rotating the image 90 degrees, running the script to reduce canon banding, then rotate the image back again. This is done on the final integrated image. But (and I mention this in another thread [http://pixinsight.com/forum/index.php?topic=6100.0]) if you take lots of bias frames, (I did 150) the banding was no longer present.

Mike