PI Batch Pre Processing Rejects Galaxies

[Phil Leigh]

If you are using AV to shoot flats you need at least +2EV on exposure compensation to get anywhere near the right level - you want the camera histogram as far right as you can get it without clipping. AV mode with no exp comp will expose way too far to the left!

Also, the ISO needs to be the same across flats/bias/darks. The temperature needs to be as close as you can get it too!

If you try and subtract an ISO 250 bias frame from an ISO 500 flat (for example) you will add quite a lot of noise.

[timtrice]

Rob, the BPP script doesn't use the CR2 files in integration but only to start in calibration. Those files are saved as FITS then are used to go thorugh the rest (cosmetized, debayered, registered, aligned, etc...). I may bail on the BPP and take a more manual approach but it's so freaking convenient as I can run it as soon as my session is done and, usually, have something to get to work with in the morning. I'm actually between jobs right now starting a new job Monday so it'll be even harder then. Maybe I'll write my own script.

[pfile]

well the script takes CR2 files, right? obviously ImageCalibration takes CR2s and produces .fits files. however, if you are integrating a master bias frame, ImageIntegration is probably running directly on the CR2 files. without re-reading the source of BPP i can't say that for sure but it seems very likely based on what happened to you.

you said it took 14 hours to run and i'm telling you that's completely abnormal. i happen to know that people have had problems directly using CR2 files in ImageIntegration - this is because PI has to load all of the CR2 files in their entirety to integrate them. your machine ran out of physical memory and spent all that time thrashing on the virtual memory paging file. FITS files can be read 'incrementally' and so PI does not need to load them into memory entirely while running ImageIntegration.

so it is worth trying the batchformatconversion script on your bias frames.

having said all that if you saved your calibration master frames in the script, then the next time you run you can load those masters and this problem will just go away.

rob

[timtrice]

From what I can see it's only during the calibration phase that the CR2's are read then they are converted to FITS. I actually cancelled the script during the final integration after I had all my calibrated files (again, debayer, cosmetic correction, alignment). It was those files which were FITS that I just did the integration on using the PI tutorial from Harry. It still was excruciatingly slow. Towards the end where it starts writing the three files and creating the three or four different - layers for lack of better term - the percentage increases were slower than the minutes passing. But again those were FITS files.

I assumed it was because I had so many lights - 162. The most I've done at one time was 50 which again only took a couple of hours. I cancelled thinking it was because I was out of memory; I rebooted flushing the RAM. Made no difference...

[pfile]

yes except bias frames are not calibrated, so they are not converted to FITS files and ImageIntegration can take them directly. so that's one source of delay.

i forgot that the original topic of this thread was StarAlignment - probably what's going on is that SA is repeatedly failing to register your frames and taking a long time. it tries really hard to register. 162 frames is a lot but i'd be surprised if it takes more than 1-2 minutes for each frame when it is working properly, so no more than 2-3 hours.

if your subs were of poor quality (as you mentioned, collimation was bad, and it's possible that bad calibration frames really destroyed your data) then it's not surprising that SA would have a hard time latching onto real stars. it may have picked up noise thinking they were stars. if you see 10s of thousands of stars detected in the console log while the registration phase is happening, that's probably what's going on.

rob

[timtrice]

The Star Alignment issued was related to my galaxy image I was working on - integration kept spitting them out but I was able to get them to stick going manually through the calibration processes. This thread turned into the banding issue as both the galaxy image and the current image I'm working on have just downright disgusting banding which I'll have to assume is only my calibration images.

I'll up my bias files to around 150 and increase my flat ISO to the same as my darks, bias and lights. I want to build a portable light box - at the present time I'm just doing sky flats which I wonder how much of an effect it has. But, nonetheless, I'm grinding through the banding and hoping I can pull something decent out. Even though the collimation turned out to be horrible for these subs, I'm using this as a learning opportunity to produce something nice.

[Carlos Milovic]

I just inspected your masters and light.

The bias looks fine, but it is still noisy. I would consider taking more than 100 exposures to really clean those vertical lines. You may also try the SuperBias module.

The dark looks pretty much the same as the bias... 60s is a relatively short exposure, so if you don't have too many hot pixels, you may consider discarding them. Also, for that exposure time, you should create at least twice the dark frames as lights you have.

The flats looks way too dark. You even have a clear pattern from the bias in there. Make sure that your flats go near half the depth of your sensor's capability (should read something close to 0.125 in PI's normalized range, for raw frames).

That's it. To reduce the vertical banding, just go through a better capture of your calibration frames. I think that the biggest culprit here is your flat, due the very low signal.

[timtrice]

I went 60 seconds so I could get rid of anything other than perfect as I was trying to catch fine-detailed nebula and didn't want any bouncing around of my mount to disrupt that. As I'll have to do a recapture I'll go 300 seconds as I try to typically do. At 300s, I'll be shooting between 100ISO and 200 ISO (the object moves from an LP-heavy part of the sky to darker area over the 3 hours I aim to capture).

So with that being said, I capture all my calibration files at 200 ISO, 150 bias at 1/8000, 150 flats at whatever TV (I'll have to figure out the ADU today as that's something I've never really done). Can I still stick to 19 darks or even cut it down to 9 (as they'll be five minutes each)?

I'm trying to get NGC 225 - I really love the cluster embedded w/ a little nebulosity and dust clouds. So I'm really trying to get as perfect as possible. I don't expect CCD quality. But I think this is the best object for me to really pull some nice details out (I'm not ready for Orion yet  )

[pfile]

imho i'd do 200 bias. 150 flats are probably not necessary as the goal is to get the flat subs well into the linear range of the sensor. i usually do between 20 and 50 with my CCD. of course, under the assumption that the banding noise is also in the flats, maybe it's not a bad idea to do more with the DSLR.

darks are the real can of worms. i think you're going to need a whole lot more than 9... 

the thermal signal in any frame builds up linearly with time and exponentially with temperature. so if you have a cooled CCD you fix the temperature someplace (say -20C or -30C), and then make darks that are longer than your lights. then there's only one variable - exposure time. naively you can simply scale your dark (after removing the bias) by the ratio of your light to dark exposure times. pixinsight goes further, scaling the dark until the noise in the calibrated frame is minimized.

canon DSLRs have the added complication that for some exposure times < some threshold, the camera firmware plays tricks with the data to try to hide the thermal signal. this means that you can't actually properly scale DSLR darks, and so probably the best strategy is to take darks of the same length as your lights and tell PI not to optimize them.

if your DSLR is not temperature regulated, then you're in the worst of all worlds. in this situation you need to take a look at your lights with exiftool or Dark Library or some other tool that can read the temperature out of the CR2 file. to avoid complete insanity, you can bin your lights into 5C temperature buckets. then you need to take a bunch of darks across different nights and similar temperatures, so that you can make clean master darks for each 5C bin.

so yeah, for best results you need to calibrate different temperature lights with their temperature-matched master darks. a true pain in the butt.

at any rate, that banding noise is also in the master dark, so it probably pays to do as many dark subs as you can stomach. i've gone as far as to make 50 1200s darks with my CCD, letting it run for a couple of nights to get them all.

if you are unable to get rid of the banding through calibration, then there's always Georg's CanonBandingReduction script. this can be applied to a batch of frames using ImageContainer and a process icon containing the script invocation. you need to experiment with the amount and the highlight protection to find out what works best for the target you are imaging. it works really well; i am working on some 2+ year old data which has awful banding patterns and the script has worked miracles on the subs.

you can also apply CBR to an integrated frame but with these multi-night projects with differing camera angles, you're pretty much sunk unless you integrate each night separately, then apply CBR to the sub-integrations and finally integrate each night's integrated result together...


rob

[Carlos Milovic]

As Rob said, there is no need for that many flats.  near 30 should be more than enough, given that you exposed them well.

On a side note, exposures should always be as long as you can. It is better to have 4x10min exposures, than 40x1min exposures. The goal is to get the signal high enough so you are dealing mainly with photon noise, and not camera noise. Then the final SNR will only depend on the total exposure time, not much on the individual subs.

[timtrice]

Well, I did decide to try and process anyway but it's freaking horrid!!! I'll start looking to see how to get properly exposed flats and get to work on that ASAP tonight. I can't leave my camera gear out too late as I don't have dew protection and it starts popping up around midnight; at worst I'll just bring the scope in and set it up inside, capture darks overnight and just start processing in the morning.

[Carlos Milovic]

Try the following with your data:

- Open the master flat, and apply ATrousWaveletTransform disabling the first 4 layers.
- Save as a copy.
- Open the master dark.
- Create an ImageContainer with all your light frames (use files instead of views).
- Apply to it PixelMath with the following equation: "($T-dark)*mean(flat)/flat"
where "dark" is the identifier of your master dark, and flat the identifier of the processed master flat (you may want to rename those views to make things easier).
This should create a whole set of calibrated images, with less artifacts. Now register them with SA and integrate.

Hope this works.

[Carlos Milovic]

Oh, I forgot to menction that you have to debayerize your data before aligning/registering.

[jkmorse]

Tim,

Another suggestion that IMHO may help is to try a different target.  Even for a high end cooled CCD, NGC7331 is a difficult target.  I image with a 12.5 inch CDK scope on an MX Paramount using an Apogee Alta F16M CCD cooled to -30C and it took me a whole night of 600 second exposures to get stacks that produce a nice image of NGC77331, and that is with Flats, Bias and Darks temp matched at -30C (I would post my results but I am on vacation in the US and all my images are back in Dubai). 

I started out in the same place as you, using a Canon 40D with a TeleVue 101is and got some great shots, but they were focused on the Messiers which give you a chance to really build your skills and work with PixInsight on bright targets much better suited to a DSLR. 

Best of luck in working through your processing issues.

Jim

[jkmorse]

Tim,

One more thought, you mentioned in an earlier post that you are not ready for Orion yet, but Orion is the perfect place to try and is extremely well suited to a DSLR.  You will be amazed at what you can do with M42/43 using a DLSR.  And, if you are game, you can even try shooting different sets of shorter duration (e.g., 60 seconds vs 300 seconds) to try your hand at integrating different sets to capture the HDR around the trapezium).  There are some great tutorials available on the PixInsight website that walk you through the whole process.  With winter coming and Orion coming into its own, don't pass it up this winter.

All the best.

Jim