Using calibration masters generated in CCDStack 2

[dayers]

Hello everyone - I'm a new guy on the block giving PI a trial on the Ubuntu Linux platform. Starting with a set of raw .fts light files and their Bias, Dark, and Flat masters, when I calibrate my images using the PI calibration tool, I get an ugly set of calibrated files that I cannot use.

If I calibrate first in CCDStack and then register and integrate in PI, the results are very gratifying, even using mostly the default settings. I'm impressed.

Is there a problem with the PI calibration tool, or should I keep trying?

Dave

[Nocturnal]

The problem is that master frames are typically application specific. So you can't use master frames from DSS in PI and vice versa. This is one of the reason I'm no longer using Light Buckets as they give you master frames that are not compatible with DSS nor PI.

I recommend creating new master frames with PI and trying calibration that way.

[dayers]

Yep. I am using LightBuckets. Thanks for the quick reply! I think I'll probably stick with LB for now and calibrate using CCDStack as usual, then switch to PI while I'm learning my way around. Or, better, maybe LB will supply me with the raw calibration data so I can put together the masters in PI. I'm not going to hold my breath on that one, though.

I'm finding the PI learning curve a bit steep, but the tutorials are a big help. I've only been at it for several days and I hope that in a week or so I'll be more comfortable using it.

Dave

[Nocturnal]

I asked for the raw frames and didn't get them. Perhaps you have better luck.

[pfile]

Yeah Steve balked at providing the bias/dark/flat subs. Kind of sucks. He released some free uncalibrated data today and sure enough I can't calibrate it with imagecalibration. I think I might be able to calibrate it with pixel math. Pain in the ass. I just can't figure out how maxim is mangling these calibration masters.

[Nocturnal]

Well it's his loss. I was ready to spend some serious money on LB but when I spent $100 for basically pointless data I gave up on it and started building out my own observatory further.

[pfile]

yeah i agree. i've talked to him in person and he's pretty reasonable, but i guess he feels its too much data to schlep around.

i've been thru this before, and there's some threads here regarding LB data and maxim data in general. right now i just noticed that if i load the master bias and dark with the range set to [<darkest pixel>,65535] then those 2 masters seem to be compatible with ImageCalibration.
the lights seem to be i16 format to begin with so PI is happy there. the master flat is still weird. i have to scale it with the histogram tool or the calibrated file is overcompensated.

[Nocturnal]

The thing is that there's no magic here and all it takes is for the file formats to be disclosed. Most importantly we need to know how the master flat is constructed. Is bias (or dark flat) already subtracted? Was it already normalized? Once that is known you can either adjust the file read process or process the file in pixelmath. Frankly it's too much trouble. I'd pursue it if I was desperate to make my LB data work but I'm not.

[pfile]

the master dark given is definitely not bias subtracted; i can see the bias noise and readout noise in the dark. it's much harder to tell with the flat. i've never made master calibration frames with maxim so i don't know what's the most likely or default case.

i still have a couple of hundred bucks on account there, so i have some motivation to figure this out. of course when everything goes right on their end, the lights are calibrated for you. in the past when their calibration automation has failed i've asked steve or alvin to fix it for me, and they've always done it.

[sreilly]

Just a suggestion and I'm not sure it will work but open the calibration frames in PI and then re-save them in PI. It is probably the issue with the ADU count that I referred to in an earlier post. PI is looking for values between 0-1 while most other programs are giving ADU values for 16 bit as 0-65,535. This would go along with the post that the frames aren't working in PI or DSS.

[pfile]

oh that's definitely the problem, the question is where the black point should be. if you just rescale them using [0,65535] the pixels in the dark and bias are way too bright and destroy the light. so i've been rescaling them with the first non-black pixel at 0 and that seems to work much better. no idea if it is correct, however.

p.s. the flats from lightbuckets appear to be fully calibrated; the fits header says they have been dark flat subtracted. even so the ADU levels there are whacked as well and i've always had to push the histogram to the right in order for them to work. this is really bad because it is completely arbitrary.

[dayers]

Interesting discussion. I plan to email Steve Cullen to find out the reason for his reluctance to give us the calibration subs. I'm guessing that it has to do with how they administer their calibration file library. Although I've had generally good results using their master files, I don't know how to evaluate my results except by visual inspection of the calibrated frames.

I hope that the upshot of this thread will be to either find an acceptable workaround using LB masters or that Steve will see that it may be in his best interests to be a little more cooperative.

[Nocturnal]

It could be he has a financial incentive to lock his customers into Maxim/CCDStack. It seems unlikely but it points that way. Distributing calibration frames would not generate more data than a percentage of traffic generated by the light frames especially since the calibration frames don't get updated that often. But whatever, it's their prerogative to be as flexible or restrictive as they like, for whatever reason they choose. Just like it's our reason to judge their service based on these factors.

[lightbuckets]

Hi All-

Sorry for the interruption but I thought I'd jump in here and share some information and clear up some of the information. 

First off, we receive zero financial incentive from either Cyanogen (Maxim DL) or CCDWare (CCDStack).  In fact, I have frequently asked both companies (Doug George at Cyanogen and John Smith at CCDWare) to just simply offer some kind of discounts for LightBuckets customers (with absolutely nothing to gain for LightBuckets) and have received no response or action.

Second, I believe I have had one, possibly two requests to make the individual calibration sets available for download.  You have to understand that LightBuckets consists of me plus a contract/part-time engineer (Alvin).  While we try to get to things as quickly as possible it does take some time to do what we do with the limited resources we have available.  And, I don't write code so it is all on Alvin.  Without automation, there is no way that we can commit to delivering the individual calibration frames since it is a manual process to package them up and put them on the server whenever we generate new masters.  That said, I have asked Alvin to look in to what it would take to create zip packages of the individual master sets (darks and bias) and put them on the server.  We are in the process of revamping our dark and bias automation so that we are more constantly refreshing the master so now is a good time to look at this.  The files likely won't be in your "Account" area, they will be on our servers where we'll have to provide a link to them.  They also likely won't be versioned so what will be on the server will be the latest and greatest sets, you'll have to make sure to retrieve them with some frequency and do versioning locally.  I doubt that we will be able to do this with flats since  they are all different and generated with relatively high frequency.  Again, Alvin is looking in to what we can do, and we will do as much as possible to make sure you can use the data you get from LightBuckets.

Third, I have reached out to the PixInsight folks to see if they might be able to help us solve the core issue that you are experiencing with the calibration masters generated with Maxim DL.  I'm not certain that I understand what the problem is so if you guys can use this thread to help us map out what you're seeing, that would greatly help.  I have asked them to take a look at this thread and then get back to me with how we can work together to resolve it.

And, finally, I am in the process of uploading the dark and bias sets for each LightBuckets observatory to our server as we speak.  You will find them at...

LB-0001 - http://www.lightbuckets.com/data/LB-0001_calibration.zip (136MB compressed)
LB-0002 - http://www.lightbuckets.com/data/LB-0002_calibration.zip (544MB compressed)
LB-0003 - http://www.lightbuckets.com/data/LB-0003_calibration.zip (958MB compressed)

Some of the sets (darks mostly) are incomplete as we are in the middle of a refresh cycle which, as you may know, takes 3+ full nights to complete (plus we're in the middle of revamping the entire calibration frame automation so that has slowed us down a bit).  These will take a bit to get to our servers in Dallas so give it a little time.

For those of you who know me, delivering a great experience, outstanding data, and excellent customer service are what we stand for.  If you ever have an issue, question, or concern, all you have to do is fire off an email to me.  I can't promise that we'll do everything tomorrow, but we are listening and running as fast as we can!

Cheers,

Steve

[Nocturnal]

Thanks Steve for jumping in here. It would be great if LB could be used by people who don't use Maxim/CCDStack.