PixInsight Forum (historical)
PixInsight => General => Off-topic => Topic started by: dayers on 2011 March 12 15:30:19
-
Hello everyone - I'm a new guy on the block giving PI a trial on the Ubuntu Linux platform. Starting with a set of raw .fts light files and their Bias, Dark, and Flat masters, when I calibrate my images using the PI calibration tool, I get an ugly set of calibrated files that I cannot use.
If I calibrate first in CCDStack and then register and integrate in PI, the results are very gratifying, even using mostly the default settings. I'm impressed.
Is there a problem with the PI calibration tool, or should I keep trying?
Dave
-
The problem is that master frames are typically application specific. So you can't use master frames from DSS in PI and vice versa. This is one of the reason I'm no longer using Light Buckets as they give you master frames that are not compatible with DSS nor PI.
I recommend creating new master frames with PI and trying calibration that way.
-
Yep. I am using LightBuckets. Thanks for the quick reply! I think I'll probably stick with LB for now and calibrate using CCDStack as usual, then switch to PI while I'm learning my way around. Or, better, maybe LB will supply me with the raw calibration data so I can put together the masters in PI. I'm not going to hold my breath on that one, though.
I'm finding the PI learning curve a bit steep, but the tutorials are a big help. I've only been at it for several days and I hope that in a week or so I'll be more comfortable using it.
Dave
-
I asked for the raw frames and didn't get them. Perhaps you have better luck.
-
Yeah Steve balked at providing the bias/dark/flat subs. Kind of sucks. He released some free uncalibrated data today and sure enough I can't calibrate it with imagecalibration. I think I might be able to calibrate it with pixel math. Pain in the ass. I just can't figure out how maxim is mangling these calibration masters.
-
Well it's his loss. I was ready to spend some serious money on LB but when I spent $100 for basically pointless data I gave up on it and started building out my own observatory further.
-
yeah i agree. i've talked to him in person and he's pretty reasonable, but i guess he feels its too much data to schlep around.
i've been thru this before, and there's some threads here regarding LB data and maxim data in general. right now i just noticed that if i load the master bias and dark with the range set to [<darkest pixel>,65535] then those 2 masters seem to be compatible with ImageCalibration.
the lights seem to be i16 format to begin with so PI is happy there. the master flat is still weird. i have to scale it with the histogram tool or the calibrated file is overcompensated.
-
The thing is that there's no magic here and all it takes is for the file formats to be disclosed. Most importantly we need to know how the master flat is constructed. Is bias (or dark flat) already subtracted? Was it already normalized? Once that is known you can either adjust the file read process or process the file in pixelmath. Frankly it's too much trouble. I'd pursue it if I was desperate to make my LB data work but I'm not.
-
the master dark given is definitely not bias subtracted; i can see the bias noise and readout noise in the dark. it's much harder to tell with the flat. i've never made master calibration frames with maxim so i don't know what's the most likely or default case.
i still have a couple of hundred bucks on account there, so i have some motivation to figure this out. of course when everything goes right on their end, the lights are calibrated for you. in the past when their calibration automation has failed i've asked steve or alvin to fix it for me, and they've always done it.
-
Just a suggestion and I'm not sure it will work but open the calibration frames in PI and then re-save them in PI. It is probably the issue with the ADU count that I referred to in an earlier post. PI is looking for values between 0-1 while most other programs are giving ADU values for 16 bit as 0-65,535. This would go along with the post that the frames aren't working in PI or DSS.
-
oh that's definitely the problem, the question is where the black point should be. if you just rescale them using [0,65535] the pixels in the dark and bias are way too bright and destroy the light. so i've been rescaling them with the first non-black pixel at 0 and that seems to work much better. no idea if it is correct, however.
p.s. the flats from lightbuckets appear to be fully calibrated; the fits header says they have been dark flat subtracted. even so the ADU levels there are whacked as well and i've always had to push the histogram to the right in order for them to work. this is really bad because it is completely arbitrary.
-
Interesting discussion. I plan to email Steve Cullen to find out the reason for his reluctance to give us the calibration subs. I'm guessing that it has to do with how they administer their calibration file library. Although I've had generally good results using their master files, I don't know how to evaluate my results except by visual inspection of the calibrated frames.
I hope that the upshot of this thread will be to either find an acceptable workaround using LB masters or that Steve will see that it may be in his best interests to be a little more cooperative.
-
It could be he has a financial incentive to lock his customers into Maxim/CCDStack. It seems unlikely but it points that way. Distributing calibration frames would not generate more data than a percentage of traffic generated by the light frames especially since the calibration frames don't get updated that often. But whatever, it's their prerogative to be as flexible or restrictive as they like, for whatever reason they choose. Just like it's our reason to judge their service based on these factors.
-
Hi All-
Sorry for the interruption but I thought I'd jump in here and share some information and clear up some of the information.
First off, we receive zero financial incentive from either Cyanogen (Maxim DL) or CCDWare (CCDStack). In fact, I have frequently asked both companies (Doug George at Cyanogen and John Smith at CCDWare) to just simply offer some kind of discounts for LightBuckets customers (with absolutely nothing to gain for LightBuckets) and have received no response or action.
Second, I believe I have had one, possibly two requests to make the individual calibration sets available for download. You have to understand that LightBuckets consists of me plus a contract/part-time engineer (Alvin). While we try to get to things as quickly as possible it does take some time to do what we do with the limited resources we have available. And, I don't write code so it is all on Alvin. Without automation, there is no way that we can commit to delivering the individual calibration frames since it is a manual process to package them up and put them on the server whenever we generate new masters. That said, I have asked Alvin to look in to what it would take to create zip packages of the individual master sets (darks and bias) and put them on the server. We are in the process of revamping our dark and bias automation so that we are more constantly refreshing the master so now is a good time to look at this. The files likely won't be in your "Account" area, they will be on our servers where we'll have to provide a link to them. They also likely won't be versioned so what will be on the server will be the latest and greatest sets, you'll have to make sure to retrieve them with some frequency and do versioning locally. I doubt that we will be able to do this with flats since they are all different and generated with relatively high frequency. Again, Alvin is looking in to what we can do, and we will do as much as possible to make sure you can use the data you get from LightBuckets.
Third, I have reached out to the PixInsight folks to see if they might be able to help us solve the core issue that you are experiencing with the calibration masters generated with Maxim DL. I'm not certain that I understand what the problem is so if you guys can use this thread to help us map out what you're seeing, that would greatly help. I have asked them to take a look at this thread and then get back to me with how we can work together to resolve it.
And, finally, I am in the process of uploading the dark and bias sets for each LightBuckets observatory to our server as we speak. You will find them at...
LB-0001 - http://www.lightbuckets.com/data/LB-0001_calibration.zip (http://www.lightbuckets.com/data/LB-0001_calibration.zip) (136MB compressed)
LB-0002 - http://www.lightbuckets.com/data/LB-0002_calibration.zip (http://www.lightbuckets.com/data/LB-0002_calibration.zip) (544MB compressed)
LB-0003 - http://www.lightbuckets.com/data/LB-0003_calibration.zip (http://www.lightbuckets.com/data/LB-0003_calibration.zip) (958MB compressed)
Some of the sets (darks mostly) are incomplete as we are in the middle of a refresh cycle which, as you may know, takes 3+ full nights to complete (plus we're in the middle of revamping the entire calibration frame automation so that has slowed us down a bit). These will take a bit to get to our servers in Dallas so give it a little time.
For those of you who know me, delivering a great experience, outstanding data, and excellent customer service are what we stand for. If you ever have an issue, question, or concern, all you have to do is fire off an email to me. I can't promise that we'll do everything tomorrow, but we are listening and running as fast as we can!
Cheers,
Steve
-
Thanks Steve for jumping in here. It would be great if LB could be used by people who don't use Maxim/CCDStack.
-
Hi Steve,
Welcome to PixInsight Forum and thanks for your help with this issue.
As Sander has pointed out, master calibration frames are application-specific in general. If you provide master frames generated with software X, then your users that don't use software X may have problems to calibrate their images. For example, to use PixInsight's image calibration system, the master bias, dark and flat frames must be generated with PixInsight's ImageCalibration and ImageIntegration tools.
If instead of providing master frames you'd provide individual calibration frames, then all of your users would have the freedom to calibrate their data with the tool(s) of their choice, and there would be no compatibility problems. This is the easiest solution: just provide the raw data.
I have downloaded one of the zip files that you've linked in your post. It contains raw bias and dark frames which are perfect for calibration with PixInsight and other applications, such as DeepSkyStacker. If you can provide also individual (raw) flat frames in the same conditions, then everything should be fine. Why don't you think this is possible? Is this a problem with limited storage space or bandwidth?
-
Hi Juan-
We'd certainly like our customers to be able to use PixInsight. Your software is probably the most powerful astro image processing product on the market so it is important that we figure this out. There are a few issues with providing the individual calibration frames from LightBuckets.
The first is that we have 3 observatories (with hopefully more on the way). We have to provide dark and bias frames at 1x1, 2x2, and 3x3 binning. For our dark masters, we use fifteen 1800s darks that are median combined. For our bias masters, we use thirty to thirty-five bias frames that are median combined. Each observatory would have 135 individual calibration frames (45 darks and 90 bias), times 3 observatories is 305 calibration frames to upload. The full data sets are probably 2-3GB so they will take a while to get to our servers in Dallas, TX. This is definitely not a show stopper though.
The second issue is that customers get their data from a private account area. In order to do this correctly, we would have to keep many versions of these individual calibration sets on our servers and properly link them to specific customer runs. Again, not insurmountable but it will require a fair amount of new code. We could do what I have just done and simply always have the latest version on our servers and customers that need them can just download them from a link we provide. However, I know that the first issue we're going to run in to if we do this is someone trying to use the latest individual files with data that is six months old and they're going to want to have access to the older calibration sets. So it is better to do it right.
The third issue is with flats. This is the biggest one. We take custom flats for customers for every run they do (more or less). We have to do this since we support instrument rotators on our telescopes. We take 9 flats for each filter used in each customer run. These are currently median combined in to a flat master. It is not unusual to have a queue of anywhere from 10 to 40 flat sets on an observatory at any one time, especially if we've had a run of good weather and haven't been able to do EL panel flats in the dome (we only do these at night with the dome closed and they take a long time to do...so we have to wait for nights with bad weather). Let's say there are 25 flat runs to get done and each run has 3 filter sets. That is 675 individual flat files that would be generated. For the telescope with the 4k x 4k camera (LB-0003) that would generate approximately 22GB of data. At 1.5MbPS with the T-1 line, that is 33 hours of continuous data transfer just for that one observatory. Let's say the other two generate some proportionally smaller set of data maybe 15 hours for LB-0002 and 10 hours for LB-0001. All totaled, just to clear out our flat queue and upload the individual flat files, we're looking at over 50 hours of transfer time. And that is if 100% of the bandwidth goes to flat transfer. This doesn't account for the fact that customer runs will be going during the 2.5 days that the transfer is happening and now customers are going to be getting their data much, much later than they would have. A whole new issue.
There's more though as we would, again, have to write a bunch of new code to link these individual flats to the customer account.
At the end of the day, it is all doable. It is non-trivial to implement as there would be a lot of new code written to support it. Our plates are already pretty full. That said, I have asked my engineer, Alvin Jeng, to take a look at this thread and let me know what he thinks is possible. Let's see what he has to say, maybe he'll tell me he can have it done in an hour (I doubt it...but you never know with engineers!).
Cheers,
Steve
-
Hi Steve,
I see. Certainly you have a data transfer issue. However, your numbers are not taking data compression into account. Appropriate compression can reduce the effective volume of your data significantly. I strongly recommend 7-zip compression for these tasks. For example, with one of your raw dark frames:
Original file size: 8 MB (2048x2048 16-bit image)
Zip compressed: 4.1 MB
7Zip compressed: 3.5 MB
So we get a 56% size reduction with default 7zip parameters (more aggressive compression would provide even better results) for a dark frame, which is basically random data (thermal noise) and hence has very little redundancy. You can expect even higher compression ratios for flat frames. It's a matter of trying it out doing some tests with real data. I think this improves things a little doesn't it ;)
-
Hi Steve,
I agree with you Steve. In most cases masters should work fine.
There should be a way to use them in all programs.
This is made easier if the output is maintained in 16 bit unsigned.
MaxIm and others may convert the files to 32 floating. I keep them in 16 bit from MaxIm's calibration
Also, you have to know if was bias/dark subtracted.
The other compatible issues are changes is the fits header. These can be user edited as you already know.
In rare cases someone might need the individual files. Encourage them to test their results against yours. The S/N of your masters should be adaquate and they will stop asking. Perhaps they found a little edge to it which you can learn from. More likely them are being a bit of the control freak. Which we all suffer from until we find out it does not matter. Obviously PI does nice job. I have found trying use the masters in Maxim tricky because of the fit header changes such as exposure time .
Max
** I have one recommendation use a rejection combine ( sigma etc ) rather than median for all calibration frames.
There is significant S/N increase with a median combine.
-
Juan,
What is the issue ? 32 IEEE?
Max
-
Hi Max,
surely you're not really suggesting storing master files in 16b format? Average values of a series of 16 integers are likely floating point numbers. You wouldn't want to throw away the fractions.
-
Hi Max,
What is the issue ? 32 IEEE?
Floating point FITS make things harder, that's true.
However, the real problem is that you can only calibrate with PixInsight (or DSS or any other application) if you have the raw calibration frames. Master frame generation is crucial for the quality of the whole calibration process. For example, we are about to release a new script that implements a multiscale image integration algorithm (see these processing notes (http://pixinsight.com/examples/NGC6914-CAHA/en.html)). If you don't have the raw data, you cannot apply these techniques to generate your masters (this is particularly important in the case of sky flats).
This is made easier if the output is maintained in 16 bit unsigned.
If a master calibration frame is the result of averaging a set of individual calibration frames, then it must be generated and stored in 32-bit floating point format. Only if a median combine operation is used, then the result is a 16-bit integer image (strictly, only if an odd number of frames is combined, but telling this is pedantic). However, median combination is a bad choice in general because it does a poor job to improve SNR (I fully agree with what you say at the end of your post).
** I have one recommendation use a rejection combine ( sigma etc ) rather than median for all calibration frames.
There is significant S/N increase with a median combine.
I agree completely; see what I say above.
-
It seems to boil down to this, if I'm hearing this correctly. There are two camps on 32 bit IEEE. PI and DeepSkyStacker seem to use one format each can read while CCDSoft, MaxImDL, CCDStack, and AIP4Win use the second. I'm sure there are other programs out there that use 32 bit IEEE but I don't have them and no knowledge of how they deal with it. As best I can tell, no one reads both. For me, locally getting my data from my observatory leaves me with two options. The raw data is acquired and saved in 32 bit IEEE Floating Point using MaxImDl. I use MaxIm mainly because my automation software (ACP) requires it. Some other programs can use CCDSoft and MaxIm but they both use the same convention for 32 bit IEEE Floating Point so that's no issue.
Having the raw data on our local machines allows us to do several things. We can ignore the calibration process in PI and continue to calibrate as we did prior to PI's calibration module or we can create masters using PI and store them separately so they don't get mixed with the ones we may have made in MaxIm as they aren't compatible. Darks made in PI would need to be named appropriately as the temperature and time is not in PI's master dark fits header.
The whole issue would be far simpler if each program could read both versions but this doesn't seem to be something anyone is willing to do. So much for standards or the lack thereof. If this isn't going to happen, what are the chances a conversion script can be written? It seems to me to be somewhat unfair to create an ongoing increase in work for the remote observatories to have to supply this because a difference in the 32 bit IEEE Floating Point issue. That's a huge amount of bandwidth that needs to be generated not to mention storage.
This, of course, is just my opinion. I haven't used any remote observing sites but if I did, I'd use the software I have to calibrate and then let PI do it's thing. After all, it wasn't long ago we were doing just that. I expect most imagers have software they can use to calibrate with.
The other issue is whether there is any advantage of calibrating in PI over the other programs. If someone is using LB or any other remote telescope service and has MaxIm, CCDSoft, or CCStack already why not calibrate in one of those programs and do the rest of the processing in PI?
-
Steve, It is not helpful to acquire anything in 32 bit format unless you have a camera that has that kind of bit depth. Unless your just fond of larger files.
Sanders you are right 32 bit is the way to go. I forgot about the increase in dynamic range from stacking even with calbration frames.
I started going back to 16 bit files when taking calbrated files into PI because of compatible issues. I am lossing accurate data via truncation. Not good :sad:
That said, there are two issues
1 Can use a Master created in one program in another program ?
2 Are their times when I need the raw data ?
Issue 1 should be possible. No?
Issue 2 is more can I use the data more effectively in PI or or program X to create a better master.
This is a function the software and the settings used.
I am really mostly interest in the details on issue 1.
(Personally, I am not sure a master dark or bias using a sigma good combine will be so different between programs.
It is not that fancy no weights, normalization, etc. What am I missing?)
Max
-
The whole issue would be far simpler if each program could read both versions but this doesn't seem to be something anyone is willing to do.
This is not quite accurate. The 32-bit thing is not the real issue here. We have flexible FITS and TIFF format modules able to read floating point data referred to any numeric range. For example, it is known that many floating point FITS images written by other applications store real values in the range [0,65535]. You can specify this as the input range in FITS Format Preferences and those files load perfectly in PI. Note that, AFAIK, no other application can do anything similar, so we win here in terms of flexibility --please correct me if I am wrong
So although storing raw CCD data in floating point format is a conceptual error, we do support it. We have even adapted our FITS format module to support applications that store physical raw CCD data as signed integers where the negative half is unused, so we can load their 15-bit 'raw' frames without problems.
There are other cases where we simply can't do anything to be compatible but just rescaling the data (hence losing any physical significance). This is the case of some 32-bit integer images where 16-bit data are stored within an infinitesimal (and undocumented) range, lost in the immensity of 232 discrete values.
Sincerely, I think we can't do much more to be more flexible and compatible. If the next step consists of dropping our own identity to emulate what other raw data processing applications are doing then sorry but that won't happen.
The other issue is whether there is any advantage of calibrating in PI over the other programs. If someone is using LB or any other remote telescope service and has MaxIm, CCDSoft, or CCStack already why not calibrate in one of those programs and do the rest of the processing in PI?
My opinion is biased, of course, but yes, there are advantages. And there will be more in the near future. It may seem we are the losing horse in this game (it seems so since at least 2004) but you see, we're still around :)
-
I thought Maxim and ccdsoft download raw images in 16 bit unsigned.
-
You are correct. I should have said after calibrating the images I save in 32 bit floating point and that probably isn't really necessary until they have been combined.
-
Steve is already producing master calibration files with CCDStack from what I understand...?
I don't know if generating those masters is a manual process, but if it is, how about someone shows Steve how to do the same with PixInsight, so he can offer master calibration files generated with PixInsight??
That's not a complete cure, but it should be a fix for users who wish to use PixInsight to calibrate their files.
Cheers,
Rogelio
-
If I've understood it well, he is generating the masters remotely with Maxim DL. I don't know if this is a completely automated process but I think this routine is part of their custom observatory control software.
Certainly PI could be driven remotely (via communication with a script) to perform these tasks, although as you say this would be only a partial solution.
-
Juan is correct, we are using Maxim DL since we use ACP to control our observatories (mostly...lots of custom stuff too). Calibration masters are automatically generated using Maxim DL scripting. Rogelio's suggestion of creating masters from Maxim and PI is something we're looking at since it doesn't look like the software vendors are going to solve this for their respective customers. It is still a pain in the backside as we'll now have to deliver Maxim and PI calibration sets to each customer account. It beats the alternative of dealing with the individual files since that would involve mountains of new code and likely be cost prohibitive for LightBuckets to fund. Basically, we're going with the lesser of two evils and not really solving the root cause of the problem...such is life in the software world...