Using calibration masters generated in CCDStack 2

[Juan Conejero]

Hi Steve,

Welcome to PixInsight Forum and thanks for your help with this issue.

As Sander has pointed out, master calibration frames are application-specific in general. If you provide master frames generated with software X, then your users that don't use software X may have problems to calibrate their images. For example, to use PixInsight's image calibration system, the master bias, dark and flat frames must be generated with PixInsight's ImageCalibration and ImageIntegration tools.

If instead of providing master frames you'd provide individual calibration frames, then all of your users would have the freedom to calibrate their data with the tool(s) of their choice, and there would be no compatibility problems. This is the easiest solution: just provide the raw data.

I have downloaded one of the zip files that you've linked in your post. It contains raw bias and dark frames which are perfect for calibration with PixInsight and other applications, such as DeepSkyStacker. If you can provide also individual (raw) flat frames in the same conditions, then everything should be fine. Why don't you think this is possible? Is this a problem with limited storage space or bandwidth?

[lightbuckets]

Hi Juan-

We'd certainly like our customers to be able to use PixInsight.  Your software is probably the most powerful astro image processing product on the market so it is important that we figure this out.  There are a few issues with providing the individual calibration frames from LightBuckets.

The first is that we have 3 observatories (with hopefully more on the way).  We have to provide dark and bias frames at 1x1, 2x2, and 3x3 binning.  For our dark masters, we use fifteen 1800s darks that are median combined.  For our bias masters, we use thirty to thirty-five bias frames that are median combined.  Each observatory would have 135 individual calibration frames (45 darks and 90 bias), times 3 observatories is 305 calibration frames to upload.  The full data sets are probably 2-3GB so they will take a while to get to our servers in Dallas, TX.  This is definitely not a show stopper though.

The second issue is that customers get their data from a private account area.  In order to do this correctly, we would have to keep many versions of these individual calibration sets on our servers and properly link them to specific customer runs.  Again, not insurmountable but it will require a fair amount of new code.  We could do what I have just done and simply always have the latest version on our servers and customers that need them can just download them from a link we provide.  However, I know that the first issue we're going to run in to if we do this is someone trying to use the latest individual files with data that is six months old and they're going to want to have access to the older calibration sets.  So it is better to do it right.

The third issue is with flats.  This is the biggest one.  We take custom flats for customers for every run they do (more or less).  We have to do this since we support instrument rotators on our telescopes.  We take 9 flats for each filter used in each customer run.  These are currently median combined in to a flat master.  It is not unusual to have a queue of anywhere from 10 to 40 flat sets on an observatory at any one time, especially if we've had a run of good weather and haven't been able to do EL panel flats in the dome (we only do these at night with the dome closed and they take a long time to do...so we have to wait for nights with bad weather).  Let's say there are 25 flat runs to get done and each run has 3 filter sets.  That is 675 individual flat files that would be generated.  For the telescope with the 4k x 4k camera (LB-0003) that would generate approximately 22GB of data.  At 1.5MbPS with the T-1 line, that is 33 hours of continuous data transfer just for that one observatory.  Let's say the other two generate some proportionally smaller set of data maybe 15 hours for LB-0002 and 10 hours for LB-0001.  All totaled, just to clear out our flat queue and upload the individual flat files, we're looking at over 50 hours of transfer time.  And that is if 100% of the bandwidth goes to flat transfer.  This doesn't account for the fact that customer runs will be going during the 2.5 days that the transfer is happening and now customers are going to be getting their data much, much later than they would have.  A whole new issue.

There's more though as we would, again, have to write a bunch of new code to link these individual flats to the customer account.

At the end of the day, it is all doable.  It is non-trivial to implement as there would be a lot of new code written to support it.  Our plates are already pretty full.  That said, I have asked my engineer, Alvin Jeng, to take a look at this thread and let me know what he thinks is possible.  Let's see what he has to say, maybe he'll tell me he can have it done in an hour (I doubt it...but you never know with engineers!).

Cheers,

Steve

[Juan Conejero]

Hi Steve,

I see. Certainly you have a data transfer issue. However, your numbers are not taking data compression into account. Appropriate compression can reduce the effective volume of your data significantly. I strongly recommend 7-zip compression for these tasks. For example, with one of your raw dark frames:

Original file size: 8 MB (2048x2048 16-bit image)
Zip compressed: 4.1 MB
7Zip compressed: 3.5 MB

So we get a 56% size reduction with default 7zip parameters (more aggressive compression would provide even better results) for a dark frame, which is basically random data (thermal noise) and hence has very little redundancy. You can expect even higher compression ratios for flat frames. It's a matter of trying it out doing some tests with real data. I think this improves things a little doesn't it

[mmirot]

Hi Steve,


I agree with you Steve. In most cases masters should work fine.  
There should be a way to use them in all programs.
This is made easier if the output is maintained in 16 bit unsigned.
MaxIm and others may convert the files to 32 floating. I keep them in 16 bit from MaxIm's calibration
Also, you have to know if was bias/dark subtracted.

The other compatible issues are changes is the fits header. These can be user edited as you already know.

In rare cases someone might need the individual files. Encourage them to test their results against yours. The S/N of your masters should be adaquate and they will stop asking. Perhaps they found a little edge to it which you can learn from. More likely them are being a bit of the control freak. Which we all suffer from until we find out it does not matter.  Obviously PI does nice job. I have found trying use the masters in Maxim tricky because of the fit header changes such as exposure time .

Max



** I have one recommendation use a rejection combine ( sigma etc ) rather than median for all calibration frames.
There is significant S/N increase with a median combine.

[mmirot]

Juan,

What is the issue ? 32 IEEE?

Max

[Nocturnal]

Hi Max,

surely you're not really suggesting storing master files in 16b format? Average values of a series of 16 integers are likely floating point numbers. You wouldn't want to throw away the fractions.

[Juan Conejero]

Hi Max,

What is the issue ? 32 IEEE?

Floating point FITS make things harder, that's true.

However, the real problem is that you can only calibrate with PixInsight (or DSS or any other application) if you have the raw calibration frames. Master frame generation is crucial for the quality of the whole calibration process. For example, we are about to release a new script that implements a multiscale image integration algorithm (see these processing notes). If you don't have the raw data, you cannot apply these techniques to generate your masters (this is particularly important in the case of sky flats).

This is made easier if the output is maintained in 16 bit unsigned.

If a master calibration frame is the result of averaging a set of individual calibration frames, then it must be generated and stored in 32-bit floating point format. Only if a median combine operation is used, then the result is a 16-bit integer image (strictly, only if an odd number of frames is combined, but telling this is pedantic). However, median combination is a bad choice in general because it does a poor job to improve SNR (I fully agree with what you say at the end of your post).

** I have one recommendation use a rejection combine ( sigma etc ) rather than median for all calibration frames.
There is significant S/N increase with a median combine.

I agree completely; see what I say above.

[sreilly]

It seems to boil down to this, if I'm hearing this correctly. There are two camps on 32 bit IEEE. PI and DeepSkyStacker seem to use one format each can read while CCDSoft, MaxImDL, CCDStack, and AIP4Win use the second. I'm sure there are other programs out there that use 32 bit IEEE but I don't have them and no knowledge of how they deal with it. As best I can tell, no one reads both. For me, locally getting my data from my observatory leaves me with two options. The raw data is acquired and saved in 32 bit IEEE Floating Point using MaxImDl. I use MaxIm mainly because my automation software (ACP) requires it. Some other programs can use CCDSoft and MaxIm but they both use the same convention for 32 bit IEEE Floating Point so that's no issue.

Having the raw data on our local machines allows us to do several things. We can ignore the calibration process in PI and continue to calibrate as we did prior to PI's calibration module or we can create masters using PI and store them separately so they don't get mixed with the ones we may have made in MaxIm as they aren't compatible. Darks made in PI would need to be named appropriately as the temperature and time is not in PI's master dark fits header.

The whole issue would be far simpler if each program could read both versions but this doesn't seem to be something anyone is willing to do. So much for standards or the lack thereof. If this isn't going to happen, what are the chances a conversion script can be written? It seems to me to be somewhat unfair to create an ongoing increase in work for the remote observatories to have to supply this because a difference in the 32 bit IEEE Floating Point issue. That's a huge amount of bandwidth that needs to be generated not to mention storage.

This, of course, is just my opinion. I haven't used any remote observing sites but if I did, I'd use the software I have to calibrate and then let PI do it's thing. After all, it wasn't long ago we were doing just that. I expect most imagers have software they can use to calibrate with.

The other issue is whether there is any advantage of calibrating in PI over the other programs. If someone is using LB or any other remote telescope service and has MaxIm, CCDSoft, or CCStack already why not calibrate in one of those programs and do the rest of the processing in PI?

[mmirot]

Steve, It is not helpful to acquire anything in 32 bit format unless you have a camera that has that kind of bit depth.  Unless your just fond of larger files. 

Sanders you are right 32 bit is the way to go. I forgot about the increase in dynamic range from stacking even with calbration frames.

I started going back to 16 bit files when taking calbrated files into PI because of compatible issues. I am lossing accurate data via truncation. Not good 

That said,  there are two issues

1 Can use a Master created in one program in another program ?

2 Are their times when I need the raw data ?

Issue 1 should be  possible.  No?

Issue 2 is more can I use the data more effectively in PI or or program X to create a better master.
This is a function the software and the settings used.

I am really mostly interest in the details on issue 1. 

(Personally, I am not sure a master dark or bias using a sigma good combine will be so different between programs. 

It is not that fancy no weights, normalization, etc.  What am I missing?)

Max

[Juan Conejero]

The whole issue would be far simpler if each program could read both versions but this doesn't seem to be something anyone is willing to do.

This is not quite accurate. The 32-bit thing is not the real issue here. We have flexible FITS and TIFF format modules able to read floating point data referred to any numeric range. For example, it is known that many floating point FITS images written by other applications store real values in the range [0,65535]. You can specify this as the input range in FITS Format Preferences and those files load perfectly in PI. Note that, AFAIK, no other application can do anything similar, so we win here in terms of flexibility --please correct me if I am wrong

So although storing raw CCD data in floating point format is a conceptual error, we do support it. We have even adapted our FITS format module to support applications that store physical raw CCD data as signed integers where the negative half is unused, so we can load their 15-bit 'raw' frames without problems.

There are other cases where we simply can't do anything to be compatible but just rescaling the data (hence losing any physical significance). This is the case of some 32-bit integer images where 16-bit data are stored within an infinitesimal (and undocumented) range, lost in the immensity of 2³² discrete values.

Sincerely, I think we can't do much more to be more flexible and compatible. If the next step consists of dropping our own identity to emulate what other raw data processing applications are doing then sorry but that won't happen.

The other issue is whether there is any advantage of calibrating in PI over the other programs. If someone is using LB or any other remote telescope service and has MaxIm, CCDSoft, or CCStack already why not calibrate in one of those programs and do the rest of the processing in PI?

My opinion is biased, of course, but yes, there are advantages. And there will be more in the near future. It may seem we are the losing horse in this game (it seems so since at least 2004) but you see, we're still around

[mmirot]

I thought Maxim and ccdsoft download raw images in 16 bit unsigned.

[sreilly]

You are correct. I should have said after calibrating the images I save in 32 bit floating point and that probably isn't really necessary until  they have been combined.

[RBA]

Steve is already producing master calibration files with CCDStack from what I understand...?

I don't know if generating those masters is a manual process, but if it is, how about someone shows Steve how to do the same with PixInsight, so he can offer master calibration files generated with PixInsight??

That's not a complete cure, but it should be a fix for users who wish to use PixInsight to calibrate their files.

Cheers,
Rogelio

[Juan Conejero]

If I've understood it well, he is generating the masters remotely with Maxim DL. I don't know if this is a completely automated process but I think this routine is part of their custom observatory control software.

Certainly PI could be driven remotely (via communication with a script) to perform these tasks, although as you say this would be only a partial solution.

[lightbuckets]

Juan is correct, we are using Maxim DL since we use ACP to control our observatories (mostly...lots of custom stuff too).  Calibration masters are automatically generated using Maxim DL scripting.  Rogelio's suggestion of creating masters from Maxim and PI is something we're looking at since it doesn't look like the software vendors are going to solve this for their respective customers.  It is still a pain in the backside as we'll now have to deliver Maxim and PI calibration sets to each customer account.  It beats the alternative of dealing with the individual files since that would involve mountains of new code and likely be cost prohibitive for LightBuckets to fund.  Basically, we're going with the lesser of two evils and not really solving the root cause of the problem...such is life in the software world...