Using dark and flat frames . . .

[Cleon_Wells]

Terry, here's a screen shoot of the BatchFormatConversion/BFC Script. Note that the file extension defaults to .jpg, change file extension to .fit.
I have a question in BFC on which Sample Format to use, the Canon T1i raw file is a 14 bit file which is converted to a 16 bit file when you BFC the raw 14 bit file with the Sample Format  (same as input file) selected. After the Calibration  process the raw files are change to 32 bit, so the question is should we/DSLR folks change the Sample Format to 32 bit when we are converting the Bias subs to .fit for integration?
I have had a problem integrating my Calibrated Dark/MasterBias subtracted subs when creating my MasterDark file. When I use  the Rejection algorithm the MasterDark Cal file gives me large scaling factor numbers, like over 100.0 . If I don't use Rejection during integration of the Cal Dark subs the MasterDark works properly. This is a new problem with my Calibration which I don't have an answer for...
Cleon

[astropixel]

...use format hints in image calibration to override DSLR_RAW Format Explorer. Type "raw" (leave out the inverted commas) in the input and output fields. That way even if DSLR_RAW FE defaults it will be overridden.

[pfile]

the batch format conversion script does not support format hints. not yet anyway.

[Terry Danks]

Well, the elephant labored and brought forth a mouse, meaning, I have a master bias frame after all of this.
Mind, I am very unsure is all is right with it.

I got a collection of .fits files from the Batch Conversion script which I ran on the bias files. I then integrated them to get a master bias.

But I know nothing about this DSLR_RAW thing that is mentioned.

There was a "Format hints" header in the Image Integration tool but it had "Input hints" only, which I set to "raw" as per ap's advice. I saw no "Output hints" section though.

[pfile]

well, unfortunately your master bias is probably debayered and is not really usable as a calibration frame.

open the "format explorer" window, and you'll see a list of all the different file handlers. the 2nd from the top is is DSLR_RAW. double click on that and you'll get a control panel that lets you configure the dcraw CR2 importer. you need to uncheck the white balance checkboxes, check the "no black point correction" box and check the box next to bayer RGB (not monochrome). this is equivalent to the input format hint "raw".

after you do this, when you open a CR2 and do an auto screen transfer function, the image will probably look all green. if you zoom in you will see a checkerboard pattern. you are looking at the raw sensor data. if you want to see the proper image, you can use the Debayer process to debayer the image, or you can go back into DSLR_RAW and change it back to bilinear (or some other debayering method).

if you have not updated to the latest Debayer module, you will not be able to debayer the image as i have described. until it was fixed recently it did not support color raw files.

anyway, that's how the DSLR_RAW importer needs to be configured to do a proper batch format conversion to .fits. after juan helps me debug the new batch format conversion script that can handle format hints, this won't be necessary any more.

[astropixel]

Terry - my apologies, it is input hints only. The problem with ageing memory.

If you read through the DSLR_RAW tutorial it details the steps and set up required for DSLR processing.

If you can watch the videos that will assist with the menu selections, although v1.6, it's not too different.

Scripts are written by third parties and are included because they are useful, consequently features that you find in the PI tools may not be present in scripts.

What pfile is saying is correct, and the DSLR_RAW workflow outlines this clearly. You must set your DSLR_RAW preferences before using the script.

If you are using imagecalibration for instance then you can override the RAW settings in Format Explorer with Format Hints 'Input' (raw will set the conversion to .fit without debayering, which is what you want)

Here is the excerpt from the DSLR_RAW workflow to set the RAW conversion correctly.

Text version starts here

Step-by-Step

Go to the menu bar and select as follows

View > Explorer Windows > Format Explorer

   Format Explorer - Double click DSLR_RAW

      (Edit Preferences - select)

         RAW Format Preferences
   
            Color Adjustment - all 3 sliders = 1.0000000
   
            White balance - deselect all

            Create RAW debayer image - select

            No black point correction - select

            OK

This sets your RAW conversion correctly for the work that we do on astro images.

Now you can use the BatchConversionScript with confidence. The system is also set up for all RAW conversion throughout all the PI tools. Again if in doubt and using a 'tool' use the Format Hint Input to override the "raw" preferences.

Regards,

ap.

PS: Reading between the lines it will be beneficial to have a good detailed read of the workflow and links as a primer for PI, just to get more comfortable with the menus and concepts. The DSLR_RAW workflow was set up with that in mind. It's not perfect but it's a good place to start for DSLR users.

[Terry Danks]

OK, pfile and ap, I have redone the master bias, hopefully correctly this time. 

I am off to look for tutorials on this stuff. Had no idea there was a DSLR_RAW tutorial. And, had I found it, I don't think I'd have been ready for it until now.

Thanks . . .

Terry Danks

[astropixel]

Terry. There is a whole bunch of tutorials. Harry's videos are relentlessly good. There are the PI videos too.

Format Hints has Input and Output with frames added. However, after reading the tool tips I'm not sure that both input and output should be raw - input set to raw makes sense. Point it I have done it both ways with no difference as far as I can tell. Strictly speaking I think input is all that's required for IC, which has a separate selection for output file type.

[Juan Conejero]

Help Juan, I'm about to be eaten alive, but I would like to settle this once and for all.

Aha, now you know how I feel when I read my email and take a look at the forums each day since January 2008 (date we published the first commercial verison of PI)

Well, I can't say anything more descriptive and more helpful than what you, Rob, Andy and the others have said (I appreciate your help guys, as always). I honestly don't think calibration in PI is difficult at all. It's very true that it could be more automatic, and we have some ongoing projects to achieve this (the aforementioned CalibrationPipeline script and other projects), but as I often say, our workload is immense and we have too many priorities right now.

I really need you to understand me here. For example, we have released a new tool a few days ago: MultiscaleMedianTransform. You could say that an automated image calibration pipeline is more important than MMT now. No, it isn't. The reason is that we already have image calibration well covered now (it can be improved, but IMHO it is rather good now), but MMT is something new that opens exciting possibilities —just take a high SNR picture and sharpen it with MMT and you'll know what I mean. Wait to see the next tool we are working on now (hint: have you ever wondered if one individual RGB channel could be synthesized from the other two channels?). PI is innovation or nothing, and as long as I have the helm it will continue that way.

As for the FITS conversion step, it is just one necessary step to use the ImageIntegration tool with DSLR raw images, and we have it automated with the BatchFormatConversion script (and Rob is now working on adding format hints to facilitate it even more). The ImageIntegration tool has been designed to be able to integrate an unlimited number of images (hundreds, thousands) using relatively low resources. To achieve that it uses a technique known as incremental reading to load image data. Basically II loads all of the images being integrated by successive pixel row strips. Unfortunately this is only possible with the FITS format, and not at all with any DSLR raw format AFAIK. Yes, I could rewrite II to load a set of DSLR raw images in memory and then integrate them without touching the disk, but then we'd have the forum (even more) plagued with messages such as 'ImageIntegration doesn't allow me to stack more than nn images', where nn depends on 32/64 bits and the amount of RAM installed. Or I could implement II in an inefficient way to reload the data and write it to temporary storage, which is an inelegant solution that doesn't fit well in PI's design standards (and would lead to too many comments such as 'PI's stacking is very slow, use XXX instead' on the forums). I frankly think the conversion step is the most efficient and modular solution. And it is just a one-time single step.

[Terry Danks]

Juan and ap:

Many thanks for the replies but I am unsure if you understand just where I am coming from.

PI is already so advanced of my abilities to use it, I can't begin to make that clear.
People refer to the tutorials. It seems I am missing a lot of them. I have watched ALL of Harry's I think. Most of them multiple times. If that guy isn't on the payroll, he should be! Without his tutorials I'd have gotten nowhere! However, even his tutorials cover only the basic uses of the tools. The myriad of optional settings is rarely explained. How am I supposed to figure them out?

I have found a few other tutorials at http://pixinsight.com/tutorials/index.html

But, when I look at the scads of tools in PI, I am lost really. Many (most? almost all?) do not seem to have the slightest information as to what they do or how to use them. Even the "tool documentation" pages are empty. Even something as elementary as simple vs dynamic cropping is a problem with no documentation. There's a tutorial on the DSLR_RAW?

The PI philosophy seems to be that "manuals are for sissies and we have no intention of ever producing one."

From my viewpoint, PI development has already far outstripped its users abilities to make any actual use of the developments. It's as if they are only for a small cadre of insiders.

I am just one poor neophyte user. I don't need more tools . . . more "development."

I need documentation, and, either a comprehensive manual, or a detailed tutorial for every single tool in the box.

Until that is available. I just scrape by, making use of the same limited tool set and, of the few I actually use, I probably make very inexpert use of.

PI is making it very hard for new users to, well . . . actually make effective use of it.

[astropixel]

Hi Terry. Know how you feel. Documentation has lagged development and there is an implied level of knowledge among users. Consequently among the many excellent technicians explanations tend be pitched from that perspective. As with any new system teaching the basics and laying a foundation is the first step. I suspect that several members, familiar with technical instruction would be happy to help - myself included.

DSLR_RAW workflow was compiled with the above in mind, and I am quite happy to assist where required. Even if you ignore the links to technical stuff - helpful and included to satisfy the technicians, even though the intention was a basic do this then do this approach.

The master frame section was I confess an afterthought - a response to an obvious gap. Unfortunately, and I am guilty of this, it has not followed the basic approach intended. I will include a section that is menu driven and brings users back to basics, just as soon as I can. In the meantime;

Read here for master frame creation. I have highlighted in bold type all the important window settings. The rest is read between. This is adapted from a tutorial written by Vicent.

 http://pixinsight.com/forum/index.php?topic=2570.msg19019#msg19019

 Here is the DSLR_RAW workflow (under the Tutorials section on the forum) - leave out the detail - go to section 3, the text version and follow step by step  -it's menu driven, so very basic. http://pixinsight.com/forum/index.php?topic=2570.0

Keep asking - PM if you like.

[pfile]

The PI philosophy seems to be that "manuals are for sissies and we have no intention of ever producing one."

From my viewpoint, PI development has already far outstripped its users abilities to make any actual use of the developments. It's as if they are only for a small cadre of insiders.

I am just one poor neophyte user. I don't need more tools . . . more "development."

I need documentation, and, either a comprehensive manual, or a detailed tutorial for every single tool in the box.

Until that is available. I just scrape by, making use of the same limited tool set and, of the few I actually use, I probably make very inexpert use of.

PI is making it very hard for new users to, well . . . actually make effective use of it.

everyone feels the same way about documentation. you are not the first person to bring this up, believe me.

things are getting better. a lot of tools have tooltips if you hover your mouse over the UI elements. and we have the documentation compiler, so it's possible for anyone to produce the in-tool documentation.

maybe PI just needs a splash screen with a link to sean's page

http://www.drakevisual.com/pixinsight-links/

one of sean's links is RBA's unofficial reference guide. that's good to read.

i guess the "problem" with pixinsight is that you probably need to know the basics of astronomical image processing before you start, or be willing to study them as you learn PI. while i believe that the tool is well within the grasp of beginners, it is quite possibly the most advanced image processing platform available anywhere. that means that you have to take the time to understand the underlying concepts... both old (calibration, deconvolution, debayering, wavelets) and new (wavelets, multiscale processing, pixelmath, etc.).

maybe someone should write a book about the basics http://www.willbell.com/aip/index.htm

[astropixel]

Terry. Screenshots for master frame calibration and creation. Try these as a starter. Hopefully helpful. These are based on Vicents process icon tutorial (dont worry about process icons for now, it's the content that matters).

http://pixinsight.com/forum/index.php?topic=2570.msg24512#msg24512

http://pixinsight.com/forum/index.php?topic=2570.msg24512#msg24512

There needs to be a basic guide to PI. What's available is not that basic and while understanding image processing is necessary it doesn't necessarily get you over the hurdle of learning a completely new system. I think that is the issue here. Other systems are pretty much set and forget while PI is hands on. It's the mental transition and overcoming preconceptions that appears to be the biggest hurdle.

[Cleon_Wells]

Terry, what pfile said is supported by my experience at the PI 2010 Adler Planetarium workshop. This was a small group of 8 gentlemen, two worked at NASA with images from space telescopes, another worked at Lockheed Martin, another teaches Astronomy at a college in California,  and the only processing I had done to that point was with PhotoShop, needless to say I was lost from the first day which was devoted to IC, but I still enjoyed the experience. PI software is an in-depth image processing platform that I will never completely understand. That said, I still get a kick out of discovering something new with this software even though I probably don’t understand how it works.
Cleon
PS astropixel has done a great job in compiling info for us.

[astropixel]

The DSLR_RAW work flow is rewritten/reformatted/decluttered. It should be easy to follow. Added some images of PI tool settings...

http://pixinsight.com/forum/index.php?topic=2570.0