Author Topic: New photometric superflat module  (Read 11606 times)

Offline rga218

  • Newcomer
  • Posts: 37
New photometric superflat module
« on: 2014 February 02 19:47:57 »
I've written a module to construct "photometric superflats" and would like to make it available to fellow PixInsight enthusiasts. The module is currently "beta" quality (being generous) but I think it's far enough along to be useful.

Photometric superflats are flat field correction frames constructed using the known fluxes of photometrically calibrated stars in the field. In general, photometric superflats are useful as "delta flats" after an initial correction using twilight or dome flats, and you would only use them if you wish to do really precise flat fielding of your data. The photometric superflats account for residual multiplicative flat-fielding errors, and after applying them residual errors will be additive in nature (and can be dealt with using other approaches, such as by subtracting off a model). You can read more about photometric superflats here:

http://www.adass.org/adass/proceedings/adass00/P1-03/

If you'd like to test the module out, you can grab the source code here:

https://github.com/robertoabraham/PhotometricSuperflat

The module reads a table produced by the AperturePhotometry script, fits a polynomial surface (of degree up to 4) to the data and produces a photometric superflat frame as well as a couple of plots showing the relative flux errors in the calibration stars with and without the application of the flat. Here is a screenshot showing it in use:

https://dl.dropboxusercontent.com/u/110230258/SuperflatScreenshot.png

The easiest way to test it out is with simulated data. To do this, generate a fake image using the CatalogStarGenerator script, then modify this image with some function using the PixelMath module. (It's obviously important that you modify the image using some sort of multiplicative process and not an additive process). Run the AperturePhotometry script on the simulated image to generate a catalog. Then fire up the PhotometricSuperflat process and click on the button to select the catalog (whose filename will end with "_WCS.csv"). Then drag the process icon onto the image you have just photometered to generate the Superflat and diagnostic plots.

Caveats: It works for me, and I'm using it as part of my research, but it may not work for you. I'll attempt to gradually improve this module, and would be happy to receive bug reports and requests for new features. That said, I'm relatively new to PixInsight and I'm not a professional programmer and this is my first attempt at doing anything non-trivial in C++ (although I have done a lot of C programming and a fair bit of Objective-C hacking). There are likely to be some ugly things in the code, and I'm sure there are also big gaps in my understanding of the optimal way to code modules for PixInsight. Again, suggestions for improvements are very welcome, and I'll do what I can to make the module better.

Known Deficiencies in the PhotometricSuperflat Module:

1. It's not OS-independent because it calls the built-in gnuplot executable to draw the diagnostic plots, and I haven't figured out how to find the location of the gnuplot executable programmatically. So unless you're running it on Mac OS X you'll need to modify line 173 in the PolynomialSurface.h file so it points to the gnuplot executable in the PixInsight distro. Right now it reads:

ExternalProcess().ExecuteProgram("/Applications/PixInsight.app/Contents/MacOS/gnuplot " + gnuFilepath);

2. There is no documentation (yet).

3. The ID of the created superflat window is simply "Superflat". I think this should probably be something more sensible, like the ID of the window I dragged the process icon onto with _Superflat appended.

4. I have not yet implemented any code to enable the command line interface.

5. I should probably include a check to allow the user to disable production of the diagnostic plots.

6. The AperturePhotometry table is reloaded each time the process icon is dragged onto the image. I did this on purpose but now I think it should be loaded only once at the beginning (as soon as the user navigates to the table file with the file selector).

7. I've got the important classes all defined in a single header file (PolynomialSurface.h). In future I'll organize the source a bit better into separate .h and .cpp files for the various classes.

I hope somebody finds this module useful,

Bob

P.S. In case you're curious, I developed this code to try to improve flat fielding as part of a project I've recently embarked upon to do some ultra-low surface brightness imaging. You can read about this project here:

http://adsabs.harvard.edu/abs/2014arXiv1401.5473A
http://adsabs.harvard.edu/abs/2014arXiv1401.5467V

(Juan - if you are tracking papers which have made use of PixInsight, you can add these two to your list. The first just appeared in PASP, and the second is coming out as an ApJ letter).

P. P. S. Juan: I'm not an expert in open-source licenses, but when prompted by GitHub I picked the 2-clause BSD license because it seemed very unrestrictive and I figured it might make things easy for you if you'd like to someday distribute this module. If you would prefer an alternative license just let me know. Also, I based the non-numerical component of the module (parameter setting and serialization, GUI, etc) on the Sandbox module, and for the source files that are basically based on Sandbox I just substituted the new module name in the boilerplate license text at the start of the files, leaving the "Copyright (c) 2003-2013, Pleiades Astrophoto S.L. All Rights Reserved" stuff there. Is that the accepted procedure? If not, just let me know what you'd like it to say and I'll make the changes.
« Last Edit: 2014 February 02 21:49:09 by rga218 »
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New photometric superflat module
« Reply #1 on: 2014 February 02 21:54:18 »
if i understand this right, technically it should be used only to correct 'suboptimal' flat fielding thru normal techniques.

however, is it of any use when there are additive gradients in an image for which it is difficult to create a proper background model? such as when imaging emission nebulae which essentially cover the entire field of view of the sensor. if the 'superflat' were subtracted from the image rather than divided, would it work? or is it not able to handle such types of gradients and only expects to find very minor "flat-like" gradients in the image?

thanks

rob

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #2 on: 2014 February 03 07:45:59 »
Hi Rob,

The module is designed to model the multiplicative component and leave the additive part untouched. Whether or not it would be useful for a given task kind of depends on your goals. If the goal is to make a pretty picture and you don't care about photometric fidelity, then I don't think it would be useful, at least not on its own. For example, light pollution and moonlight gradients are additive (they change the local background level but preserve the backround-subtracted star counts), and it should leave those untouched. On the other hand, it might be useful if you want to model the background properly and preserve the true relative star brightnesses, since you could divide out the multiplicative errors first (e.g. flat fielding errors), then subtract off the additive part (e.g. light pollution gradients). In any case, it's intended mostly for situations where you care about precision photometry, in which case it is likely to be quite helpful.

Bob
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New photometric superflat module
« Reply #3 on: 2014 February 03 08:45:12 »
yeah, i gathered that, i am just trying to understand if it can be repurposed. as i mentioned sometimes it is very difficult to properly compute the background gradient in an image with extensive nebulosity. while it can be done and look good to the eye using DBE, it becomes obvious that the subtracted gradient is false when you try to make a mosaic out of two such images that have been background subtracted in such a way.

so i was hoping that using this technique could derive the proper additive background model as well.

rob

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #4 on: 2014 February 03 09:08:00 »
Ah, I see. For purposes of stitching together a mosaic I bet it _would_ help if you applied it before doing the DBE, as then the additive corrections would be better constrained and the tiles of the mosaic might well blend together more naturally. But I think you'd have to try it out and see.

Bob
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New photometric superflat module
« Reply #5 on: 2014 February 03 10:13:44 »
will do; need to grab your source code and compile it.

rob

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #6 on: 2014 February 03 15:06:01 »
Excellent. I'm very curious to learn if it proves useful for you.

Bob
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: New photometric superflat module
« Reply #7 on: 2014 February 05 03:48:41 »
Hi Robert,

First of all, congratulations on a great new tool, and welcome to PixInsight development.

I think this tool has the potential to change the way we understand flat fielding, and more. We'll talk about that later; since it would be nice if we could release this module as soon as possible, let's solve the only game-stopper problem:

Quote
1. It's not OS-independent because it calls the built-in gnuplot executable to draw the diagnostic plots, and I haven't figured out how to find the location of the gnuplot executable programmatically.

The PixInsight Core application defines a number of environment variables on all platforms (mental note: I have to describe all of them in PCL's documentation). One of these variables is PXI_BINDIR, which is the full path to the bin installation directory. So in your case you can do something like this:

Code: [Select]
IsoString mbBinPath( ::getenv( "PXI_BINDIR" ) );
String ucBinPath = mbBinPath.MBSToWCS();
ExternalProcess().ExecuteProgram( ucBinPath + '/' + gnuFilepath );

which should be portable to all platforms (even Windows). Since getenv() pertains to stdlib.h, you shouldn't need any additional headers (stdlib.h is #included everywhere).

As for licenses, PCL allows you to release your module and/or source code under any software license of your choice. However, if you like the BSD license, you can release your module under the PCL License (PCLL):

http://pixinsight.com/license/PCL_PJSR_1.0.html

which is, as you see, a modern BSD license with two additional clauses: (3) to prevent uncontrolled 3rd party endorsements, and (4) to guarantee that we are somewhat recognized. You already have included PCLL at the beginning of each source code file, but please tell me what you want to do if we release this officially as part of the PCL distribution (note that we don't claim you to release your code under PCLL just as a result of deriving it from the SandBox module (another mental note: release SandBox under a public domain license)).
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #8 on: 2014 February 05 15:55:07 »
Hi Juan,

Thanks... I hope the tool proves useful.

I made your suggested changes to try to make the source OS-neutral and have pushed them onto the GitHub repository. (I tested it out on my Mac and it finds gnuplot but I don't have PI installed on anything other than a Mac OS X machine so I haven't been able to test the module on any other OS). I also changed the licence to the PCLL. Please feel free to distribute the module in any way you'd like. From time to time I'll try to improve it but you should always feel free to distribute it in whatever way you think would be most useful.

One thing to note going forward is that PhotometricSuperflat reads text files produced by the AperturePhotometry script, so if that script's output file format changes then PhotometricSuperflat will break.

All the best,

Bob
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New photometric superflat module
« Reply #9 on: 2014 February 05 17:13:33 »
i only have OSX as well but i downloaded the latest from github, installed gnuplot (via macports) and then built the module against PCL-02.00.09.0651-20131214. it seems to work; i got two plots out as well as the superflat.

then i started wondering if the module actually applies the flat it calculates, so i divided the original image by the new imageā€¦ and when i tried to compute the STF of the result, PI threw an API error. i need to upgrade PI to the latest and try again and then report that bug. i'm guessing this means that the flat is not actually applied by the process, is that correct?

thanks

rob

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #10 on: 2014 February 05 17:49:04 »
It creates the superflat but doesn't apply it... I figured the user would want to look at the flat and only apply it with PixelMath after inspection. I also assume the user will want to apply the flat to many frames, presumably as part of a pipeline.

Gnuplot is built in to pixinsight... No need to install it!
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: New photometric superflat module
« Reply #11 on: 2014 February 05 17:53:16 »
ah okay, that explains what juan wrote then; i didn't see how it was finding gnu plot in /opt/local from the code fragment he wrote.

should the flat be applied with straight division?

rob

Offline rga218

  • Newcomer
  • Posts: 37
Re: New photometric superflat module
« Reply #12 on: 2014 February 05 17:54:19 »
Yes just straight division.
Roberto Abraham
Professor of Astronomy & Astrophysics
University of Toronto

Offline cmm

  • Newcomer
  • Posts: 1
Re: New photometric superflat module
« Reply #13 on: 2014 April 20 07:58:27 »
This looks like a potentially really nice option to deal with sub-optimal flat fielding.

I would like to try out this script, but I have never tried scripting before.  How do I take the source code and convert it into a usable script?

Offline gvanhau

  • PixInsight Old Hand
  • ****
  • Posts: 345
Re: New photometric superflat module
« Reply #14 on: 2014 June 21 16:52:08 »
Hello Bob
I managed to get it compiled in Windows. I momentary disabled the GNUplot because for some reason it could not start the external aplication. Will see if I can resolve this.

I was trying this module because when I calibrate my images, I get over corrected corners....
Now the result is not what I expected, unless  I am doing something wrong.

I took one of my images, solved it, then run the Aperture Photometry script on it.
I then run The superflat module using the .csv file generated by the script.
Then I divided the original image by the superflat.

I tried different fit degrees...

You can see the results in the attached image.
Top left original calibrated image (over corrected corners), top midle detected stars on aperture photometry script, top right background output of aperture photometry, bottom left generated superflat, bottom right original image divided by superflat.


Regards
Geert
Geert Vanhauwaert