Author Topic: Comet stacking suggestions  (Read 14129 times)

ruediger

  • Guest
Comet stacking suggestions
« on: 2011 August 03 11:05:41 »
Hi all,
two days ago I had the opportunity to capture comet Garradd near Messier 15. My goal is to combine a star aligned with a comet aligned stack. I know, that Deep Sky Stacker offers an automatic function for this, but the result was very poor.
For the comet alignment I wrote a small Javascript, that performed a linear transformation to the star aligned pictures with respect to the comet's speed. This wasn't as straightforward as I thought, because I measured the comet's position only on the first and the last picture. For interpolation, I need to access exposure and acquisition time from the calibrated fits files. How do I access this?
For a quick and dirty workaround, I extracted exposure and acquisition time with perl and exiftool, and pre-computed x- and y-shifts for each frame in perl. So basically, I ended up with an array like this:
var pics = [
[ "debayer_IMG_5464_c_r.fit" ,  0, 0 ],
[ "debayer_IMG_5465_c_r.fit" ,  1.79   , -0.70   ],
 ...
[ "debayer_IMG_5548_c_r.fit" ,  186.00   , -72.80   ],
[ "debayer_IMG_5549_c_r.fit" ,  188.63   , -73.83   ]
];


Now both stacking results are somewhat poor. Even with a high rejection ratio of 20% the star field stack still shows some comet's remains and the comet stack still shows star trails.
I'm mainly interested in removing the star trails from the comet stack. Maybe I remove all bright stars before I do the shifting? This could be done automatically within the JavaScript. Or are there better ways?

Regards,
 Rüdiger

BTW: EOS 500Da + Canon EF 5,6/400mm L on Astrotrac. 23x2min + 51x3min unguided.

Offline georg.viehoever

  • PTeam Member
  • PixInsight Jedi Master
  • ******
  • Posts: 2132
Re: Comet stacking suggestions
« Reply #1 on: 2011 August 03 11:19:05 »
Interesting project.
This thread may help with extraction of FITS keywords: http://pixinsight.com/forum/index.php?topic=1904.msg11826#msg11826

Georg
Georg (6 inch Newton, unmodified Canon EOS40D+80D, unguided EQ5 mount)

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Comet stacking suggestions
« Reply #2 on: 2011 August 03 13:32:46 »
Yes, an interesting project for sure.

Please note that in the above thread I had to write a workaround to a bug we had in late releases of PI version 1.5.

Here is an updated script that prints all FITS header keywords for a set of selectable FITS images:

Code: [Select]
#include <pjsr/DataType.jsh>

/**
 * Opens a FITS file at the specified fitsFilePath path and reads all keywords
 * from the primary HDU. Returns an array of FITSKeyword objects.
 *
 * Limitations:
 * - HIERARCH convention not supported, i.e. all keyword names must
 *   be strings of (possibly padded with white spaces) eight characters.
 * - Reads only the first HDU; extension HDUs are ignored.
 */
function LoadFITSKeywords( fitsFilePath )
{
   function searchCommentSeparator( b )
   {
      var inString = false;
      for ( var i = 9; i < 80; ++i )
         switch ( b.at( i ) )
         {
         case 39: // single quote
            inString ^= true;
            break;
         case 47: // slash
            if ( !inString )
               return i;
            break;
         }
      return -1;
   }

   var f = new File;
   f.openForReading( fitsFilePath );

   var keywords = new Array;
   for ( ;; )
   {
      var rawData = f.read( DataType_ByteArray, 80 );

      var name = rawData.toString( 0, 8 );
      if ( name.toUpperCase() == "END     " ) // end of HDU keyword list?
         break;

      if ( f.isEOF )
         throw new Error( "Unexpected end of file: " + fitsFilePath );

      var value;
      var comment;
      if ( rawData.at( 8 ) == 61 ) // value separator (an equal sign at byte 8) present?
      {
         // This is a valued keyword
         var cmtPos = searchCommentSeparator( rawData ); // find comment separator slash
         if ( cmtPos < 0 ) // no comment separator?
            cmtPos = 80;
         value = rawData.toString( 9, cmtPos-9 ); // value substring
         if ( cmtPos < 80 )
            comment = rawData.toString( cmtPos+1, 80-cmtPos-1 ); // comment substring
         else
            comment = "";
      }
      else
      {
         // No value in this keyword
         value = "";
         comment = rawData.toString( 8, 80-8 );
      }

      // Perform a naive sanity check: a valid FITS file must begin with a SIMPLE=T keyword.
      if ( keywords.length == 0 )
         if ( name != "SIMPLE  " && value.trim() != 'T' )
            throw new Error( "File does not seem a valid FITS file: " + fitsFilePath );

      // Add new keyword.
      keywords.push( new FITSKeyword( name, value, comment ) );
   }

   f.close();

   return keywords;
}

function main()
{
   var ofd = new OpenFileDialog;
   ofd.multipleSelections = true;
   ofd.caption = "LoadFITSKeywords: Select FITS Files";
   if ( ofd.execute() )
      for ( var i in ofd.fileNames )
      {
         console.writeln( "<end><cbr><br><b>", ofd.fileNames[i], "</b>" );
         var keywords = LoadFITSKeywords( ofd.fileNames[i] );
         for ( var i in keywords )
            console.writeln( keywords[i].name, ',', keywords[i].value, ',', keywords[i].comment );
      }
}

main();

Regarding alignment, have you tried with DynamicAlignment? DA can align images with one, two, and >= three alignment points. For a single alignment point, a simple translation is applied to align the images. For two points, translation + rotation + scale change on the line joining both alignment points. For >= 3 points, DA uses thin plates (2D surface splines).
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Comet stacking suggestions
« Reply #3 on: 2011 August 03 13:51:42 »
I have found this thread that I think you may find interesting:

http://pixinsight.com/forum/index.php?topic=2008.msg13040#msg13040

This was a nice set of images of McNaught by Thomas W. Earle (Wade) and the problem was similar to yours. Solving this problem was a lot of fun to me and I think I found a rather good solution to remove the stars without degrading the comet image. I don't know if it's applicable to your images by I think it's worth trying, and/or perhaps it may give you some ideas.

Note that at the time of posting this example we still hadn't the LinearFit tool, which you can use now instead of StarAlignment + PixelMath to normalize the images, in case you have some gradients. Keep in mind that this technique requires a very accurate normalization to remove the stars, which if you have some additive gradients, you probably will have to carry out manually before ImageIntegration.

Hope this is useful. Good luck and have fun!
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

ruediger

  • Guest
Re: Comet stacking suggestions
« Reply #4 on: 2011 August 04 08:39:46 »
Thanks for your comments! I checked the McNaught thread and found some valuable tips of how to stack the comet.

I also run the fits metadata script on some selected images. Within PixInsight, ISO, acquisition date/time, shutter speed is lost. Is it possible to move these information during the ImageCalibration process from the RAW (.CR2) files to the fits files?

For comparison, both DSS and Fitswork store more information in the FITS files:
Code: [Select]
# === PixInsight calibrated and registered image ===
Z:/TEMP/Garradd.comet/debayer_IMG_5464_c_r.fit
SIMPLE  ,                    T , file does conform to FITS standard             
BITPIX  ,                  -32 , number of bits per data pixel                 
NAXIS   ,                    3 , number of data axes                           
NAXIS1  ,                 4770 , length of data axis 1                         
NAXIS2  ,                 3178 , length of data axis 2                         
NAXIS3  ,                    3 , length of data axis 3                         
EXTEND  ,                    T , FITS dataset may contain extensions           
COMMENT ,,  FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT ,,  and Astrophysics', volume 376, page 359; bibcode: 2001AA...376..359H
CREATOR , 'PixInsight Standard 01.07.00.0702' , Software that created the file 
PINSIGHT, 'PCL 01.00.94.0369'  , PCL: PixInsight Class Library                 
COLORSPC, 'RGB     '           , PCL: Color space                               
RESOLUTN,                  72. , PCL: Resolution in pixels per resolution unit 
RESOUNIT, 'inch    '           , PCL: Resolution unit                           
ICCPROFL, 'ICCProfile'         , PCL: File includes ICC profile extension       

### Deep Sky Stacker ###
Z:/TEMP/dss-test.FTS
SIMPLE  ,                    T , file does conform to FITS standard             
BITPIX  ,                   32 , number of bits per data pixel                 
NAXIS   ,                    3 , number of data axes                           
NAXIS1  ,                 4770 , length of data axis 1                         
NAXIS2  ,                 3178 , length of data axis 2                         
NAXIS3  ,                    3 , length of data axis 3                         
EXTEND  ,                    T , FITS dataset may contain extensions           
COMMENT ,,  FITS (Flexible Image Transport System) format is defined in 'Astronomy
COMMENT ,,  and Astrophysics', volume 376, page 359; bibcode: 2001AA...376..359H
BZERO   ,           2147483648 , offset data range to that of unsigned long     
BSCALE  ,                    1 , default scaling factor                         
ISOSPEED,                  800                                                  ,
EXPTIME ,                 120. , Exposure time (in seconds)                     
EXPOSURE,                 120. , Exposure time (in seconds)                     
SOFTWARE, 'DeepSkyStacker 3.3.2'                                                ,

### FITSWork ###
Z:/TEMP/fitswork-test.fit
SIMPLE  ,                    T                                                  ,
BITPIX  ,                  -32                                                  ,
NAXIS   ,                    2                                                  ,
NAXIS1  ,                 4770                                                  ,
NAXIS2  ,                 3178                                                  ,
DATE-OBS, '2011-08-01'                                                          ,
TIME-OBS, '22:20:35'                                                            ,
INSTRUME, 'Canon EOS 500D'                                                      ,
EXPTIME ,  1.2000000000000E+02                                                  ,
COMMENT ,,   ISO 800                                                             
COMMENT ,,   Aperture 5.7                                                         
COMMENT ,,   Focal Length 400                                                     
COMMENT ,,Bearbeitet mit Fitswork am 04.08.2011                                   
COMMENT ,,FWHistPar Min     2007 Max    17974 Gam 1.77 Typ 1                     
COMMENT ,,FWHP3 11B44FAE0EF468C6C3A537F                                           
FitsWork has everything I need: TIME-OBS,DATE-OBS, EXPTIME and even more comments about used equipment and setting.

Quote
Regarding alignment, have you tried with DynamicAlignment?
I measured the comet's position with the DynamicAlignment tool, but only on two pictures and I think, it would be a pain to do it on more than even 10 or so.

Regards,
 Rüdiger

Offline Juan Conejero

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 7111
    • http://pixinsight.com/
Re: Comet stacking suggestions
« Reply #5 on: 2011 August 04 12:20:59 »
Quote
Is it possible to move these information during the ImageCalibration process from the RAW (.CR2) files to the fits files?

Yes, this isn't difficult to do. If you are a Linux/UNIX or Windows user I can try to release an updated ImageCalibration module with this feature. If you are a Mac OS X user then this would have to wait until September. I am (supposed to be) on vacation...
Juan Conejero
PixInsight Development Team
http://pixinsight.com/

ruediger

  • Guest
Re: Comet stacking suggestions
« Reply #6 on: 2011 August 05 02:09:05 »
Quote
If you are a Linux/UNIX or Windows user I can try to release an updated ImageCalibration module with this feature.
I'm using Windows 64bit, but take your time to update the IC module.

For the comet stacking, I will try the following approach: I remove the stars before stacking by some sort of automated clone stamping. Basically I build a star mask from the star field stack and use this inverted star mask to replace all stars in an Image A with a randomly shifted version B of itself. The amount of shift is about the largest star diameter. The PixelMath is pretty simple... only "B" as function :). My preliminary result is attached. The star mask is not covering all stars, but most of the stars are nearly perfectly replaced with background from the close vicinity.

The modified image A will then automatically aligned to the comet's position (this step I have already programmed). Now I'm trying to fully automate the whole process with javascript, but this will take some time. But I'm having a lot of fun with it :)

Rüdiger

Offline A.Steinel

  • Newcomer
  • Posts: 4
Re: Comet stacking suggestions
« Reply #7 on: 2011 August 06 08:29:06 »
Hi,

I'm also very interessted in this. I have 105 images a 30 seconds of the same comet on the same day to stack and DSS is also producing very bad results.

Best,
Andreas

Offline RBA

  • PixInsight Guru
  • ****
  • Posts: 511
    • DeepSkyColors
Re: Comet stacking suggestions
« Reply #8 on: 2011 August 06 09:20:07 »
I am (supposed to be) on vacation...

I'm somewhat familiar with that concept  ;)

Offline zvrastil

  • PixInsight Addict
  • ***
  • Posts: 179
    • Astrophotography
Re: Comet stacking suggestions
« Reply #9 on: 2011 August 06 13:13:28 »
For the comet stacking, I will try the following approach: I remove the stars before stacking by some sort of automated clone stamping. Basically I build a star mask from the star field stack and use this inverted star mask to replace all stars in an Image A with a randomly shifted version B of itself. The amount of shift is about the largest star diameter. The PixelMath is pretty simple... only "B" as function :). My preliminary result is attached. The star mask is not covering all stars, but most of the stars are nearly perfectly replaced with background from the close vicinity.

Hi Rüdiger,

you could also try to use MorphologicalTransformation for star removal. Erosion filter applied on image masked with StarMask should do the same (or better) with less effort. This operation is frequently used to make stars smaller and less prominent, but with amount 1 and sufficient number of iteration, it will remove stars completely.

regards, Zbynek

ruediger

  • Guest
Re: Comet stacking suggestions
« Reply #10 on: 2011 August 07 08:33:20 »
In the meantime I probably scripted any possible method I could think of, but there are still trails showing up in comet stack. The overall quality didn't improve much compared to picture I posted here in the thread start. I tested the following:
- building small groups of any n-th picture, where stars don't overlap, stack them with minimum method (or aggresive percentile clipping), then summing up the groups with a less aggressive rejection method
- clone stamping the stars as I had posted before, then stacking the whole set
- removing the stars with morphology transformations as suggested by Zbynek
- replacing the stars with the proper content from the star field stack where the comet is designed to be inserted. This method is the most promising, because every star, which disturbes the stacking result, is replaced with meaningful data from the destination in the final picture.
However, this method needs access to a comet free star field, taken e.g. a day before or after comet's visit. If i use the data I have right now, some comet spots will show up at the wrong places.
Right now, it looks like a chicken-and-egg problem: For a comet free star field stack, I need a good separated comet picture and for getting this, I need a comet free star field stack.

Well, I don't want to give up on this yet, instead I try to capture the star field again. M15 and even NGC 7094 look nice already, and compared to today's APOTD (http://apod.nasa.gov/apod/ap110806.html), maybe I get a slightly better overall result.

Rüdiger

Offline zvrastil

  • PixInsight Addict
  • ***
  • Posts: 179
    • Astrophotography
Re: Comet stacking suggestions
« Reply #11 on: 2011 August 29 12:46:22 »
Hi Rudiger,

what worked best for me (stacking Garrad at M71 this weekend) was Median integration with Windsorized Sigma Clipping and Sigma High at relatively low value (like 0.5). Some star trail artifacts were still there but reasonably small. I was able to remove them with aggressive Wavelet Noise Reduction at 4 or 5 first layers (comet as large-scale structure was unaffected by this). There was just small residual of M71 which I clone-stamped.
More about the process and final picture is in this thread http://pixinsight.com/forum/index.php?topic=3351.0.

regards, Zbynek

ruediger

  • Guest
Re: Comet stacking suggestions
« Reply #12 on: 2011 September 03 10:10:35 »
what worked best for me (stacking Garrad at M71 this weekend) was Median integration with Windsorized Sigma Clipping and Sigma High at relatively low value (like 0.5). Some star trail artifacts were still there but reasonably small. I was able to remove them with aggressive Wavelet Noise Reduction at 4 or 5 first layers (comet as large-scale structure was unaffected by this).
Thank you, this worked for me as well.

If the comet trail passes bright large stars, a prerecorded clone stamp session can be applied to remove these stars from the starfield image before shifting for comet aligning. This further reduces the remaining star trails.
I did this for my Garradd/Cr399-shot which is shown in the gallery.

Regards
 Rüdiger

ruediger

  • Guest
Re: Comet stacking suggestions
« Reply #13 on: 2011 September 05 07:25:20 »
...and after one month finally I have a result :)

Instead of trying to remove the comet from the starfield images and then stacking them again, I waited for another day with good weather condition to take again the starfield without comet. Some final layering with cut masks and adjustment layers was done in Photoshop CS2.

Comet stacking really should be easier. But the fully automated tool I had in mind and already started to program is on the "probably-never-gets-finished" list, so I'll lean back and wait for Zbynek to complete his programming work ;-)



Larger:
http://www.flickr.com/photos/38957679@N04/6112813331/sizes/l/in/photostream/

Regards,
 Rüdiger