PixInsight Forum (historical)

PixInsight => Tutorials and Processing Examples => Topic started by: MortenBalling on 2014 September 12 08:55:40

Title: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 12 08:55:40
(http://cdn.astrobin.com/images/thumbs/721f6471c3c444b3140e052b00a21627.1824x0_q100_watermark.jpg)

Hello everybody

Some time ago I sugested a collaboration on The Danish Astronomical Society's forum. We ended up being 8 amateur astronomers using 7 very different telescopes. The result was the planetary nebula Jones-Emberson 1. I collected all the data, and made a composite using PI. The image was selected as Image Of The Day at Astrobin.

http://www.astrobin.com/55058/ (http://www.astrobin.com/55058/)

The collaboration gave me an idea: What if I could get every amateur astronomer in the world to capture photons from the same object, just one night, and then combine all the data into one single image. The idea haunted me for a while, and I decided to make some tests, using whatever images I could find on the internet. The early tests looked promising, and I chose to try to gather as many pictures of the Andromeda Galaxy as I could, and combine them all. I ended up with 556 images, and the result blew me away.

When you have that many data, the combined image has extremely high SNR, due to several thousand hours of exposure. On the JNER1 project we ended up with 125+ hours, but thousands of hours really make a difference. Furthermore I noticed some interesting side effects. I did all the work in 100 MPixel, so I had to make substacks of 50 images to be able to handle all the data, even on a dual Xeon Mac Pro with 16 GB memory and a SSD raid. When I compared the substacks they looked very similar:

(https://lh3.googleusercontent.com/-bkKrKyTZWDI/UkvnzsVRkFI/AAAAAAAACnY/i7SxNvMGMQ0/s1280/Screen%2520Shot%25202013-09-30%2520at%25206.12.28%2520PM.png)

We all have seen images of the object we are working with, so we have an idea of what we would like it to look like. However there are smaller or bigger differences in the final results. Some images are to green others to magenta. Sometimes the guiding drifts a little, and perhaps the image hasn’t been flattened correctly. All those small differences are averaged using this method. The red version is compensated by the green one, the elongated stars in one direction is compensated by another image with stars elongated in another direction, and so on. Of course our personal preferences will show through the averaged stacks, but people tend towards a neutral white balance in RGB and LRGB images, and PI has tools for making that.

Most of all, noise is minimized to the extreme. Normally when we work on an image, there is a certain amount of exposure, which is optimal. The SNR of a stack of images, increases with the square root of the increase in exposure. Twice as long exposure gives 1.4 times better SNR, and 10 times longer exposure gives 3.2 times better SNR. Furthermore most of us struggle to find enough CS nights, so normally we end up in the 1-20 hour range.

The following is a rough description of the method I’ve developed during tens of projects so far.

First i gather images. I now only use Creative Commons licensed images. Astrobin, Flicker, Google Images, Wikimedia commons etc. are excellent sources. Be careful not to use copyrighted images, and also make sure that you are allowed to “modify, adapt, or build upon” the images you use. I’ve found this webpage especially useful:

http://search.creativecommons.org

However the internet is a goldmine, once you start searching. Remember that you must credit the people who’s images you use, so write them down while collecting images. I personally use a spreadsheet. Also remember to share your final CI image with a CC license.

The first step I do is to roughly crop all the images. Remove frames, text and noisy edges etc. Then I rotate (and/or flip) each image to get roughly the same side up. Next I choose a master image, and rescale it to twice the resolution I want to end up with. I save that image as a “TransformMaster”.

Then I register all images in Pixinsight using StarAlign. I use distortion correction, to compensate for the different optics and enable “Generate Masks”. More on that later. Normally most images will register, but you might get errors. In that case ImageIntegration skips to the next image. In most cases around 80-90% of the images works fine. The ones that didn’t register, can be further cropped, to see if that helps, and it often does.

After registering I make an ImageIntegration of all the registered images. Use equal weight (1:1) and disable normalization. The monochrome registered images has to be converted to RGB first. Because not all images cover the full field of the TransformMaster, you will get vignetting towards the edge of the image. This is where the masks come in handy. First integrate all of them again using 1:1 weight and no normalization. Then you use the stacked masks as a “flat frame”. In PixelMath you make a simple formula: RGB_Integration/Masks_Integration, and create a new image using that. The result is a perfectly flat field.

A few examples. First the averaged RGB stack:

(http://cdn.astrobin.com/images/thumbs/7b70bfe75de59f184b6f81c1b6922d09.1824x0_q100_watermark.jpg)

Next the masks combined:

(http://cdn.astrobin.com/images/thumbs/14014aeea98ebe03500b5a33fb113817.1824x0_q100_watermark.jpg)

And finally a PixelMath of RGB/masks:

(http://cdn.astrobin.com/images/thumbs/78d37bb10a7830057463b5b34735c64d.1824x0_q100_watermark.jpg)

This is the basic method and will get you a long way. Next some more detailed processes I’ve found useful:

Luminance is a major part of the visual experience, so I use all sorts of images for making a luminans layer. Some RGB, others narrowband and even monochrome like Ha are all combined, and luminance is extracted from that. Furthermore, I make two stacks. One with the sharpest half of the images, and one with the ones with the best signal. I then combine those using masks.

The FlatRGB stack has very good color fidelity. If you’ve blended RGB and Narrowband, you can make separate stacks of RGB and HST palette if you want. But basically you don’t need to do much to the color layer, and you definitely shouldn’t change the color balance of the RGB stack.

With the LuminanceStack, on the other hand, you can go crazy. A Deconvolution with a generated PSF is a good start. Because the stack has such high SNR, you can use most filtering and processing a lot more effective than on a normal stack, without having problems with noise. That is very very motivating.

Once you’ve used all your tricks on the Luminance layer, try to start all over. Do that several times, and combine your different versions into one (perhaps even using masks). Even here you often get a better result by stacking your efforts.

Finally use LRGB combination to mix your final Luminance with the RGB stack.

There is a lot of room for improvements in the method mentioned above, but it gives a rough idea of the principle. You can make a stack of HSO narrowband, and extract each channel for luminance use. You can use PixelMath to subtract one channel from another to pick out certain details etc. It’s very inspiring to be able to experiment with the stacks, so knock yourselves out :)

I've used the technique on more than 20 objects so far, and the biggest problem is finding enough CC licensed images to use. Especially for the more rare objects. So if you want your name in future credit lists, share your images. Let's face it. We are probably never going to get rich from our hobby anyway. Thanks in advance.

Clear skies

Morten :)

PS. Here is a few links to images I’ve made using the method, including M31 in 36 MPix resolution with 300+ images:

http://www.astrobin.com/119854/ (http://www.astrobin.com/119854/)

http://www.astrobin.com/120204/ (http://www.astrobin.com/120204/)

And a large version of the Trunk: http://cdn.astrobin.com/images/thumbs/486ca58dfac069513c39aa23d13bad33.16536x16536_q100_watermark.jpg (http://cdn.astrobin.com/images/thumbs/486ca58dfac069513c39aa23d13bad33.16536x16536_q100_watermark.jpg)

Image credits for IC1396:

Adam Evans, Alexis Tibaldi, Álvaro Pérez Alonso, Andolfato, Arturo Fiamma, AstroGG, ASTROIDF, Chris Madson, Claustonberry, dave halliday, dyonis, Eric Pheterson, Frank Zoltowski, Fred Locklear, Gaby, Giuliano Pinazzi, Jorge A. Loffler, Juan Lozano, Jürgen Kemmerer, Jussi Kantola, Konstantinos Stavropoulos, Lonnie, Luca Argalia, Luigi Fontana, Michele Palma, Mike Markiw, milosz, Miquel, Morten Balling, NicolasP, Pat Gaines, Paul M. Hutchinson, PaulHutchinson, Peter Williamson, Phil Hosey, Phillip Seeber, Ralph Wagter, Richie Jarvis, RIKY, s58y, Salvatore Iovene, Salvopa, Stephane_B, Steve Yan, stevebryson, Thomas Westerhoff & Werner Mehl
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 12 08:59:23
this is really interesting - i sort of thought the fact that all the images are stretched, and stretched differently, would negatively affect the outcome. but it seems not.

very cool.

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: chris.bailey on 2014 September 12 09:08:31
Very cool indeed. I have done combinations of four or five peoples images but only in the calibrated fits format but this is in all altogether different league. The masks trick is a great one!

Chris
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 12 09:32:27
Thanks :)

@Rob & Chris

In a perfect world, having all data as linear fits would be awesome, but just keeping track of the images used here is quite a task. I first thought about emailing as many people as I could, but I also learned from the JNER1 project, that it's a lot of work just to keep track of what you receive. Sometimes members sent me new stacks, that included older data, and I tried to avoid using the same photons twice, as it just introduce more noise. The more data the merrier. I've made tests that I sadly can't show publicly, because they included copyrighted material, but the Draco Dwarf Galaxy and M78 looks pretty cool, when one is able to stretch the field as much as this technique allows. I know the field on the M31 is pretty deep, but I haven't found a way to measure it on nonlinear data.

If you have full coverage in all the datasets then this method isn't really optimal. Then normal stacking with weight based on SNR etc. is better. However the mask flattening is useful whenever the coverage is not 100%. The idea behind CI is that you use a truckload of data, and therefore you can accept loosing a little information by stacking with a simple average. I'm finding new techniques all the time as I go along, and if I have many images, I make separate stacks with sharp images, low noise images, low exposure (for example for Trapezium), and so on. I then combine those into a luminance Master for further processing.
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 12 12:08:52
so how do you deal with differing FWHM in the different images? or if you limit yourself to similar FOVs are you seeing comparable FHWM across images?

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 12 12:30:00
@Rob

I don't :)

Everything is equaled by the stacking, even FWHM. I normally start out with one of my own images, resample it like 400% and use that as TransformMaster. I haven't been around the new Drizzle features in PI yet, but working in major oversampling has been my trick so far. After aligning I look at the MasksStack and choose a framing that will give me good coverage, typically at least 50% at the edges of the final image. That equals out the FWHM, but of course you get best FWHM in the center of the image. Not that different from when I stack my own data, especially earlier on, when I used a large chip on a C8, which has a small corrected field. Bad combo!

After integration the image/stars are rather soft all over the field, so the first thing I do is to make a PSF and a deconvolution, to see what that does. Many times it's easier to use manual Deconvolution (in steps), using masks made with ATWTs. Oh yeah, btw: Van Cittert is very effective on some images, because noise is so low. I make several star masks for small and bigger stars, and use a lot of subtle MorphologicalTransform (with amount set to 0.1-0.7). I've also started using two stacks, one with the sharpest images, and one with every image and high SNR. Then I use a combination of the MasksStack and LuminanceMattes and wavelets to blend the sharp stack onto the one with best SNR. A simple LumMatte will get you far.

But all in all, it's statistics (and distortion correction) that does most of the job.

Morten

Edit. The combination of the sharpest detail in the brightest areas, and the softer areas in the darker parts of the field gives a Depth Of Field effect that i like. I've also always been a sucker when it comes to spikes (to the point of using piano wire on my refractor :P), but this method gives some beautiful spikes (without cheating). I'm currently working on M45, and wow does that have some nice spikes.

2nd edit. A non-scientific experience: I've been working with visual effects in the movie industry for almost 25 years. Back when computers were slow, I once tried to down sample 2K images (2048x1536 pixels) to SD Pal (720x576). That removes 8/9 of the image information. Then I upscaled the SD Pal material back to 2K, using a Lanczos algorithm, that uses ringing to introduce sharpness while up sampling. The difference when you blinked between the original and the resampled images were amazingly small. In the cinema, most people won't notice the difference. I think it's due to two things. First our visual system (eye/brain) fills in the missing details, just the way it compensates the eye's blind spot, and secondly you can store a lot of information in a pixel. When you upscale and sharpen, with CI images, you get the information smeared out, and then sharpened. Hope this bable makes sense ;)
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: georg.viehoever on 2014 September 13 10:58:50
Do you know this paper http://arxiv.org/abs/1406.1528 ? They are using ranked based statistics to estimate the pixel values, even if the images are non-linear.
Georg
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 13 11:31:17
Do you know this paper http://arxiv.org/abs/1406.1528 ? They are using ranked based statistics to estimate the pixel values, even if the images are non-linear.
Georg

Hi Georg.

No i didn't now that one. I'll give it a try :) I've been thinking about making an image of a dark area of the sky, like behind M51, using this method, to see how deep you can go with a lot of exposure.

Right now, I'm working on M45, which is my favorite part of the (Northern) night. I'm going to try to be extra thorough with that one, trying to keep the nebulosity soft but detailed, and the stars as small as possible, so that hopefully IC396 will be clearly visible. I'm using 252 images, and aim for 100 Mpix.

Cs (not here  ;))

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 13 21:02:08
@Georg

Thanks for the link. Very interesting article! I've only read it quickly so far but there seems to be several good ideas that I'll have to try. My goal has primarily been visual, but we share the same problems, and the way they enhance is especially interesting.

Morten  :)
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 13 21:13:03
i have been messing around with this with small #s of images (maybe 20 or so) and i'm finding that the mask integration does not completely flatten the image. i have not really spent any time thinking about what's wrong though.

also it seems like if you have "clip low pixels" turned on in ImageIntegration then you'll get an image which is equivalent to the integration of all the mask images in the low clipping output image. again not totally sure if it's 100% the same.

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 14 05:08:31
@Rob

I think 20 images is to little. From what my own (visual and non scientific) experiments show, 50 is a good minimum number. 50 images also seems to be the minimum amount, for getting similar colors on different stacks (like Andromeda shown above).

Using smaller numbers of images can be done, but then you will have to preprocess them separately before stacking. Just a simple flattening of the field (ABE, or DBE depending on your time), LinearFit and ColorCalibration will help, but that is more time consuming than gathering 30 images more. Also you loose the interesting averaging of the colors.

Great if the concept is catching on! :) That was sort of my idea. That the sky is very big, and I can't make Crowd Images of it all. I'm looking forward to see what you get out of it. Also sharing experiences is great for learning. I've been holding lecture's about the subject (and smaller collaborations) to danish amateur astronomy societies, and among others, a guy like Johannes Jensen (he's at Astrobin) quickly caught on.

@Georg

Hope you haven't been trying to find IC396 in The Pleiades. When I wrote it, my head was spinning full of numbers. Of course i meant IC349:

(https://lh5.googleusercontent.com/-2BksEk1qsPY/VBWEtXPi0uI/AAAAAAAADQY/iRYectuOD0U/s512/Screen%2520Shot%25202014-09-14%2520at%252013.09.54.png)

Cs

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 14 16:41:17
whoops - the low rejection map has to be inverted in order to equal the integration of the masks.

i have to say the hardest part of this process is scouring the web for appropriate images. obviously dustin already solved this problem and i guess the actual reason why astrometry.net was his PhD was to get the community to "do his work for him" for the real projects like comet orbit extraction, etc. but a quick trip to astrometry.net yields a whole lot of duplicate images, with most of the unique ones being unsuited for this purpose (at least on M51, more popular targets may fare better). interestingly when you land on one of the image pages, there's an "Enhance!" button which i suppose runs the algorithm in the paper that Georg pointed to. the results aren't that hot. but i suspect it does not do too well unless you really guide it with good input images.

rob


Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 14 17:40:26
@Rob

The hardest work, as you mention is gathering the data, and writing down all the credits (I use a spreadsheet). The most time consuming, processing wise, is the alignment. Start with popular object if you want to publish it. I've also made some CI, using whatever best (also copyrighted) pictures I could find (including some of yours :-[), but those were for my own viewing pleasure, and interesting enough, they weren't much better than what I've been able to pull out of Creative Commons images, as long as I use a lot of them.

From back when I started working with digital images, I've always hated noise. Every time you try to work with a noisy image it's uphill. Also, I noticed that Drizzle can be a strong tool together with sharpen enhancements. I have a very visual approach to this, and a personal philosophy, that all the information is in the smear, I just need to get it out. LocalHistogramEqulization seems to be a pretty powerful tool, but because the SNR is very high, and I oversampled heavily while working with the images, you can go much more to some extremes with some parameters, than you normally would.

To sum up:

Find as many images as possible.

Crop them to remove edges.

Select one image for a TransformMaster and resample it to 2-400%.

StarAlign using Distortion Correction (remember to enable 2-D Surface Splines!), enable Generate Masks, and leave everything else default for starters.

Integrate the xxxxx_r.fit files, using Average, No Weights, No Normalization. Disable all Pixel rejection! Also disable Evaluate Noise and select "Average absolute deviation..." as Scale Estimator (speeds up things). Rename to RGB_Stack.

Next integrate the xxxxx_m.fit files, again using the above parameters, but this time chose "Iterative K-sigma..." as Scale Estimator (the only one that works). Rename to Masks_Stack.

Finally use PixelMath to generate a new image using the simple formula RGB_Stack/Masks_Stack.

Bingo! :)

I've not had time to thoroughly read the article yet, but my next idea is to start out with a very wide field image of Cygnus, and find as many images that I can covering parts of that area. As I wrote earlier 50 images seems to give a good result. You can check your coverage, by simply measuring the K-value on the stack. It'll be 1,0 in the center, and should be at least 0.5 at the edges if you use 100 images in total. With the Cygnus Mosaic, I presume I'll have to go down to lower coverage in some areas.

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 14 21:00:10
yes i just scraped about 75 images of m42 off of flickr without regard to license so yeah, i won't be posting the result anywhere...

no problem using my images, especially for a project like this! i pretty much never set the license type "properly" on any service, and that's mainly an oversight/laziness.

anyway the result is quite remarkable; as you say the SNR is very high and the data lends itself well to all kinds of sharpening techniques. the most interesting thing to me is that the color apparently converges to the "correct" colors, which came as a surprise. i am starting with images that look "right" to my eye though.

i think i hit a couple of PI bugs along the way. at one point i used the Divide process to "flatten" the image and PI hung; had to attach to gdb to unwedge it. and then i tried creating the drizzle files and it hung during ImageIntegration... need to narrow these down and report them.

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: alvinjamur on 2014 September 15 15:38:57
This idea is fantastic, even more the processing!!!!!

Folks here that are playing with the idea : could u kindly "crowd" write a workflow for this?

Technically, u would likely need about 32 images with a decent enough fwhm....but the s/n would go lower the more u add. I'm wondering about the way in which the slope of the s/n curve would tail off as a function of fwhm.....ummm....
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 15 15:52:26
@alvinjamur

Thanks

I live in a very light polluted city center, with a lot of clouds and bright summer nights, so I have to find alternative ways of collecting data. Meanwhile I keep a single project running on the telescope (currently HH555) shot through both an UHC and a red kodak visual filter. Not optimal  ;) Gotta have those Ha, SII and OIII filters soon.

Here's a link to M45:

http://www.astrobin.com/120897/ (http://www.astrobin.com/120897/)

I deliberately didn't stretch the image, so the best way of viewing it is to follow the download description, load it into PixInsight, stretch and zoom.

cs

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: marekc on 2014 September 16 12:54:33
I have to admit, I'm really quite amazed by this approach to imaging! The results that MortenBalling has posted look really good to me.

When I looked at the M31 image, I checked the credits to see if my image was in there... it wasn't, IIRC. That's neither here nor there, but it made me think about something: Images like these may have a bit of a `Google Earth effect' for some viewers. When people first use Google Earth, they immediately try to go find their house. When people go up in an airplane, they say "I can see my house from here!" When a crowd-sourced image gets posted, people who've posted their own individual images will probably check the credits, to see if their work is in there. Interesting!

This has also made me think about licenses and copyright more than I had before. I'll have to go check the default CC license setting on my (exceedingly modest) little imaging blog. Maybe I didn't allow people to modify the images, or something like that. I never thought much about that until I saw this thread.

What a neat technique! I don't know if there enough M33 images out there to make a good M33, but I'd love to see that. It's a personal favorite of mine.

It also occurs to me that crowd-sourced images are kind of like mineral deposits or oil fields. All of the easy stuff will be gone through fairly quickly. The supergene deposits and big surface anticlines were all discovered long ago. In some cases, those mining districts and oil fields still yield commodities today. That's kind of like if someone decided to apply really advanced PI processing to the crowd-sourced M31 data, to try and squeeze a slightly better result out of it. The rarer deep-sky objects will be like the more recently-discovered resources. Yes, they're there, and we can find them and extract them, but it's much harder to do so. Like trying to make a crowd-sourced image of some obscure Sharpless object that has very few publicly posted images. It could, technically, be done, but it would be very hard to do. In the case of minerals and oil, sufficient demand/scarcity/high price can make a `tough' deposit worth going after - there's a profit motive, at least at certain times. But with crowd-sourced imaging, the lack of a profit motive might, I'd guess, provide a fairly strong dis-incentive against trying to do the tough stuff via publicly-posted images. At a certain point, it's easier to just buy one's own gear, go on a road trip to a dark site, and shoot that sucker oneself.

It's like a little economics lesson! What a fascinating thing - I'm glad someone did this!

- Marek
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 16 14:10:47
@Marekc

Interesting points.

I'm sorry I didn't use your images. As mentioned earlier, I started doing these images as an experiment, mainly to test the concept, using whatever images I could find, so I probably did anyhow. I ended up with something I called Andromeda500 with about 550 copyrighted images, in 200 Mpix, and went "WOW!". I love zooming into large images and finding details. With Andromeda, I also just gazed into the field trying to see if I could see the Big Bang ;) (not really).

Anyways, I was astonished by the Crowd Imaging potential, and sad that I couldn't share it with anyone. So much that I started looking for loopholes in the copyright laws (they are actually there, if the result is research related). Then I realized that the quality of the separate images didn't seem to matter that much to the final result. The main thing is lots of data. Of course I got better results with sharp, deep highres images, but as I mentioned, more than 50 random images produce very similar results. That was the time I started to look for Creative Commons images, and considered starting all over.

I picked some of the most popular objects (M31, M42, M45 etc.) as we've all tried imaging those. Therefore there were many images, and also the results are easier to evaluate for anyone. Recently I was looking at SkySafari trying to figure out what to shoot one night, and saw The Draco Dwarf Galaxy. That framed perfectly with my rig, so I shot that three nights in a row. On the final stack I could see a fuzzy, but to be honest it was probably noise and a non-flat field. So I went to the god old interweb, and found all the images of Draco Dwarf that I could find. The result was the most detailed image of the very faint galaxy I've ever seen. Again I had this sad feeling that more people should see it, but I'm afraid I'll have to settle with the fact, that at least I've seen something nobody else has. Pretty mindbogling :D

Btw. There are a lot of M33s out there, so knock yourself out and try it. I've done one, and it's a beautiful galaxy! I might also do one with CC images in the future, but I'd rather try some more exotic objects next.

In the long run I hope this idea catches on. Georg's link shows, that I'm not the only one who thought about it, even though I started before those guys. The JNER1 collaboration I participated in, was an idea I had, because once again I had spent many nights shooting that with no result, before I realized that I had to work with others, to capture enough photons. I later found out, that others had done collaborations as well. The Crowd Imaging was just an extrapolation of collaborations.

What i really hope is that people start sharing their data. Just 100 amateur astronomers with newtons, refractors, RCs and SCTs spending one night on the same object would be awesome, but very hard to organize. Searching the web for useable images is much more achievable. Also a dedicated place to upload data, like Astrobin, with easier search routines would be cool. There are a lot to be found out there, but searching for images is pretty time consuming.

Finally, the method leaves a lot of room for improvement. So far I weigh all images equally. Both because it's "easy", but also to try to be fair to the people spending their nights catching faint light. They all work equally hard. But to be honest, some pictures are less noisy, sharp etc. than others. Fair enough. I know how hard this is. A better way of evaluating and weighing images would still be a major improvement. I've tried, but PI (or me?) gets pretty confused, and I haven't tried making scripts (yet).

Cs

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: georg.viehoever on 2014 September 16 22:44:26
...
What i really hope is that people start sharing their data. Just 100 amateur astronomers with newtons, refractors, RCs and SCTs spending one night on the same object would be awesome, but very hard to organize. Searching the web for useable images is much more achievable. Also a dedicated place to upload data, like Astrobin, with easier search routines would be cool. There are a lot to be found out there, but searching for images is pretty time consuming.
...
That also sounds like an interesting idea. Especially if people would share their RAW, or at least their linear (i.e. calibrated+stacked) data. This could more easily be stacked than already processed images. I wonder if platforms such as astrobin.com would provide the necessary functionality - I never looked into this.
Georg
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 17 03:13:24
@Georg

I think Salvatore is already up to his neck in files. It's an amazing site, and a huge job that he does (for free). Also Fits file sets are a lot heavier on the server than jpg's, but Moore's Law works for us.

Linear data would be very nice, but nonlinear are actually not that bad for this technique. At least for the visual side of things. I'm trying to figure out how to analyze the jpg files, to evaluate the amount of useable information stored in each image, and hopefully it suddenly pops up in my head. Things normally do.

In the meantime I would again like to urge people to use Creative Commons licenses for their astro images if they don't plan on selling them.

Morten  :)
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 19 13:24:48
well astrobin can and does host fits, for a fee. you can publish them publicly or keep them private…

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 20 08:01:45
@Marek

Here's a M33 for you:

http://www.astrobin.com/121906/ (http://www.astrobin.com/121906/)

Morten ;)
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: marekc on 2014 September 20 12:51:09
Thanks, Morten!

That's a nice-looking image  ^-^  It's got a lot of resolved stars, plus HII regions and the faint outer portions of the galaxy. Good `ol M33!

- Marek
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 28 07:42:30
A little more info.

The images used for this technique doesn't have to be perfect. To illustrate that I've recently made a CI of The Rosette Nebula.

Here is a Youtube clip with the different images used:

http://youtu.be/KbYGnDrj43Q (http://youtu.be/KbYGnDrj43Q)

And here is a detail from the final image:

(http://cdn.astrobin.com/images/thumbs/2ff0616d82149e413f940f5c9ff3560b.1824x0_q100_watermark.jpg)

Image Credits (Thanks for sharing!):

Adam Evans, aknotwot, Andolfato, Andrea Pistocchini, Andreas Fink, Asaf Braverman, AstroGG, ASTROIDF, Ben Browning, Carsten Frenzl, Chris Lasley, Chris Madson, Claustonberry, Fernando Fogel, Francescodib, FranckIM06, Fred Locklear, Jan Curtis, Janos Barabas, Javier R., John Lanoue, john.purvis, Luc Jamet, Luis Argerich, Luis Martinez Martin, Marcelo Domingues, Mauro Narduzzi, Michael Karrer, Mike Markiw, Mike Markiw, milosz, neptun, nicoairbus, NicolasP, pascvale13, Pat Gaines, Paul T., petersdome-observer, pfile, Phil Hosey, Phil Hosey, Ralph W, Ram Viswanathan, Rhett Herring, Richie Jarvis, s58y, Salvatore Iovene, Salvopa, Serge, Shekhar Phatak, Stan McQueen, Stephen Rahn, Steve Yan, stevebryson, stseiya, Tim.

Full version: http://www.astrobin.com/122809/0/ (http://www.astrobin.com/122809/0/)

Image @100%: http://www.astrobin.com/full/122809/0/?real=&mod= (http://www.astrobin.com/full/122809/0/?real=&mod=)

Another thing I've been wanting test the method for is Deep Field. Because the final image stack has such high SNR, it can be stretched enormously. A negative luminance of NGC3628 shows QSOs out to (at least) 10-12 billion lightyears away (Z=2,4):

(http://cdn.astrobin.com/images/thumbs/4efa4f09b606289f3dfcccac4bdf981a.1824x0_q100_watermark.jpg)

Image Credits (Thanks for sharing! :)

Adam Evans, Alvinillo, Angel Requena, Anton, Armelle & Eric, AstroGabe, AstroGG, Astroluc63, Ben Browning, Bob Familiar, budman1961, Cano Vääri, Carsten Frenzl, chripell, Cobbler, Cody Garges, Creedence, David L Milligan, Eduardo Mariño, Eric Gorski, Ferran Ginebrosa, Flavastro, fragro, FranckIM06, Fred Locklear, Fredrik Ödling, Fryns, geco71, Gerardo Blanco, harbinjer, Heiko Günther, Hewholooks, jdiwnab, John Bowles, Jorge A. Loffler, jpstanley, Juan Lozano, Jürgen Kemmerer, Jussi Kantola, Ken_Lord, Ljubinko Jovanovic, Luca Argalia, M.W.Hoy, Marc Van Norden, Marcelo Domingues, Matthew, Maxvlt, Miodrag Sekulic, Morten Balling, NicolasP, PaulHutchinson, Pavel (sypai) Syrin, pbkwee, pbkwee, Pete Collins, Peter Williamson, pfile, Phillip Seeber, Ram Viswanathan, Rhett Herring, Richie Jarvis, RIKY, Rob Glover, Roberto Ferrero, s58y, Salvatore Iovene, Salvopa, samuele, Sendell, Serge, Shane Poage, Stephen Rahn, Steve Elliott, Steve Yan, stevebryson, Surfus_1980, theilr, Tim, Tom Harrison, Vincent Bhm, Wayne Young, zemt-fr.

Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: Josh Lake on 2014 September 29 19:49:32
Morten, I'm a bit late to the party here but I am positively blown away! You've got to take this technique on the road to astronomy conferences!

I strongly believe that astrophotgraphers these days are far more willing to share, especially if their data sets are being credited and used in something excellent like this.

I'd like to get involved, I'd like to spread the word, and I'd love to process some of these beautiful data sets. I'm going to go back and read your article and try it for myself, but are you also willing to post some of the merged data sets?

I'm a moderator over at Reddit's /r/astrophotography (http://www.reddit.com/r/astrophotography/) forum and we tried something like this on M42 with bad results. But with your technique, the sky's the limit. Would you allow me to link to your post here and your work on Astrobin? Or would you like to post something yourself? With a community of 10's of thousands of imagers, some with really excellent data sets, I think there might be a gold mine in the making.

Thanks so much for posting, I'm bursting with the excitement and possibilities!
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 30 04:36:52
Hi Josh

Thanks :)

First of all: You are more than welcome to spread the word. That was my idea about this thread. I've tried sending some images to APOD and to Phil Plait (Bad Astronomy), not to brag, but to try to spread the idea, but I don't think the really understand the concept (yet). The more people getting involved the merrier! (The sky is big).

You can link to anything I do, and use the images as you like. I think crediting the participants are important with this (currently looking at incorporating credits in the bottom of the images). All my CI images are made with Creative Commons files, and so they are automatically also CC. Also I encourage everyone interested to try the technique themselves. The biggest problem is finding enough images, but for testing and practising, you can use copyrighted material as well. Just don't publish those. I personally use that a lot, and I've seen things you people wouldn't believe ;)

If I can assist you in any way please let me know. If you send me a PM with you email, I can send you some of the raw stacks, so that you can try to process them.

I've been trying to spread the word about the idea myself. In Denmark (where I live), I've made a few lectures to some of the astronomical societies, and people are starting to, at lest, do collaborations.

The idea came to life after I suggested a collaboration of the planetary nebula JNER1 here in Denmark. I didn't know about collaborations back then, but the idea seemed pretty straight forward. To bring out details and a deep field, you need lots of exposure time, and parallax is not a problem with so distant objects as we image. At the same time I noticed that we're all shooting the same Messier objects over and over, so I started thinking about how much exposure time that added up to. A lot!!!

The first tests I did looked very promising but because the images were so differently framed, I had problems with the edges of the field, unless I cropped a lot. That brought out the idea of making a sort of flat frame with the coverage of each image, and using that to make a simple flat compensation (by division).

Another thing that I really like about CI is that it's kind of an opposite to something I've seen a lot of in the astro photo community. Everybody wants to be the best. Better than Nasa, better than Robert Gendler, best at Astrobin and so forth. I think that's a shame. When I started out with a DSLR on a tripod, I was amazed at what you could do. Once you get better you start to make challenges for yourself, and the natural reaction is to be proud if you succeed and wanting to show it to everyone. Problem is that I think most of us forgot that this is mainly something we do for fun (and searching for QSOs ofcurz ;)). It is great if somebody else can have an experience with what we create with the telescopes/cameras, but once you've seen 100+ M42's they start to look the same. If you add them up they suddenly become something special, and everyone participating can have a feeling of having been involved in the final result. Together we're strong, and all that...

Best Regards

Morten
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: pfile on 2014 September 30 08:55:51
I personally use that a lot, and I've seen things you people wouldn't believe ;)

like attack ships on fire off the shoulder of orion?

rob
Title: Re: Crowd Sourced Astro Images (Crowd Imaging or CI)
Post by: MortenBalling on 2014 September 30 08:57:21
Yep! That and much more...