PixInsight Forum

PixInsight => Image Processing Challenges => Topic started by: vicent_peris on 2012 March 05 08:14:55

Title: ALHAMBRA Survey, field 7
Post by: vicent_peris on 2012 March 05 08:14:55
Hi all,

I have just uploaded a set of images of the ALHAMBRA Survey (http://alhambra.iaa.es:8080/alhambra/). This is a cosmological survey containing about 600.000 galaxies down to mag 26.5 in four square degrees of the sky, acquired with  the 3.5 meter Zeiss telescope at Calar Alto Observatory (http://www.caha.es/CAHA/Telescopes/3.5m.html). The data set is composed of 23 images which have been registered and flux equalized. 20 out of the 23 are midband image acquired with 30 nm wide filters; these images have been acquired with LAICA camera (http://www.caha.es/CAHA/Instruments/LAICA/index.html). This camera has four, 16 megapixel sensors located at the prime (f3.9) focus, and these images correspond to the ones acquired by one of these four sensors. The filter system is described in page 5 of this PDF article (http://alhambra.iaa.es:8080/alhambra/Documents/Publications/Jenam_moles.pdf). The other three images have been acquired with Omega2000 (http://www.caha.es/CAHA/Instruments/O2000/index.html), a 4 megapixel infrared camera, with the standard wideband J, H and K filters.

The challenge here is to combine a RGB image. Good luck!
Vicent.
Title: Re: ALHAMBRA Survey, field 7
Post by: Josh Lake on 2012 March 05 17:35:45
I'm very interested in trying to work with this kind of data. Are the files on the PixInsight files server? I still haven't figured out how sharing is done with this system... do I have to ask you to share me in or can you post it somewhere for anyone to grab it?
Title: Re: ALHAMBRA Survey, field 7
Post by: Alejandro Tombolini on 2012 March 05 19:42:17
Nice challenge Vicent, I have no idea how to start but I'll make a try. The data is very interesting.

Saludos. Alejandro. 
Title: Re: ALHAMBRA Survey, field 7
Post by: Juan Conejero on 2012 March 06 00:27:33
Hi Josh,

Quote
Are the files on the PixInsight files server? I still haven't figured out how sharing is done with this system

They are on the Forum Shared Files (FSF) repository.  To share your files, just create a new folder on FSF and upload them to it. Everybody has read/write permissions on FSF (with the exception of the folders where the original data have been stored, which I've protected by removing the write permission).

I am still trying to configure ajaxplorer to improve the way we can use the PixInsight Files service. It has severe limitations to share files with all users, which I've confirmed with the author. Despite these limitations, ajaxplorer is the best online file management package I've tested. I think we can use it with FSF for now.

Something that I am investigating now is the possibility to allow all PI users to use PixInsight Files automatically, without having to email us. This requires writing an ajaxplorer plugin to provide external user authentication. There are some technical problems but I am working on it. Once I have this sorted out I'll announce PixInsight Files officially to all PI users, and it will be available from the main website.
Title: Re: ALHAMBRA Survey, field 7
Post by: vicent_peris on 2012 March 06 01:24:14
Hi Josh,

Quote
Are the files on the PixInsight files server? I still haven't figured out how sharing is done with this system

They are on the Forum Shared Files (FSF) repository.  To share your files, just create a new folder on FSF and upload them to it. Everybody has read/write permissions on FSF (with the exception of the folders where the original data have been stored, which I've protected by removing the write permission).

I am still trying to configure ajaxplorer to improve the way we can use the PixInsight Files service. It has severe limitations to share files with all users, which I've confirmed with the author. Despite these limitations, ajaxplorer is the best online file management package I've tested. I think we can use it with FSF for now.

Something that I am investigating now is the possibility to allow all PI users to use PixInsight Files automatically, without having to email us. This requires writing an ajaxplorer plugin to provide external user authentication. There are some technical problems but I am working on it. Once I have this sorted out I'll announce PixInsight Files officially to all PI users, and it will be available from the main website.


It would be very nice to access the files from the navigator inside PixInsight and drag and drop the files to the workspace.  O0


V.
Title: Re: ALHAMBRA Survey, field 7
Post by: vicent_peris on 2012 March 06 01:27:00
Nice challenge Vicent, I have no idea how to start but I'll make a try. The data is very interesting.

Saludos. Alejandro.


The key is in PixelMath. Build your own RGB palette, it's only a matter of multiplying and adding images. :) A lot of different palettes can be constructed from different documentary arguments.

V.
Title: Re: ALHAMBRA Survey, field 7
Post by: oldwexi on 2012 March 06 09:25:40
Vincent!
Thanks for uploading such great data to which i never would have access to.
Great idea.

A question:
I let the 20 narrowband images just run through PI-animation and got the naive impression
a few images are not 100% calibrated. on some images the borders of the 4 cameras can be seen
and for example the image with the 536nm to 567nm filter looks like flat fielding
did not succeed or is missing? Do you have the flatfields per filter also available?
Or, am i expecting too much cleanness from such deep data?

Thanks for giving me the possibility to play with this data!

Aloha
Gerald
Title: Re: ALHAMBRA Survey, field 7
Post by: vicent_peris on 2012 March 06 10:05:26
Vincent!
Thanks for uploading such great data to which i never would have access to.
Great idea.

A question:
I let the 20 narrowband images just run through PI-animation and got the naive impression
a few images are not 100% calibrated. on some images the borders of the 4 cameras can be seen
and for example the image with the 536nm to 567nm filter looks like flat fielding
did not succeed or is missing? Do you have the flatfields per filter also available?
Or, am i expecting too much cleanness from such deep data?

Thanks for giving me the possibility to play with this data!

Aloha
Gerald


Hi Gerald,

Yes, that's true. LAICA is not a nice camera, and it has problems with flatfielding. All the images have been calibrated by the Survey staff. At the end, when you combine the 20 filters, these problem are greatly reduced, although you'll get a very noisy and patchy background.

Regards,
Vicent.
Title: Re: ALHAMBRA Survey, field 7
Post by: Alejandro Tombolini on 2012 March 06 17:23:36
At the end, when you combine the 20 filters, these problem are greatly reduced, although you'll get a very noisy and patchy background.
Regards,
Vicent.
Thats right. This is being an extremely interesting challenge...
Title: Re: ALHAMBRA Survey, field 7
Post by: Alejandro Tombolini on 2012 March 08 17:24:18
Hi,

The image 644 has some background information that does not seem to be a pattern of the camera and I assumed that they are not clouds, perhaps a wrong assumption. If so, the following is not all valid.  :-\

What I did was a combination, only the sum, of:

B = 365 +396 +427 +458 +489
G = 520 +551 +582 +613
R = 644 +675 +706 +737 +768 +799 +830 +861 +892 +923 +954 + H + K + J

For the combined image:
ABE
DBE
HT (STF)
ACDNR to Crominance
SCNR to green

After that I separated the stars and created a mask from the background image 644.
With the mask protecting the background in the main image applied InterChannelCurves with target in CIE L and  CIE C and the reference channel in green.
After that, in the same image without stars:
LHE
CurvesTransformation
CurvesTransformation to stars with Lmask.
PixelMath to add stars again
Erosion (a little bit)
SCNR

The result was too noise...
(http://2.bp.blogspot.com/-2IfmVQAhMgI/T1lYXcky9EI/AAAAAAAAAL0/cA8lGGHnEzw/s1600/Image04.jpg)

Crop 1:1
(http://3.bp.blogspot.com/-tuoJzt2WAPk/T1lYaYxTMwI/AAAAAAAAAL8/r2h160LLQXo/s1600/Image04_Preview01.jpg)

Then I built a luminance with the integration of all images. No rejection
MMT
HT (STF)
LHE
And combine with the image.
ADCNR to Crominance
 
I tried to "repair" the defects in the bottom left corner with masks made over the ones that had the defects (520 and 635) and applying HT

The final:
(http://2.bp.blogspot.com/-MI-dqQ9xNRQ/T1lZLgBMBsI/AAAAAAAAAME/e_UpFABoE8o/s1600/Image05.jpg)

Crop 1:1
(http://4.bp.blogspot.com/-FNbaGmZLqag/T1lZN2zR9LI/AAAAAAAAAMM/lYdAX_MJvMY/s1600/Image05_Preview01.jpg)

Saludos. Alejandro.
Title: Re: ALHAMBRA Survey, field 7
Post by: Josh Lake on 2012 March 09 05:50:50
Well done, Alejandro! I have been waiting to get through my work projects so that I could play with this data over the weekend. You have set a nice, high bar to start.   :D
Title: Re: ALHAMBRA Survey, field 7
Post by: Joe DePasquale on 2012 March 12 12:52:47
Hi all,

 What a great data set to practice with! Thanks Vicent!

 For my combination, I wanted to get a good color balance.  Technically speaking, this should be a very red image given the filters used, but rather than constrain the filters to their associated colors, I spread them across the RGB channels evenly to give a more natural color balance to the image.  I did end up using PixelMath for this with the following parameters:

R=830+861+892+923+954+J+H+K
G=613+644+675+706+737+768+799
B=365+396+427+458+489+520+551+582

 After the RGB combination, I did some background neutralization as well as color calibration using a selection of spiral galaxies (and the preview aggregator script) as my white reference.  After a round of ABE subtraction, I used Histogram Transformation to de-linearize the image.  I then tried the old trick of using the luminance channel in LRGB for chromatic noise reduction, and color saturation followed by SCNR to remove the green cast.  A slight curves adjustment got me to a point where I wasn't satisfied with how the faint sources were showing up.  I then extracted a new lum channel and overlaid as an inverted mask, to protect the cores of the bright stars and did another round of Histogram Transformation to bring up the faint stuff.  This last step increased the noise a good bit, so a subtle use of ATrousWaveletTransform helped with that.  I also tried to reduce some background variation with another round of ABE to less than desired effect (I still see some variations).  One final curves tweak and here it is. 

(http://dl.dropbox.com/u/1360537/jdepasq_alhambra.jpg)

Thanks,
Joe
 
Title: Re: ALHAMBRA Survey, field 7
Post by: Josh Lake on 2012 March 12 14:36:39
Ah, a wild imaging expert appears!  ;)

Beautiful job, Joe, and sound logic for the combination in PixelMath. I am going to take a crack at this with similar parameters. What was your rationale for picking the specific galaxies for white reference that you chose?
Title: Re: ALHAMBRA Survey, field 7
Post by: Joe DePasquale on 2012 March 12 20:12:39
Ah, a wild imaging expert appears!  ;)

Beautiful job, Joe, and sound logic for the combination in PixelMath. I am going to take a crack at this with similar parameters. What was your rationale for picking the specific galaxies for white reference that you chose?

Thanks, Josh!  To take this one step further, I would've brought it into Photoshop to clean up those artifacts in the bright stars, but I wanted to keep it 100% PixInsight. 

The white reference galaxies were just chosen quickly by eye, I was looking mainly for spirals that appeared almost white in the auto stf.  There's a large spiral, almost edge on in the top third of the image, right of center that was a good addition to the references. 
Title: Re: ALHAMBRA Survey, field 7
Post by: Juan Conejero on 2012 March 13 02:13:48
Very nice result, Joe!

Quote
I would've brought it into Photoshop to clean up those artifacts in the bright stars, but I wanted to keep it 100% PixInsight.

Actually, you can achieve both things---I mean cleaning up and 100% PixInsight :) Do you know PixInsight's CloneStamp tool?
Title: Re: ALHAMBRA Survey, field 7
Post by: Joe DePasquale on 2012 March 13 07:03:21
Thank you, Juan!  I am familiar with the CloneStamp tool, and although it's great for bad pixels and streak type artifacts, I find it to be fairly limited when it comes to repairing star blooms.  Perhaps there's something I'm missing with it.  I'd love to hear a new perspective on it. 
Title: Re: ALHAMBRA Survey, field 7
Post by: Joe DePasquale on 2012 March 13 12:31:10
Attempt number 2!

2 Major changes on this attempt:

[1] I finally gave Deconvolution a try...something I've been meaning to do for a long time.  I used this post from Juan as a guide:
http://pixinsight.com/forum/index.php?topic=2727.msg18512#msg18512 (http://pixinsight.com/forum/index.php?topic=2727.msg18512#msg18512)
I think it produced pretty decent results, but it was a challenge to deal with the small scale noise brought up.  A few rounds of ATrousWaveletTransform helped with that.

[2] I had a major epiphany in using the CloneStamp tool for removing those star bloom artifacts!!  (Thanks Juan for leading me down this path).  I just had to adapt my Photoshop brain to PixInsight to make this work.  The trick is to make a preview of the star that you want to fix, pull it out into its own window, and then rotate it 90 deg and use the unbloomed diffraction spike as a replacement for the bloom in the original image using the clone tool.  It works surprisingly well! 

Here's the new version:
(http://dl.dropbox.com/u/1360537/jdepasq_alhambra_v2.jpg)
Title: Re: ALHAMBRA Survey, field 7
Post by: Juan Conejero on 2012 March 14 10:10:39
Hi all,

Here is my take on these wonderful ALHAMBRA data. I have been interested exclusively in finding new ways of combining the data, not in producing a finished image. I have explored several interesting possibilities and (inevitably!) have ended writing a small JavaScript script that performs the whole task of loading the images, computing the weights of each ALHAMBRA image for individual RGB filters, and synthesizing the final RGB image. The script is a bit long to include it inline (approx. 500 lines) so you'll find it attached to this post.

In order to evaluate the results some mouseover comparisons are very useful, which are not possible in a forum post. For this reason I have written a brief document in PIDoc format, where you'll find complete information on my approach to this task:

http://pixinsight.com/doc/docs/ALHAMBRA-1/ALHAMBRA-1.html

(you'll need an SVG enabled browser, which does not include Internet Explorer before version 9).

These are three crops with the best (IMO) filter set defined in the script (extended filter set):

(http://forum-images.pixinsight.com/20120314/ALHAMBRA/extended_image_Preview01.png)

(http://forum-images.pixinsight.com/20120314/ALHAMBRA/extended_image_Preview02.png)

(http://forum-images.pixinsight.com/20120314/ALHAMBRA/extended_image_Preview03.png)

I would like to draw your attention to an interesting fact: a visual representation (what is often called 'natural colors', whatever that means) is not appropriate to represent this ALHAMBRA dataset. In fact, the RGB filter set that I have called 'visual set' is much better than what visual perception actually is, and even with this concession the result is very poor. The best filter set is the extended set---I hope everybody will agree here---because it is the one that maximizes information representation. This is the documentary criterion that defines our deep-sky color calibration methodology.
Title: Re: ALHAMBRA Survey, field 7
Post by: Joe DePasquale on 2012 March 15 09:31:56
Hi Juan,

I have to say I'm fascinated with this approach!  Not only did I download and run your code, I went as far as to print it out and review it by hand, and run a few filter weights on my calculator to really understand the concept.  As a side note, I had no idea it was possible to create plots in PixInsight - how cool!

I just did a side-by-side comparison of your approach to my approach of giving all filters equal weight and just running them through pixel math (as referenced in my previous post).  Your approach gives you a very clean, color balanced image right off the bat, whereas my image requires a lot of color calibration before it even starts looking natural.  I totally agree that the 'extended filter' set is clearly the best option because it truly utilizes all of the available data while adhering to a rigorous approach to color. 

Just out of curiosity, how did you choose the central wavelength for the various filter sets (and specifically the extended set)?  I often find myself combining data from many different wavelengths (Radio, IR, optical, UV, X-ray, etc) often in the same image (I'm the Science Imager for NASA's Chandra X-ray Observatory).  How would you adapt your approach here to something like that?  Often times, when working in such disparate spectral regimes, you don't see the kind of overlap of morphological structures as you do here with the Alhambra dataset.  I wonder how that would affect the choice of spectra weights?

Many thanks for such an informative post!
-Joe
Title: Re: ALHAMBRA Survey, field 7
Post by: Alejandro Tombolini on 2012 March 15 17:06:40
Wonderful! not only the good final result, also the work you did explaining it. Thank you!

I have compared the result of the script (the extended image) with my approach and I have intense color in some galaxies due the lack of information of the other channels, and the big noise that I notice was considerable reduce due to the inclusion of more images.

Now is very noticeably the importance of having in consideration the weight of each filter but I would not have happened!
 
It was funny and very instructive, when the next challenge?

Saludos
Alejandro.
Title: Re: ALHAMBRA Survey, field 7
Post by: Juan Conejero on 2012 March 15 17:44:18
Hi Joe,

I am glad to know you like my script and the approach we have taken to render these data. Vicent Peris applied a similar concept when he processed the first ALHAMBRA fields (http://www.caha.es/alhambra-the-history-of-the-universe-at-sight.html) back in 2006.

Overlapping filter response curves are crucial to this color methodology, since they provide rich and dense chroma components without any gaps. Typical RGB CCD filters usually cover almost disjoint regions of the spectrum with little overlapping, which leads to poor color renditions. For the same reason, DSLR cameras tend to provide richer color representations. See for example the spectral response curves for some popular digital cameras:

http://www.maxmax.com/spectral_response.htm

Quote
how did you choose the central wavelength for the various filter sets

For the visual set I chosen the approximate maxima of 'natural color' curves: 450, 550 and 650 nanometers. These central wavelengths are very similar to those implemented by most DSLR cameras (see the page linked above). For the other filter sets I simply distributed the central wavelengths equally across the main ALHAMBRA filter subset (from 370 nm to 950 nm, approximately), trying to gather as much data as possible with each set. The purpose of these sets is not to emulate human color  perception, but to use color as a means to maximize differentiation of image structures, and hence of the different objects represented in the image.
Title: Re: ALHAMBRA Survey, field 7
Post by: Warhen on 2012 March 15 19:31:58
Joe, it's great to see you here buddy! Hello to all.
Title: Re: ALHAMBRA Survey, field 7
Post by: Lex on 2012 April 25 12:49:21
Hi all,

I think I did not get something, I am really burning to try the process of such a wonderful professional data but.... I do really not know where to find it?
Could anyone help me out please?

Thanks
Title: Re: ALHAMBRA Survey, field 7
Post by: astropixel on 2012 May 08 20:33:58
Yes, me too!