PixInsight Forum

PixInsight => Announcements => Topic started by: Juan Conejero on 2018 February 05 08:42:20

Title: New RAW Module - Ready for Testing
Post by: Juan Conejero on 2018 February 05 08:42:20
Hi all,

Today I have released a first version of the new RAW PixInsight module. RAW implements support for digital camera raw formats based on the LibRaw open-source library (https://www.libraw.org/). RAW replaces the old DSLR_RAW module, which was based on a custom adaptation of the (now discontinued) dcraw (https://www.cybercom.net/~dcoffin/dcraw/) program by Dave Coffin. The use of LibRaw provides us with new demosaicing methods, interesting post-processing features, such as new noise reduction algorithms, and the guarantee that we'll support the latest camera models now and in the foreseeable future.

Availability of the RAW module

The new module is stable and has been thoroughly tested. However, since this module provides critical functionality, and given the complexity and difficulty to test it on a large amount of different raw formats, I prefer to have it extensively tested in real-world scenarios before releasing it as an official update. For this reason you have to use our development update repository to install it. Do the following:

1. In PixInsight, select RESOURCES > Updates > Manage Repositories from the main menu.

2. On the Manage Update Repositories dialog box, click the Add button.

3. This will show the Add PixInsight Repository dialog. Copy and paste the following in the URL field:

https://pixinsight.com/update-devel/

4. Click the OK button on the Add PixInsight Repository dialog.

5. Click OK on the Manage Update Repositories dialog.

6. From the main menu, select RESOURCES > Updates > Check for Updates.

7. A new update with the RAW module should appear. Accept the update, then exit PixInsight to install it, and wait for the application to restart, as usual.

This update will replace the old DSLR_RAW module with the new RAW module. You'll get full support for the latest camera models, such as the Canon EOS-1D X Mark II, EOS 5D Mark IV, Fujifilm X-T20, etc., and added features for management of raw images.

Here is a screenshot of the RAW Format Preferences dialog box, showing all features available in the new module:

(http://forum-images.pixinsight.com/20180205/RAW-preferences.png)

New Demosaicing Algorithms

Besides the superpixel, half-size, bilinear, VNG, PPG and AHD demosaicing methods, which were already available in the old DSLR_RAW module, the new RAW module introduces the following demosaicing algorithms:

* DCB demosaicing algorithm (http://www.linuxphoto.org/html/dcb.html) by Jacek Gozdz.

* DHT demosaicing algorithm by Anton Petrusevich.

* Modified AHD demosaicing algorithm (AAHD) by Anton Petrusevich.

I suggest you review the In-Depth Demosaicing Algorithm Analysis (https://www.libraw.org/node/2306) article on LibRaw's website. Keep in mind that not all algorithms available in LibRaw have been included in the new RAW module. Unfortunately, I cannot include algorithm implementations released under the GPL license, since GPL (any version) forbids inclusion in proprietary applications, even when the relevant parts of the application using those implementations are released as open-source products (as is the case with the RAW module).

From the tests I have done, I have to say that the new algorithms, especially DCB, are great for daylight images. However, my personal view is that VNG remains the best option for underexposed data, including deep-sky astronomical images. I'd love to know your opinions and personal experiences in this regard.

FBDD Noise Reduction

After quite a few years in this image processing business, I don't get impressed easily. But I have to say that the FBDD noise reduction algorithm (http://www.linuxphoto.org/html/fbdd.html), by Jacek Gozdz, has been a nice surprise. I am still asking myself how on earth I didn't discover it before. This algorithm is simply fantastic to remove small-scale bright artifacts from mosaiced data without altering significant image structures. In the following example you can see how efficient FBDD noise reduction is to suppress hot pixels in an uncalibrated raw frame acquired with a Canon EOS 450D camera:

(http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D-tn.jpg) (http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D.png)
Click on the image for a full-size version (http://forum-images.pixinsight.com/20180205/FBDD-comparison-450D.png)

I still have to figure out how FBDD can be optimally integrated with our image preprocessing pipelines. For now, this noise reduction algorithm is available for postprocessing (demosaicing) of raw frames, and is enabled by default (with three FBDD iterations). The good news is that this algorithm works on mosaiced data, that is, before interpolation of CFA frames. This opens the door to integrating FBDD in the complete preprocessing flow, all the way from calibration up to drizzle integration. More news on this topic soon.

X-Trans Sensor support

The new RAW module fully supports Fujifilm X-Trans CFA frames. See how the X-Trans CFA pattern can now be loaded as pure raw data (in raw RGB mode in this example):

(http://forum-images.pixinsight.com/20180205/x-trans.png)

This means that once you install the new RAW module, X-Trans raw data can be calibrated. However, for a complete support of X-Trans CFA frames, we need to implement X-Trans CFA interpolation in the Debayer module (which, by the way, should not be called Debayer anymore after such update), so the StarAlignment and ImageIntegration tools can work to register and integrate the demosaiced data, respectively. The DrizzleIntegration module requires no change at all, since it is already able to work with X-Trans CFA patterns (with arbitrary mosaic patterns, actually). I will be working on this very soon.

Open-Source Module

As promised, the new RAW module has been released as an open-source product under PCL license. The entire source code is now available at our official GitHub repositories:

https://github.com/PixInsight/PCL/tree/master/src/modules/file-formats/RAW

A custom adaptation of LibRaw is also part of our PCL development framework:

https://github.com/PixInsight/PCL/tree/master/src/3rdparty/libraw


That's all for now. The RAW module, along with an updated Debayer tool with X-Trans support, will be included in the next version of PixInsight, which will be released soon for all platforms, including FreeBSD ;) I really need to know how the new RAW module works in the real world, so *please* let me know anything you find out as soon as possible, be it positive or not. Enjoy!
Title: Re: New RAW Module - Ready for Testing
Post by: Niall Saunders on 2018 February 05 08:53:30
Quote
. . . we need to implement X-Trans CFA interpolation in the Debayer module (which, by the way, should not be called Debayer anymore . . .

Hi Juan, so if we can't call it the Debayer module, then how about the "Desander" module - in memory of our good friend who put to much effort into it  :)
Title: Re: New RAW Module - Ready for Testing
Post by: Juan Conejero on 2018 February 05 08:58:29
;) I plan to include a special mention to Sander in the next version. No doubt he deserves it for his work on the Debayer module.
Title: Re: New RAW Module - Ready for Testing
Post by: Carlos Milovic on 2018 February 06 12:42:51
Hi Juan!

Does the new implementation support Canon's dual pixel raw frames? Have you seen that technology?
Title: Re: New RAW Module - Ready for Testing
Post by: Juan Conejero on 2018 February 07 00:35:49
Hi Carlos,

Nice to see you here! ;) Yes, with this new module, we support Canon dual pixel raw frames. For example, you can download this sample file:

http://www.imaging-resource.com/PRODS/canon-5d-iv/E5D4hSLI000100DPRaw_FINE.CR2.HTM

and open it with the new RAW module without problems. I am not sure about how LibRaw manages these dual image frames (we can investigate this). Note that the resolution of the interpolated image is indeed very high. On a side note, compare the results with the VNG and DCB demosaicing algorithms. DCB produces an interpolated image with virtually no color or ringing artifacts:

(http://forum-images.pixinsight.com/20180207/VNG-DCB-tn.jpg) (http://forum-images.pixinsight.com/20180207/VNG-DCB.png)
Click on the image for a full-size preview (http://forum-images.pixinsight.com/20180207/VNG-DCB.png)
Title: Re: New RAW Module - Ready for Testing
Post by: Carlos Milovic on 2018 February 07 09:45:53
Great!
Indeed, this demosaicing algorithm looks pretty good! :) I will try the new functionality.

I bought recently a Canon 80D (that is not modified, and will not be). It also has the DP sensor, and it intrigues me a lot how they are using this to micro-adjust the focus. I will try to take a deeper look into this, now that we may actually read it. :D


BTW, I'm due to defend my thesis mid-April. Right now I am finishing the writing of my 3rd paper, that will be the last chapter of my thesis. I hope that I will have more free time soon.

Title: Re: New RAW Module - Ready for Testing
Post by: llpastro on 2018 February 08 06:08:21
I also have an 80D.  Correct me if I am wrong, but I was under the impression that Dual Pixel is only used for auto focus.  Comments above indicate that the camera can take DP images as an option.  Can someone enlighten me?

Larry
Title: Re: New RAW Module - Ready for Testing
Post by: bulrichl on 2018 February 08 08:17:05
Hi Larry,

this is indeed for the 5D MarkIV and not for the 80D, but I guess it will apply nevertheless (3 pages):

http://learn.usa.canon.com/resources/articles/2016/eos-5d-markiv/eos5d-markiv-dual-pixel-raw.shtml

Bernd
Title: Re: New RAW Module - Ready for Testing
Post by: llpastro on 2018 February 08 14:53:37
Just as I thought.  The 80D only uses Dual Pixel for autofocusing, need the 5D to take DPR images.

Larry
Title: Re: New RAW Module - Ready for Testing
Post by: Carlos Milovic on 2018 February 09 10:40:53
My bad. I was sure I saw the option to save as dual pixel somewhere in the options. :(
Title: Re: New RAW Module - Ready for Testing
Post by: llpastro on 2018 February 10 06:09:41
I have searched but if you find it, post it here as I would love to use it.

Larry
Title: Re: New RAW Module - Ready for Testing
Post by: OldSkyEyes on 2018 February 10 13:41:03
It does seem slower than the previous one but it does the job.
Title: Re: New RAW Module - Ready for Testing
Post by: StefanG on 2018 February 15 13:51:37
Hi Juan,

it is great to hear that you plan to support for x-trans sensors in the debayer module. I would like to make more use of my Fuji X-T2 in astrophotography but I am struggling as sky conditions have have to deal with require proper frame calibration. I would be most happy to do some testing here!

My experiences with the new raw module are that it works nicely in demosaic mode, as did dcraw previously. However, there is still something odd when importing in pure raw mode. dcraw failed completely with all pixel values beeing zero. The new raw module imports different pixel values but the maximum values correspond to 12 bit ADC while 14bit ADC would be correct.
Am I doing something wrong here? Would a simple $T*4 solve the problem?

Best regards,

Stefan
Title: Re: New RAW Module - Ready for Testing
Post by: speach on 2018 February 15 15:32:51
I downloaded this new module yesterday. It ran with no problems, then today opened up Pixinsight and couldn't process any raw frames. Looking in format explorer there was no RAW file selection, so the programme didn't recognize the RAW frames. Where have they gone? And how do I get them back? Will I have to uninstall p/i and re-install it?

2 days later I found out what it may be, the latest win10 update was downloaded between the new RAW module working and it not working. I did a system restore this morning to before this win10 was downloaded and the RAW files came back.
Title: Re: New RAW Module - Ready for Testing
Post by: OldSkyEyes on 2018 February 24 10:26:14
After it worked for some time PI didn't want to startup anymore. (pop-up at startup telling that I need to reinstall it)
I uninstalled and installed again, not sure if it is related to this code.
Title: Re: New RAW Module - Ready for Testing
Post by: lage on 2018 February 28 05:30:08
Hi.
Does it support Pentax pixel-shift RAW files?

BR,
Lage
Title: Re: New RAW Module - Ready for Testing
Post by: acmalko on 2018 March 13 23:57:10
After it worked for some time PI didn't want to startup anymore. (pop-up at startup telling that I need to reinstall it)
I uninstalled and installed again, not sure if it is related to this code.

Hi.
Exactly the same for me. Since this update, I'm no longer able to launch Pixinsight using my project file. It always requires an msi file. Sorry.
Title: Re: New RAW Module - Ready for Testing
Post by: JayS_CT on 2018 March 14 15:28:47


That's all for now. The RAW module, along with an updated Debayer tool with X-Trans support, will be included in the next version of PixInsight, which will be released soon for all platforms, including FreeBSD ;) I really need to know how the new RAW module works in the real world, so *please* let me know anything you find out as soon as possible, be it positive or not. Enjoy!

Juan,

Maybe I'm the only one who picked up on this, but are you somewhat pre-announcing a soon to be V2 of PI or a point release?  I saw the other post about another version where you mentioned something possibly during the summer.  The reason for the question is I'm currently on the Trial license.  Reading the licensing agreement indicates that those purchasing PI within a window of time before a new version would get the new version for free, while if outside the window, it is an entire new purchase price.  I'd hate to miss the upgrade by a month or two. (and I do intend on purchasing, but now appear to have to consider timing)?  Appreciate any insight.  It is an incredible product..

Title: Re: New RAW Module - Ready for Testing
Post by: Luigi on 2018 March 14 18:37:37
I'm not Juan, but I think it's pretty much guaranteed there won't be a 'V2' anytime soon. Buy that license now :)

https://pixinsight.com/forum/index.php?topic=9786.0
Title: Re: New RAW Module - Ready for Testing
Post by: JayS_CT on 2018 March 15 08:49:09
I'm not Juan, but I think it's pretty much guaranteed there won't be a 'V2' anytime soon. Buy that license now :)

https://pixinsight.com/forum/index.php?topic=9786.0

Thanks...  I think I was taking some of the references to a new version at 2.0 when in reality it was talking to a new point release, e.g. 1.8.5 .. 

Jay S.
Title: Re: New RAW Module - Ready for Testing
Post by: msmythers on 2018 March 15 08:56:14
Jay

I've been using PI since 2012 and Ver1.6.4 I think or maybe it was 1.5 something. Anyway in that time with many, many new tools and scripts and complete rewrites of the core code there has not been an upgrade fee if that's what your concerned with.


Mike
Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 March 21 15:13:14
Wondering if the new issue with the new raw module has been resolved?  I would like to test but i dont want to corrupt my existing raw capability.
Title: Re: New RAW Module - Ready for Testing
Post by: msmythers on 2018 March 21 19:18:02
What issue are you talking about? I'm running the new module on 2 systems, one with Windows 7 and the other Windows 10 without issues that I know.



Mike
Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 March 26 12:16:32
There were a couple of users here earlier last month that had issues after installing, it possibly related to MS patches.

Wondering if the issues were resolved before i dive in.
Title: Re: New RAW Module - Ready for Testing
Post by: msmythers on 2018 March 26 12:38:22
As far as I know ??? there has been no update to the module and I can't say that I've had any problems. All my Windows computers are always up to date on all MS updates and hardware driver updates from other vendors. If you are concerned that you would not be able to recover from a software problem when installing a module that has not been officially released then my suggestion is don't install it. That is always my suggestion for anyone dealing with non-official ,beta or development software.

Quote
Availability of the RAW module

The new module is stable and has been thoroughly tested. However, since this module provides critical functionality, and given the complexity and difficulty to test it on a large amount of different raw formats, I prefer to have it extensively tested in real-world scenarios before releasing it as an official update. For this reason you have to use our development update repository to install it.
https://pixinsight.com/forum/index.php?topic=12126.0 (https://pixinsight.com/forum/index.php?topic=12126.0)



Mike
Title: Re: New RAW Module - Ready for Testing
Post by: Philippe B. on 2018 March 27 00:20:47
I have also tested this module for all my Sony A7s images taken last week in Chile
I had to integrate again my "old" bias dark and flat I still use. Maybe the decoding is not exactly the same.
But all is working fine (used drizzle bayer procedure)

Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 March 28 19:34:31
Ok. I understand. I guess worst case is I have to reinstall ripley.  The fact that you have had success with it is a relief. I often use a canon 1DX mark ii and have been having some issues with the production raw module. Mostly having to remember to set some manual settings each time before processing images with a different body.

I’m hoping the new module will at least detect the body and preset some of these automatically.  I’m sure there are other benefits to the upgrade as well.

As far as I know ??? there has been no update to the module and I can't say that I've had any problems. All my Windows computers are always up to date on all MS updates and hardware driver updates from other vendors. If you are concerned that you would not be able to recover from a software problem when installing a module that has not been officially released then my suggestion is don't install it. That is always my suggestion for anyone dealing with non-official ,beta or development software.

Quote
Availability of the RAW module

The new module is stable and has been thoroughly tested. However, since this module provides critical functionality, and given the complexity and difficulty to test it on a large amount of different raw formats, I prefer to have it extensively tested in real-world scenarios before releasing it as an official update. For this reason you have to use our development update repository to install it.
https://pixinsight.com/forum/index.php?topic=12126.0 (https://pixinsight.com/forum/index.php?topic=12126.0)



Mike
Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 April 01 21:17:49
Just a follow up that the raw update went smoothly on win10 and was able to read my 1dx mark ii files as well as after the march MS patch rollup was installed.

Now to figure out where i want all of these new settings to be.  :)
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 05 02:26:15
Hi There. Two days ago I installed a Trial version of PI. I use a Fuji X-T2. In format I hit the "Pure Raw" option but when I open a file is black. I guess it has to do with the X-trans sensor of the camera. I have read here that it was not possible to open these .RAF files in PI but now there is a new RAW module that allows it. Does anybody know which should be the settings to open these files, if possible?
Thanks

Mendi
Title: Re: New RAW Module - Ready for Testing
Post by: Juan Conejero on 2018 April 05 12:19:54
Hi Mendi,

With the new RAW module you can open Fuji X-T2 raw files without problems. However, unfortunately the current Debayer module cannot interpolate X-Trans frames.

The good news now: I have a working Demosaic module, which will replace the current Debayer module in the next version of PixInsight. Demosaic implements the same X-Trans interpolation algorithm as libraw does, but in a fully parallelized version. This means that X-Trans frames can now be used just as Bayer frames throughout the whole preprocessing chain, from ImageCalibration to DrizzleIntegration. If you and other X-Trans users are interested, I can make a prerelease of the new Demosaic module, along with a modified version of BatchPreprocessing to support X-Trans demosaicing, through the update-devel repository.
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 05 22:45:32
Hi Juan

Glad to read about this news although I am not very sure of what you are saying. As I wrote I tried to open the RAF files (using pure raw setting) but I only got back is a black image. Which settings do I have to use to open these raw images?

About interpolation you mean stacking them? It would be great to have the opportunity to check that update. As I am very new in PI program and web, where could I find the update-level depository?

Thanks for your time

Mendi
Title: Re: New RAW Module - Ready for Testing
Post by: bulrichl on 2018 April 06 10:14:34
Hi Mendi,

Quote
About interpolation you mean stacking them?

No. Juan wrote it in his post:

Quote
With the new RAW module you can open Fuji X-T2 raw files without problems. However, unfortunately the current Debayer module cannot interpolate X-Trans frames.

'Interpolation' (= demosaicing) is the conversion of a Color Filter Array (CFA) image to a RGB image.

Presupposed that you did install the new Raw module according to the instructions given at the top of this thread and selected 'Pure raw' as option in Format Explorer/RAW, you should be able to open Fuji X-T2 raw files. When opening a raw image, a monochrome CFA image is displayed on the screen. It is normal that this image is very dark. Execute PROCESS/IntensityTransformations/ScreenTransferFunction (STF) and let it perform an Auto Stretch (radioactive symbol) to the image. This will display a stretched version of the monochrome CFA image.

Thus with the new RAW module you can open and display the monochrome CFA image, but the current Debayer module cannot convert it to a RGB image. Therefore, at present, you cannot align and integrate images that are taken in the Fuji X-T2 raw format.

However, Juan offered to make a prerelease (before the next version of PI is released, in which all this will be working) of the new Demosaic module, along with a modified version of BatchPreprocessing to support X-Trans demosaicing, if there is interest.

Bernd
Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 April 07 08:36:47
Mendi, I suspect since you are in trial mode that you are a new user and still learning the complex interface. While I’m sure Juan can make it available for testing, you may not have sufficient time to evaluate it before your trial expires.

Are you able to evaluate the product using a different method?  It’s not ideal but if you can convert to jpg in camera you can at least test some of the processing modules before your trial ends.

Pixinsight has a very steep learning curve and takes a very long time to master. Please start out with basic processing first and watch some of the fantastic videos on YouTube which show step by step how to perform the more advanced steps. 

In the mean time Juan appears committed to get the x-trans conversion working and he usually never misses a promise.

Enjoy.
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 07 11:16:17
Hi Bernd,

Thanks so much for your answer.

Presupposed that you did install the new Raw module according to the instructions given at the top of this thread and selected 'Pure raw' as option in Format Explorer/RAW, you should be able to open Fuji X-T2 raw files.

I have installed the new RAW module (not done before) but now I miss the "format explorer" tab that was in the left side and can not find the way to choose "Pure Raw" anymore. During the installation there were some fail messages (attached an image) that maybe are the cause of this situation.

Thanks for your help

Mendi
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 07 11:22:14
Hi Eastwind

Thanks for your help. I am still struggling trying to open the raw files... As you say I am learning with some great videos in the Internet but I got stuck in this very beginning part withe the RAWs in my fuji.

I will definitely try to convert in JPG but the problem now is that I have "lost" the Format Explorer Tab and can not choose the kind of format image...

I guess I am already very new in PI and as you say the learning curve is steep.

Regards

Mendi
Title: Re: New RAW Module - Ready for Testing
Post by: pfile on 2018 April 07 11:30:19
the module did not load due to some kind of segmentation fault (illegal memory access.) i wonder if you are the first OSX user to try to load this module (i do not use a DSLR anymore, so i have not tried.)

juan would have to comment about this.

rob
Title: Re: New RAW Module - Ready for Testing
Post by: bulrichl on 2018 April 07 14:56:18
Hi Mendi,

I am sorry that the new RAW module did not install correctly. On my computer (Windows 7 Professional) it installed and worked flawlessly (tested with Canon CR2 raw files).

Bernd
Title: Re: New RAW Module - Ready for Testing
Post by: EastWind on 2018 April 09 15:29:53
Sorry you are having trouble. No telling what is causing the issue. Just to be safe, try uninstalling Pixinsight completely. Restart your system. Then reinstall using all defaults. Then after reinstalling the trial license if needed, reinstall the new raw module as before.

This should also resolve the lost tab issue but I’m sure there is a way to restore that. But considering the other issue, retrying the install would seem like a good plan B.

Hi Eastwind

Thanks for your help. I am still struggling trying to open the raw files... As you say I am learning with some great videos in the Internet but I got stuck in this very beginning part withe the RAWs in my fuji.

I will definitely try to convert in JPG but the problem now is that I have "lost" the Format Explorer Tab and can not choose the kind of format image...

I guess I am already very new in PI and as you say the learning curve is steep.

Regards

Mendi

Title: Re: New RAW Module - Ready for Testing
Post by: msmythers on 2018 April 09 16:57:46
Mendi

If you haven't been able to get the new module working and would still like to know if you raw files will work with it I would suggest uploading one or two raw images to a file hosting site like Dropbox or Google Drive and post a link here. Some of us who have the module I'm sure would be more then happy to see if it works. I would suggest one of the images if possible be a daytime image maybe with some grass in the shot. The grass will make it very apparent if the right settings are chosen for color.



Mike
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 10 07:20:59
Hi Eastwind

Try what you said

When installing the missing Tab was still missed and when installing actualization they gave no error but can not find th "format Explorer" tab... so the Raw files are still black

Sorry you are having trouble. No telling what is causing the issue. Just to be safe, try uninstalling Pixinsight completely. Restart your system. Then reinstall using all defaults. Then after reinstalling the trial license if needed, reinstall the new raw module as before.

This should also resolve the lost tab issue but I’m sure there is a way to restore that. But considering the other issue, retrying the install would seem like a good plan B.
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 10 08:00:05
Here you have Mike

https://www.dropbox.com/s/ijm4v2s7izv0m8f/DSCF4697.RAF?dl=0

Thanks for the help

Mendi

Mendi

If you haven't been able to get the new module working and would still like to know if you raw files will work with it I would suggest uploading one or two raw images to a file hosting site like Dropbox or Google Drive and post a link here. Some of us who have the module I'm sure would be more then happy to see if it works. I would suggest one of the images if possible be a daytime image maybe with some grass in the shot. The grass will make it very apparent if the right settings are chosen for color.



Mike
Title: Re: New RAW Module - Ready for Testing
Post by: bulrichl on 2018 April 10 09:34:29
Hi Mendi,

I use a computer with Windows 7 Professional, Pixinsight v 01.08.05.1353 Ripley (x64) with the new RAW module. In FormatExplorer/RAW I have set 'Pure raw'

When your file "DSCF4697.RAF" is opened, a monochrome image is diplayed. This is the output to Process console:
---------------------------------------
Reading 1 image(s):
C:/Users/BL/Downloads/DSCF4697.RAF
Reading metadata: done

Camera ........... Fujifilm X-T2
Timestamp ........ 2018-03-23T19:15:38Z
Exposure ......... 1/250s
ISO speed ........ 6400
Focal length ..... 19 mm
Aperture ......... f/3.2 = 5.9 mm
Author ...........
CFA pattern ...... RBGBRGGGRGGBGGBGGRBRGRBGGGBGGRGGRGGB
Raw dimensions ... w=6160 h=4032
Image geometry ... x=0 y=0 w=6032 h=4032

Raw decoding parameters:
Output mode ............... cfa
Auto crop ................. enabled
White balancing ........... disabled
Auto rotate ............... disabled
Output image .............. w=6032 h=4032 n=1 Gray

Reading RAW data: done
Scaling colors: done
Pre-interpolating: done
Converting to RGB: done
---------------------------------------

(I deleted the Author from the output.)

When I tried to debayer the image (Bayer/mosaic pattern: Auto), the error
"Unsupported or invalid CFA pattern 'RBGBRGGGRGGBGGBGGRBRGRBGGGBGGRGGRGGB'"
is displayed. So the module is working correctly here and the failing of the debayer process is expected behavior according to what Juan wrote in his post.

Your problem is indeed that the new RAW module didn't install correctly. Rob seemed to suppose that you are using OSX and that the problem of the failing module installation is connected with the operating system.


Bernd
Title: Re: New RAW Module - Ready for Testing
Post by: mendimundi on 2018 April 10 09:53:12
Thanks, Bernd

That's correct. My computer is an iMac and I guess there is some kind of problem there but I can not fix it. I hope that Juan could give me an idea for the correct installation.

Regards

Mendi
Title: Re: New RAW Module - Ready for Testing
Post by: StefanG on 2018 April 15 05:02:10
Dear Juan and other X-Trans users,

I would just like to wave hands and say that I am definitely interested in having a go with the new demosaic module should it be made available via update-devel channel. As I mentioned earlier, I would like to make more use of my Fuji X-T2 in astrophotography.

Regards,

Stefan
Title: Re: New RAW Module - Ready for Testing
Post by: RoMe on 2018 May 03 04:19:59
Another waving hand here.  :)

Apart from the missing demosaicing the import seems to work fine with my Fuji pictures.

However, I've noticed that this module eats RAM like crazy. When I integrate 50 frames from compressed RAW files (Fuji X-T2) then it fills up my 16GB RAM and my 16GB swap partition after about 40 pictures. Of course PI crashes at this point and the system gets unstable and needs a restart.

When I convert the files to 32 bit float xisf first then PI only uses between 3GB and 5GB of RAM with the same settings (that sounds reasonable).

Title: Re: New RAW Module - Ready for Testing
Post by: bulrichl on 2018 May 04 01:52:42
Quote
...
When I convert the files to 32 bit float xisf first then PI only uses between 3GB and 5GB of RAM with the same settings (that sounds reasonable).

Yes, that's public. The reason is: DSLR raw files cannot be integrated with incremental disk file read operations.

Bernd
Title: Re: New RAW Module - Ready for Testing
Post by: GuillaumeKIO on 2018 May 12 09:01:26
Hello  :),
I am very satisfied to see your interest in fuji RAF.
I arrive well has to open them, thank you very much !

The good news now: I have a working Demosaic module, which will replace the current Debayer module in the next version of PixInsight. Demosaic implements the same X-Trans interpolation algorithm as libraw does, but in a fully parallelized version. This means that X-Trans frames can now be used just as Bayer frames throughout the whole preprocessing chain, from ImageCalibration to DrizzleIntegration. If you and other X-Trans users are interested, I can make a prerelease of the new Demosaic module, along with a modified version of BatchPreprocessing to support X-Trans demosaicing, through the update-devel repository.

I am interested for test the new module  ;)
Title: Re: New RAW Module - Ready for Testing
Post by: blueridgeDSIA on 2018 May 23 10:58:31
how can you fine tune fbdd?
Title: Re: New RAW Module - Ready for Testing
Post by: blueridgeDSIA on 2018 July 17 08:59:54
Im not sure what is going on with Pixinsight lately, it seems like there have not been any updates in months.

I have a serious issue with this module.  First of all, I have no way to uninstall it from Pi, I have uninstalled the program itself but upon re installation the new raw module is still loaded in the program.  The raw module works fine for a period of time and then mysteriously disappears from the format explore window.  Once this happens I can no longer open ANY dslr files. It completely brickwalls my workflow and I am no longer able to use Pixinisght to open and process my astrophotography.


Title: Re: New RAW Module - Ready for Testing
Post by: RoMe on 2018 July 17 09:42:15
Im not sure what is going on with Pixinsight lately, it seems like there have not been any updates in months.

I was hoping for a slightly faster progress as well. I started my trial period when I saw this announcement because now I could use my Fuji files directly with PI. I couldn't get past the debayer step in my tests but I really liked what I saw up until this point. Sadly my trial license has expired many weeks ago.

I had to put PI back on the shelf again. I've stopped monitoring the official channels so please also post in this thread if there is an update. Maybe I'll have a look at PI again when the nights are getting longer. :-\
Title: Re: New RAW Module - Ready for Testing
Post by: hy on 2018 October 27 16:22:33
I'm checking in about the availability of Debayering Fuji (X-T2). You mentioned that if there was sufficient interest, you might release a beta version or beta tool for testing. Please let me know if that is a possibility.

I have been trying to export the calibrated files, and run through libraw to debayer on my own (and the re-import to PI), but so far no luck.

Thanks,
Hy
Title: Re: New RAW Module - Ready for Testing
Post by: RoMe on 2018 October 27 17:24:10
That would be great. I'm currently exporting my calibrated files as FITS, do my own demosaicing and then import the resulting RGB FITS files back into PI.

It works but it is very cumbersome and my home brew demosaicing algorithm is not the best. An integrated solution would be a huge step forward for my workflow.
Title: Re: New RAW Module - Ready for Testing
Post by: slewmaster on 2018 November 26 03:56:49
I also experienced the new RAW module disappearing after using it. Unfortuantely, I did not create a windows restore point before updating PI. The fix is to search windows registry of PixInsight entries and delete them. Next, search windows hard drive for PixInsight entries and deleted them too. Finally re-installed PI and DSLR-RAW module appeared. Next time, I'll create a windows restore point before updating PI.
Title: Re: New RAW Module - Ready for Testing
Post by: arthuss on 2018 December 15 03:00:38
Sorry for may bad English

Is Raw nikon Z6 supported ?

My image is bad

Thanks

Jm
Title: Re: New RAW Module - Ready for Testing
Post by: sharkmelley on 2018 December 15 09:28:15
Nikon Z6? Very unlikely.

The author of LibRaw added Z6 support only a few days ago: https://www.dpreview.com/forums/post/62005403

Mark
Title: Re: New RAW Module - Ready for Testing
Post by: arthuss on 2018 December 23 08:24:20
not supported in pixinsight
   :-\ :-\
Jm
Title: Re: New RAW Module - Ready for Testing
Post by: msmythers on 2018 December 23 08:54:59
Here is a response from Juan last week.

Quote
The authors of LibRAW have released a new snapshot a few days ago that supports the Nikon Z6. This snapshot was available a couple of days before the PixInsight 1.8.6 release, and I was considering the possibility to include it. Since the RAW module is a critical component, I decided to keep the latest 0.19.1 stable release of LibRAW to stay on the safe side. I can release a development version of the RAW module built with the snapshot through the update-devel repository; I'll keep you informed about this.
 
Juan Conejero
PixInsight Development Team
http://pixinsight.com/


https://pixinsight.com/forum/index.php?topic=12986.msg79168#msg79168 (https://pixinsight.com/forum/index.php?topic=12986.msg79168#msg79168)


Mike
Title: Re: New RAW Module - Ready for Testing
Post by: arthuss on 2018 December 23 10:00:24
I saw. I also asked there

Jm