Author Topic: BPP failures due to multiple monitors?  (Read 2194 times)

Offline ysaito

  • Newcomer
  • Posts: 8
BPP failures due to multiple monitors?
« on: 2020 February 03 01:59:43 »
I am a newbie user of PixInsight. I have processed fits files taken by a ZWO ASI OSC camera without any problems for about 6 months, integrating by BPP or manually. I tested my Canon APS-C DSLR the other day and had two problems with BPP.

PixInsight, Version 1.8.8-4 Ripley (x64), was reinstalled with some patches but the same results, on my desktop computer with triple monitors, Windows 10 Pro, Core i5 8400 and 32GB memory including variable 20GB ImDisk ram disk drive.

The first problem is that flats overcorrect lights without darks. There is a warning pointing out the lack of darks starting BPP and ends with an overcorrected master light (see picture 01).

I can get a decent master light integrating manually with PixInsight or using DeepSkyStacker though, even if there are no darks. The master light shown on an attached picture may not be decent but please note that 300sx17 lights were taken in a city of Bortle class 8-9 in Japan, with a 71mm refractor (see picture 02).

The second one might be more serious. I then took dark files and tried again including them. There are no warnings on settings starting BPP but terminates with an error dialog box that says, "ERROR integrating light flames" and the error message below.
   ***Error: R:/PixInsightWorkPlace/registered/
   L_0014_300s_ISO200__21C_c_cc_d_r.xsif: Zero or insignificant signal
   detedted (empty image?)

A WBPP ends with the same result. Besides a WBPP process cannot be executed with an error message below regardless processing tiffs or raw files, which forces me to give up using it.
   ***Error [228]: C:/Programs Files/PixInsight/src/scripts/
   weightedBatchPreprocessing/WeightedBatchPreprocessing-engine.js, line
   2721: SyntaxError; JSON.parse: bad control character in string literal

I searched about those problems on the Internet and found that they were solved in the newer version of PixInsight. I also found that multiple monitors may cause some glitches. Then I tested PixInsight running on the main display instead of subs, having the same results. I would appreciate it if I could be given any hints solving or avoiding the problems above.
« Last Edit: 2020 February 03 16:41:46 by ysaito »

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: BPP failures due to multiple monitors?
« Reply #1 on: 2020 February 03 09:24:54 »
well the overcorrection thing is almost always due to improper calibration, including no calibration, of flats. so i don't know how any other package can produce proper results unless it somehow calibrated the flats. if there were bias frames present then those other programs may have used the bias frames to calibrate the flats, which usually works fine because the flats are generally of short duration.

the "insignificant signal" problem is almost always caused by some issue with the darks - either the darks don't match the temperature or gain/ISO of the lights, or perhaps the darks were captured with some program that saved them as FITS but you are using CR2 files for lights. same goes for bias frames that don't match the gain/temperature of the lights.

lastly if you happen to be using master frames created with older versions of PI, the old frames are probably incompatible with how newer versions of PI handle DSLR RAW frames. also, if you are using masters created in another program with BPP this can cause problems as well.

rob

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #2 on: 2020 February 03 19:21:49 »
ron, thank you for your reply.

I'm glad to know these problems have nothing to do with multiple monitors.

As for overcorrection without darks, please let me know further. As said before, I integrated flats and lights manually and was given a proper stacked light. What is the difference between BPP and manual integration.

First I calibrated lights with Master Bias, ticking off Calibrate, without Master Dark(see picture 01), ticking on Calibrate gives any difference though, then integrated calibrated flats, which created the proper Master Flat (see picture 02), which seems proper to me.

Calibration of lights with the Master Bias and the Master Flat without Master Dark brought forth proper calibrated lights, one of which is shown in picture 03.

Am I right to understand that flats are calibrated by Master Bias in manual integration but not calibrated by any in BPP? Are dark files  musts in PixInsight?

Thank you again.
ysaito

Offline pfile

  • PTeam Member
  • PixInsight Jedi Grand Master
  • ********
  • Posts: 4729
Re: BPP failures due to multiple monitors?
« Reply #3 on: 2020 February 03 20:41:17 »
it is generally thought to be OK to calibrate flat subs with only bias frames as long as the flats are of short duration. the less dark current your camera's sensor has, the better in this regard. having said this, with an uncooled camera the dark current is going to be higher than with cooled sensors. still i see that your flats are 1.6-2s long and i'd imagine that calibrating the flats with only a bias frame should be OK.

calibration of bias frames is generally not necessary at all. i'd say this is a very specialized technique that only a few types of sensors would require. i think you might need to configure the overscan settings above the bias filename in order for bias calibration to even do anything. so i'm not surprised that nothing changed when you asked to calibrate the bias.

BPP's integration is considered to only be a preview because all of the pixel rejection stuff should be tuned to get a proper master light frame. for instance in theory you should integrate your master light with no pixel rejection and check the SNR, then set up a rejection method and adjust the rejection sliders before running again. you should then look at the rejection maps to make sure you don't see the structure of your object in the rejection maps, and you should test the new master light to make sure the SNR is still similar to the one made with no rejection.

i think BPP has always wanted dark frames and tries to scale them for calibration of the flats. there is a new checkbox in the "flats" section of BPP that tells BPP to look for shorter dark frames that match the flat duration and use those darks to calibrate the flats, instead of trying to scale (optimize) the darks or master dark that corresponds with the light duration. for you i guess you'd have to make darks of 1.6s and 2s and any other duration that your flats happen to be for this checkbox to work right.

the BPP script always creates calibrated master flats, so if you made a calibrated master flat by hand, you can just load that master flat and tick the "use master flat" checkbox. then load your lights and the darks that match the lights and run, and BPP should calibrate the lights using that master flat. this is a good way to do it with a DSLR for a couple of reasons - one is that sometimes people will use a different ISO for the flat and so you'd need a different ISO dark or bias to calibrate the flat and BPP doesn't really support different ISOs, so you need to present it with an already calibrated master flat. another reason is that if you do it this way you don't have to worry about making special darks for the flats nor do you have to worry about BPP matching them all up correctly. the only darks and bias it will care about when using a  master flat are the ones that match the lights, so there is no confusion.

you can also do a "dummy" run of BPP with throwaway lights where the purpose is just to get BPP to build a properly calibrated master flat. if you wanted to calibrate the flat subs with bias only, you would do that by loading your bias frames as darks, load a few lights (maybe even just one would work, not sure) and then just run and save the master flat which is produced. then you reset BPP and load that master flat plus all your lights and darks and bias and run again, this time for real. and after that step you should then have a master dark and a master bias that you can load the next time around instead of the bias and dark subs.

rob


rob

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #4 on: 2020 February 03 23:26:09 »
rob, thank you again for the detailed explanation, which helps me understand PI better.

I took 20 flat darks with 1.6s duration, and put them in the Darks tab, ticking on "Calibrate with flat darks only" in the Flats tab (see pic 01).

No overcorrection, thanks. But the color of the nebula seems weird or bluish (see pic 02), which remains the same after applying DBE. I hope it will  improve after applying PCC, which failed maybe due to a lot of noises.

I will make a rule to take flat darks as darks when using BPP and DSLR.

Thanks a lot!
ysaito

 

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: BPP failures due to multiple monitors?
« Reply #5 on: 2020 February 04 02:35:26 »
The weird color of IC 2177 might be a result of a wrong bayer/mosaic pattern. Note that the new raw decoding software LibRaw that is used by the PixInsight's RAW module since core version 1.8.8 works differently than previous versions when cropping the optically black region. Before this change in LibRaw, as a rule ALL Canon cameras had RGGB. Now there are e.g. some Canon cameras, whose bayer/mosaic pattern is thereby changed from RGGB to GBRG, see https://pixinsight.com/forum/index.php?topic=14206 and https://pixinsight.com/forum/index.php?topic=14219 .

The easiest way to find out whether your camera is affected is to open an image in raw (CR2) format and take a look at the Process Console output: view the value of 'CFA pattern'.

For regular digital cameras it is not necessary to specify the bayer/mosaic pattern explicitely. So you can use the 'Auto' setting for the bayer/mosaic pattern when processing images of the Canon camera. (This does not necessarily apply to dedicated astro cameras. In this case it depends whether the acquisition software uses the FITS keyword 'BAYERPAT' or not. If this keyword is not used, the bayer/mosaic pattern has to be specified explicitly.)

Bernd

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #6 on: 2020 February 04 05:25:31 »
Hello, Bernd

Thank you for your suggestion. You are right. My camera is EOS 600D and the Bayer pattern is thought to be GBRG (see pic 01).

I then installed the RAW-compat module as you suggested. There, Process Console shows RGGB now (see pic 02).

But when I ran BPP, a calibration error occurred, which is shown in pic 03, saying "Error: Incompatible image geometry".

I assumed that the master bias and flat are incompatible with light frames, because master frames were integrated in the older (or newest) version. So I integrated bias frames, which went smoothly, then tried to calibrate flat frames with this master bias, without master dark, which bumped into the same problem as the lights of "Incompatible image geometry" (see pic 04).

I am sorry to bother you but would you please let me know how to get off this hook?

Thank you.

ysaito
 

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: BPP failures due to multiple monitors?
« Reply #7 on: 2020 February 04 08:07:30 »
I am sorry this is a misunderstanding. I did not want you to install the Raw-compat module. When you use master calibration files prepared with the current standard RAW module and calibrate the CR2 light frames, all is appropriate.

The compat-module is only for the case when you want to continue using old master calibration files that were prepared with the old RAW module.

So please follow Juan's advice in https://pixinsight.com/forum/index.php?topic=14219 for installing the standard RAW module again (i.e., you reverse the install of the compat-RAW module):

----------
"To return to the standard RAW module, remove the update-compat repository enabled with the above procedure. Then add the following repository URL:
https://pixinsight.com/update-opt-in/1.8.8-4/
and repeat the check for updates process, etc. This will reinstall the standard RAW module using the latest LibRaw version available."
----------

With this standard RAW module installed, the bayer/mosaic pattern for your camera (Canon EOS 600D) is now GBRG and no longer RGGB. So for debayering, the debayer/mosaic pattern must be set to GBRG in future. However, as I described in the post above, for regular digital cameras you need not specify the bayer/mosaic pattern explicitely, the 'Auto' setting will work as well.

Bernd

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #8 on: 2020 February 04 17:14:40 »
Bernd, thank you again.

I misread your explanation, but now I think I grabbed the situation. Delete the RAW-compt module, reinstall the newer one, and set the Bayer pattern to GBRG explicitly (see pic 01).

Then I got the stacked light frame with natural color (see pic 02), which can be applied by PCC, though it cannot work on the weird colored one with various parameters.

I can use this Canon DSLR for astrophotography as well as ZWO ASI OSC camera. I would like to give many thanks to you.

ysaito

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: BPP failures due to multiple monitors?
« Reply #9 on: 2020 February 05 04:56:52 »
Hm, the integration (image 02) still looks weird. I would do the following, starting from scratch (and at first WITHOUT BPP):

1. Integrate the dark frames to the MasterDark,
2. Integrate the flat-darks to the MasterFlat-Dark,
3. calibrate the individual flat frames with the MasterFlat-Dark,
4. integrate the calibrated flat frames to the MasterFlat,
5. calibrate the light frames with MasterDark and MasterFlat only:
   - uncheck the section 'MasterBias'
   - check the section 'MasterDark', select the MasterDark, disable both options 'Calibrate' and 'Optimize'
   - check the section 'MasterFlat', select the MasterFlat, disable the option 'Calibrate'
6. debayer the calibrated light frames,
7. register the calibrated and debayered light frames,
8. integrate the calibrated, debayered and registered light frames.

This approach does not use bias frames or a MasterBias at all.

Please then show the resultant integration.

Bernd

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #10 on: 2020 February 05 05:31:55 »
Now I'm just taking the Seagull Nebula with the same perspective as the one above for testing, as it is about 10:30 p.m. here in Japan. I am interested in the process of editing the images you just presented.

I am going to take darks, which was not planned at the outset though, as well as flats and flat-darks after taking the same number of lights as the previous ones, i.e. 17, then process them as you suggested tomorrow and post the results here.

Talk to you then. Thanks.

ysaito

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #11 on: 2020 February 06 00:12:25 »
I applied 15 darks taken yesterday to the lights, which had been integrated by BPP and had already been posted. I processed them as Bernd suggested, plus cosmetic correction, and got the result shown in pic 01.

I thought at first that it is just the same as the one without darks and by using BPP. Closely viewed, enlarged to scale 2:1, though, they are distinctly different. On the right of the pic 02 is the one using darks instead of biases, which is much better than the one on the left using biases, having less noise.

I am wondering if it boils down that you had better take darks rather than biases or turn to manual integration rather than BPP. I first thought darks are musts for ZWO ASI cameras due to their amp glow, but not for DSLR cameras. Does BPP always create poorer results than the manual one? Are these just layman's ideas?

ysaito
« Last Edit: 2020 February 06 00:23:25 by ysaito »

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: BPP failures due to multiple monitors?
« Reply #12 on: 2020 February 06 01:47:58 »
When comparing the two image sections (image "02 Comparison.jpg"), the left one shows "walking noise" caused by warm/cold pixels and by drift during the entire capturing session. The difference actually is even larger than the shown images might suggest: you must consider that in the right image STF applied a stronger stretch than in the left image. Actually for the right image a STF using the same Midtones parameter as for the left image should have been applied.

When comparing image "02 BPP result.jpg" of reply #8 and "01 WoBias_0201_Manually.jpg" of reply #11 there is another important difference:
In "02 BPP result.jpg" of reply #8 the flat field correction seems to have overcorrected the vignetting. It will be hard to correct the gradients by DBE or ABE.
In "01 WoBias_0201_Manually.jpg" of reply #11 the flat field correction has done a better job, leaving on the whole an almost linear gradient which should be easier to correct with DBE or ABE.
The mistake in the BPP settings for "02 BPP result.jpg" of reply #8 was to use BOTH MasterDark and MaterBias without applying dark frame optimization.

Regarding BPP: if you set the right parameters in BPP you will get the identical result as with manual preprocessing (including the steps image calibration, debayering and registering) because BPP relies on the the same processes of PixInsight: ImageIntegration, Debayer and StarAlignment. However, for ImageIntegration, the recommendation is to consider BPP's result only as a starting point. You always should optimize the ImageIntegration process manually to have full control over pixel rejection. This will enable you to exclude artifacts like streaks caused by satellites, airplanes, cosmic ray artifacts, etc. completely, and at the same time to get the best SNR in the integration result.

Bernd

Offline ysaito

  • Newcomer
  • Posts: 8
Re: BPP failures due to multiple monitors?
« Reply #13 on: 2020 February 06 06:18:27 »
Bernd,

Thank you for your comments, including incompatible STF settings, dark frame optimization and the relationship between BPP and manual integration process, which are highly informative for me.

I think I now understand what you were going to point out and why I have been sometimes bothered by the incomplete flat corrections, which cannot been flattened even if I try to apply DBE twice or three times. "Optimize dark frames" has been always ticked off so as to erase amp glow completely, while processing fits files produced by an ASI camera.

Now I am struggling with the overcorrection problem using BPP again, which once was solved with putting flat darks in the darks tab but is not replicated at all.

I have to get more comfortable with manual preprocessing, which seems to give satisfying results for the meantime, and then return to BPP.

Anyway thanks again.

ysaito

Offline bulrichl

  • PixInsight Guru
  • ****
  • Posts: 524
Re: BPP failures due to multiple monitors?
« Reply #14 on: 2020 February 06 07:23:59 »
I have to get more comfortable with manual preprocessing, which seems to give satisfying results for the meantime, and then return to BPP.
In my opinion, this is exactly the way to go. As I recently read: "Never automate until you can do it all manually first.". There is much experience in this dictum.

Bernd