Easy way to drop these reals?

[wkitlasten]

Hi, I am getting a few realizations that give exceptionally low values but the obs is not tagged as prior data conflict since technically the obs dist fall within the bounds of the prior. I have ies_bad_phi_sigma=3, but phi for the entire realization does not meet the requirement to drop out. Parameter upgrades seem to just be moving those far away realizations closer, for an iteration or two and then not much actions. I assume these are not ideal for the empirical jco :).

I am considering dropping any realizations where outliers like this occur for any obs. Any concerns or thoughts? Any clever approaches others have used?

[jtwhite79]

Hey Wes - I was thinking maybe you could add those same flow obs to the ctrl file twice and the second set of them could be inequality obs with huge weight so that when the condition is violated, it would give you a huge phi, so that they would be dropped as bad. but if the condition is satisfied (like flow > -2 in your plot), then these obs wouldnt have any influence. Does that sound reasonable?

[cnicol-gwlogic]

One thing I've done with really low SFR flow rates is set a minimum value that i treat as basically zero (in my flow obs processing workflow). That might still artificially affect things, not sure, but I've had decent success with flow matching using this approach.

[wkitlasten]

Makes sense, but I think I am failing at the implementation for some reason. In my control file I have:

ies_bad_phi 1e+30

And I am getting:

  ---  pre-drop initial phi summary  ---
       phi type           mean            std            min            max
       measured    2.62271e+74    2.51207e+75        11.1603    3.51509e+76
         actual    2.32244e+75     2.5042e+76          1e+30    3.90354e+77
     note: 'measured' phi reported above includes
           realizations of measurement noise,
           'actual' phi does not.

  ---  observation group phi summary ---
       (computed using 'actual' phi)
           (sorted by mean phi)
group                                   mean            std            min            max        percent            std            min            max
less_flow                        **2.32244e+75     2.5042e+76    1.63448e+68**    3.90354e+77            100    1.11947e-14            100            100
flow                                 7056.54        73317.8       0.218202    1.12752e+06    1.58398e-69    9.63157e-69     3.7087e-73    1.64644e-67
tritium_measured_112_28              15.1189        70.5093     0.00124885        931.839    1.16841e-68    1.22281e-67    9.68084e-76    1.96581e-66
tritium_measured_32_29               11.2122        43.4684      0.0018284        756.491    3.54206e-68    4.78784e-67    8.13368e-76    8.23266e-66
tritium_measured_105_46              8.68849        36.9992    3.12595e-08        502.673    1.69155e-69    1.04968e-68    6.01733e-80    1.20939e-67
tritium_measured_10_60               8.60003        52.6968    1.58108e-05        858.918    1.08997e-68    1.40093e-67    5.31212e-78    2.41201e-66
tritium_measured_128_47              7.75544        56.9455     8.7188e-05        952.565    2.04696e-69    2.11702e-68    1.50324e-77    3.62478e-67
tritium_measured_60_0                7.31389        19.1306    3.68077e-07        200.427    9.56312e-69    1.36593e-67    8.82841e-80    2.36102e-66
tritium_measured_130_36              7.27636        49.6586    4.46808e-05        795.054    2.10746e-69    2.12575e-68    1.51298e-76     3.6199e-67
tritium_measured_127_47              7.20132        57.1272    4.35301e-05        952.738    8.12052e-70    5.17936e-69    3.38948e-77    7.17046e-68
tritium_measured_45_31               7.03243        17.4748    0.000140871        211.184     1.9741e-68     2.7018e-67    5.62831e-76    4.63662e-66
    Note: 'percent' is the percentage of the actual phi for each realization.
    Note: only the first 10 highest mean phi groups are listed. See rec file for full listing

...checking for prior-data conflict...

  ---  WARNING: 1 non-zero weighted observations are in conflict with the prior simulated ensemble.
  ---
...see rec file or hfx_wairau_14_s1_500.pdc.csv for listing of conflicted observations

...dropping conflicted observations
...number of non-zero weighted observations reduced from 2005 to 2004

...updating localizer
reading 36006502 elements, 26900 rows, 28257 columns
dropped 24912 from localizer rows because forgive_missing is true
...updated observation data information written to file  hfx_wairau_14_s1_500.adjusted.obs_data.csv
...not dropping 'base' real even though phi is 'bad'

  ---  **dropping realizations as bad:  17**  ---
...dropping par:obs realizations: 1 : 1 , 12 : 12 , 19 : 19 , 66 : 66 , 78 : 78 , 95 : 95 , 117 : 117 , 118 : 118 , 206 : 206 , 221 : 221 , 229 : 229 , 230 : 230 , 253 : 253 , 259 : 259 , 260 : 260 , 268 : 268 , 285 : 285 ,

  ---  initial phi summary  ---
       phi type           mean            std            min            max
       measured    1.88814e+69    3.17635e+70        11.0832    **5.34345e+71**
         actual    8.69805e+74    1.12467e+76          1e+30    1.78821e+77
     note: 'measured' phi reported above includes
           realizations of measurement noise,
           'actual' phi does not.

  ---  observation group phi summary ---
       (computed using 'actual' phi)
           (sorted by mean phi)
group                                   mean            std            min            max        percent            std            min            max
less_flow                        **8.69805e+74    1.12467e+76    1.63448e+68**    1.78821e+77            100    1.12197e-14            100            100
flow                                 28.5637        85.1318       0.218202        663.193    1.55111e-69    9.90897e-69     3.7087e-73    1.64644e-67
...

Why would my max phi still be 5e+71 if I should have dropped all reals with phi >1e+30?

version: 5.1.24
binary compiled on Oct 7 2022 at 10:46:45

[jtwhite79]

What version are you using? That was a fix in there recently for phi values greater than 1e30...

[wkitlasten]

I updated my pestpp version, but I am still not making sense of it. Using default for ies_bad_phi_sigma. It looks like my "flow" tag is picking up less_flow and flow obsgp. But I though all reals with any contribution from "less_flow" would have been thrown out? Some pertinent bits.

version: 5.2.0
binary compiled on Feb 5 2023 at 17:36:20

ies_bad_phi: 1e+30
ies_bad_phi_sigma: 1e+300

--- pre-drop initial phi summary ---
phi type mean std min max
measured 1.9465e+75 2.16139e+76 48837.3 3.41679e+77
actual 2.39975e+75 2.54353e+76 1.36725e+71 3.95908e+77
note: 'measured' phi reported above includes
realizations of measurement noise,
'actual' phi does not.
--- observation group phi summary ---
(computed using 'actual' phi)
(sorted by mean phi)

...bad_phi_sigma 1e+300 and mean 1.9465e+75 yields bad phi threshold of inf

--- dropping realizations as bad: 133 ---
...dropping par:obs realizations: 1 : 1 , 2 : 2 , 5 : 5 , 7 : 7 , 8 : 8 , 11 : 11 , 12 : 12 , 15 : 15 , 16 : 16 , 18 : 18 , 19 : 19 , 23 : 23 , 26 : 26 , 27 : 27 , 29 : 29 , 31 : 31 , 34 : 34 , 36 : 36 , 40 : 40 , 45 : 45 , 46 : 46 , 47 : 47 , 51 : 51 , 52 : 52 , 55 : 55 , 61 : 61 , 63 : 63 , 65 : 65 , 66 : 66 , 67 : 67 , 68 : 68 , 69 : 69 , 73 : 73 , 74 : 74 , 75 : 75 , 78 : 78 , 81 : 81 , 83 : 83 , 84 : 84 , 86 : 86 , 88 : 88 , 90 : 90 , 92 : 92 , 95 : 95 , 98 : 98 , 99 : 99 , 108 : 108 , 112 : 112 , 116 : 116 , 117 : 117 , 118 : 118 , 120 : 120 , 121 : 121 , 122 : 122 , 123 : 123 , 124 : 124 , 125 : 125 , 127 : 127 , 128 : 128 , 131 : 131 , 133 : 133 , 134 : 134 , 140 : 140 , 141 : 141 , 142 : 142 , 143 : 143 , 144 : 144 , 147 : 147 , 149 : 149 , 151 : 151 , 152 : 152 , 155 : 155 , 157 : 157 , 158 : 158 , 162 : 162 , 163 : 163 , 164 : 164 , 169 : 169 , 170 : 170 , 175 : 175 , 177 : 177 , 183 : 183 , 186 : 186 , 187 : 187 , 188 : 188 , 196 : 196 , 201 : 201 , 204 : 204 , 205 : 205 , 206 : 206 , 210 : 210 , 212 : 212 , 215 : 215 , 217 : 217 , 220 : 220 , 221 : 221 , 222 : 222 , 226 : 226 , 227 : 227 , 229 : 229 , 230 : 230 , 231 : 231 , 232 : 232 , 234 : 234 , 236 : 236 , 242 : 242 , 243 : 243 , 244 : 244 , 245 : 245 , 246 : 246 , 247 : 247 , 253 : 253 , 254 : 254 , 255 : 255 , 256 : 256 , 258 : 258 , 259 : 259 , 260 : 260 , 263 : 263 , 265 : 265 , 267 : 267 , 268 : 268 , 274 : 274 , 278 : 278 , 281 : 281 , 285 : 285 , 288 : 288 , 289 : 289 , 290 : 290 , 294 : 294 , 295 : 295 , 297 : 297 , 298 : 298 ,

--- adjusting weights using phi factors in file balance_phi_groups.csv ---

--- pre-weight-adjustment initial phi summary ---
phi type mean std min max
measured 1.6078e+71 2.07774e+72 48538.9 2.68503e+73
actual 4.70673e+73 7.0726e+73 1.36725e+71 4.71011e+74
note: 'measured' phi reported above includes
realizations of measurement noise,
'actual' phi does not.
--- observation group phi summary ---
(computed using 'actual' phi)
(sorted by mean phi)

group mean std min max percent std
less_flow 4.70673e+73 7.0726e+73 1.36725e+71 4.71011e+74 100 1.40803e-14
flow 409071 358619 38956.3 1.77337e+06 9.86789e-66 3.40439e-65
tritium_measured_32_29 741.088 345.847 0.590784 1177.7 2.87015e-68 1.08343e-67
tritium_measured_45_31 458.344 255.865 0.149516 763.187 1.46471e-68 6.0458e-68
...

--- checking phi factors in file balance_phi_groups.csv ---
file tag 'FLOW maps to groups FLOW,LESS_FLOW,
file tag 'HEAD maps to groups
file tag 'TRITIUM_MEASURED maps to groups

...using mean residuals for reweighting
...original mean phi: 4.70673e+73
...file tag 'FLOW' original mean phi (factor): 4.70673e+73 (1)
...file tag 'HEAD' original mean phi (factor): 4305.84 (9.14828e-71)
...file tag 'TRITIUM_MEASURED' original mean phi (factor): 5778.73 (1.22776e-70)
...file tag 'FLOW' adjusted mean phi (factor): 4.70673e+75 (100)
...file tag 'HEAD' adjusted mean phi (factor): 4.70673e+75 (100)
...file tag 'TRITIUM_MEASURED' adjusted mean phi (factor): 4.70673e+75 (100)
...saved adjusted weight ensemble to haxwn_wairau_14_s1_500.adjusted.weights.jcb

--- initial phi summary ---
phi type mean std min max
measured 1.1638e+76 6.23127e+75 6.9343e+75 6.22473e+76
actual 1.41202e+76 9.67782e+75 6.33308e+75 7.74729e+76
note: 'measured' phi reported above includes
realizations of measurement noise,
'actual' phi does not.
--- observation group phi summary ---
(computed using 'actual' phi)
(sorted by mean phi)

[jtwhite79]

That might need to be greater_flow if you want to keep flow values greater than a threshold? Maybe I'm misunderstanding...

[wkitlasten]

Nah, I flipped between using "greater_tritium' (for reals with low tritium concentrations) and 'less_flow' (for reals with high flows) and neglected to tell anyone.

I think 'ies_phi_factor_file' was picking up 'less_flow' with the 'flow' key. Does that sound plausible? I changed the obsgp to 'less_flw' and it seems to have fixed the last issue I feebly attempted to communicate.

But the approach works well for me. Thanks.