fpe0 error

[wmputman]

I tried building and running with FPE0 in place of our FPE3 options. I landed on this line with a floating point exception:

forrtl: error (75): floating point exception
Image PC Routine Line Source
GEOSgcm.x 00000000035C3D1B Unknown Unknown Unknown
libpthread-2.22.s 00002B3E6BD4DCE0 Unknown Unknown Unknown
libMAPL.base.so 00002B3E5BD31694 mapl_locstreammod 1965 MAPL_LocStreamMod.F90
GEOSgcm.x 0000000001E510C5 geos_surfacegridc 4413 GEOS_SurfaceGridComp.F90

[mathomp4]

Huh. That is in some boring code:

    if(associated(   LAI)) then
       call MAPL_LocStreamTransform( LOCSTREAM,    LAI,    LAITILE, RC=STATUS) 
       VERIFY_(STATUS)
    endif

to:

MAPL/base/MAPL_LocStreamMod.F90

Lines 1962 to 1968 in d6b49c8

	if (usableTRANSPOSE) then
	else
	where(FF<=0)
	!jk
	OUTPUT = MAPL_Undef
	end where
	endif

@atrayano would be our best bet but as he's off, I'm going to cc @tclune and @bena-nasa as well as @weiyuan-jiang as he is our land contact.

My first thought is somehow FF is...denormal or stupid large? That's the only way I could see an FPE happening on an if < 0 line.

So, perhaps the LAI (or GRN, etc.) files are not good? Unfortunately they are tile files. @biljanaorescanin Does the land group have an LAI/GRN "viewer"?

[biljanaorescanin]

We do have as part of monitoring BCS package under specific resolution monthly plots for LAI for that resolution. Is that what you mean by "viewer"?
Our LAI raster is at ~1km resolution so maybe you are running it too close to raster resolution? @gmao-rreichle any ideas?

What ever you've run you will have bcs files if you go to that directory, for example:

/discover/nobackup/projects/gmao/bcs_shared/legacy_bcs/Icarus-NLv3/Icarus-NLv3_Ostia/CF3072x6C_DE2880xPE1440/clsm/plots/lai.jpg

you will have "lai.jpg" you can check does it look reasonable. Similar plot should exist in any bcs set this run was using.

[mathomp4]

@biljanaorescanin Yes. I was thinking of something like that. Where we could feed in the stretched grid lai/grn files and maybe see a picture. (And, perhaps, get a min/max/mean of the fields?)

My concern is that there are stray denormals or NaNs in the files that we don't know about and then it gets hit.

[weiyuan-jiang]

@biljanaorescanin I shared this concern

[gmao-rreichle]

Not entirely sure what the issue is, here. Has anyone tried reading the LAI values and looking if one or more values are not in the expected range (non-negative and less than, say, 10.0)?
The LAI clim is still binary, right? Below is an old matlab reader from when the data were monthly. Now we have 8-day data, so should be 46 (?) records instead of 12. @biljanaorescanin, can you adapt the reader and take a look?

function [ lai_monthly ] = read_monthly_lai( fpath, N_tile_g )

% read monthly climatological LAI values from GEOS-5 parameter file "lai.dat"
%
%  fpath     = path to "lai.dat"
%  N_tile_g  = number of tiles in global "lai.dat" file
%
% reichle, 16 March 2011
%
% ---------------------------------------------------------------

%fpath = '/l_data/geos5/bcs/SiB2_V2/DE/SMAP_EASE_M36/VEGETATION-GSWP2/LAI_GRN_CLIMATOLOGY';
  
%N_tile_g = 110903;
   
% ----------------

int_precision   = 'int32';      % precision of fortran tag
float_precision = 'float32';    % precision of data in input file

fname = [ fpath, '/lai.dat'];

disp(['reading from ', fname])

ifp = fopen( fname, 'r', 'b' );

lai_monthly = NaN*ones(N_tile_g,12);

for i=1:12
  
  fortran_tag = fread( ifp, 1, int_precision );
  tmp_data    = fread( ifp, [1 N_tile_g], float_precision );
  fortran_tag = fread( ifp, 1, int_precision );
  
  lai_monthly(:,i) = tmp_data;
  
end

fclose(ifp);

% ==================== EOF =========================================

[gmao-rreichle]

PS: I don't think the plotting package would necessarily show a handful of bad numbers. Reading the data and checking for values outside of [0,10] should be a more robust way of identifying potential issues with the LAI or GRN values.

[tclune]

To @wmputman : My fear, as I relayed to Atanas before his travel is that this is just the tip of the iceberg. You may encounter a series of issues to get fpe0 to work with GEOS. I would argue it is very useful for us to fix these, but the timeline is potentially open-ended.

[biljanaorescanin]

I just ran a check on the LAI values (as @gmao-rreichle suggested).
/discover/nobackup/projects/gmao/osse2/stage/BCS_FILES/CF1536x6C_CF1536x6C/lai_clim_1536x9216.data
For the following file there is no LAI values lower than 0.0 or greater than 10. Also, an IDL check for NaN does not return any counts.

[mathomp4]

Well...dang. Can you try the files in these as well:

/gpfsm/dnb03/projects/p38/stage/BCS_FILES/CF1080x6C_CF1080x6C
/gpfsm/dnb03/projects/p38/stage/BCS_FILES/CF0540x6C_CF0540x6C
/gpfsm/dnb03/projects/p38/stage/BCS_FILES/CF2160x6C_CF2160x6C

Also, can you check all the other land-ish files? green, lai, lnfm, and ndvi? If you can script that up. Just in case it's not LAI but one of the others.

Thanks

[gmao-rreichle]

I don't think NDVI is used in the GCM. I forget what LNFM might be. Maybe something with carbon (CatchCN). Since I can't remember, LNFM probably not used either.
GRN is used and should be [0,1], but if LAI is ok, chances are GRN is also ok.

[biljanaorescanin]

@mathomp4 I've tested other stretched bcs resolutions and both LAI and GREEN are in range with no NAN's. LNFM is used just with CatchCN and as Rolf said NDVI is not used so I didn't do those two.

[mathomp4]

Well, nuts. Thanks for looking, @biljanaorescanin. It was a long shot but at least a theory that could be easily tested and solved if the issue.

Now I'm afraid it's some sort of memory corruption and the LAI/GRN was just the place that caught it.

[wmputman]

This issue has now come up 2 more times, once in July for me, and now again for Donifan.

I was able to run with my debug executables if I set my GEOSDIR to a build Matt made, and thus the LD_LIBRARY_PATH was different.

Anyone able to reproduce this problem within the SI team?

[atrayano]

I tried to reproduce the same issue for Donifan, running a debug executable. While I was not able to reproduce the issue in T2G, I found few other issues: division by zero due to uninitialized variable (PK), and another usage of uninitialized variable (imsize). Also found a usage of a scalar variable (ALPHA) calculated inside one set of nested loops and used also in another set of nested loops, which would have caused a layout reproducibility issues.The T2G issue could have been caused by memory corruption and or corrupted tile file. I am not 100% sure that the issues I found could explain the T2G

[atrayano]

Did all these instances of T2G crashed happen after a fresh build? I have seen occasionally unexplainable errors related to a "stale" cmake build

[dbarahon]

v12_rc1 has this issue again, even when running the default microphysics (BAC_1M). I guess it was never fixed.

[atrayano]

@dbarahon Thanks for reporting the issues. I am almost sure the the MAPL_LocStream code (T2G) is fine, and the code is "catching" a NaN in the input data. Could you point me to a reproducer?

[dbarahon]

I did a fresh checkout of the feature/sdrabenh/gcm_v12-rc1 tag, built with the -debug option and try it in c90 exp. My built is located at /gpfsm/dnb34/dbarahon/models/GCMv12_rc1 and the experiment here /gpfsm/dnb34/dbarahon/DEV/v12_rc1. The latest log file is slurm-32334802.out. This issue may be related to the #805 issue I reported before.