Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Run GEOSgcm.x from install directory #184

Open
mathomp4 opened this issue Dec 1, 2020 · 12 comments
Open

Run GEOSgcm.x from install directory #184

mathomp4 opened this issue Dec 1, 2020 · 12 comments
Assignees
@mathomp4
Labels
question Further information is requested

Comments

@mathomp4
Copy link
Member

mathomp4 commented Dec 1, 2020 •
edited

This is a question that might be useful to fix. As @bena-nasa found in his testing with the git GEOSadas, the DAS folks all run from install/bin. But gcm_run.j does not. Instead gcm_setup copies the executable into the experiment directory and then we copy that into the scratch/ directory and run from there. Why? Historical reasons? I'm not sure.

But this has interesting effects currently. It is precisely because of this fact why we had to add $GEOSDIR/lib to the LD_LIBRARY_PATH. You can see why if we look at ldd. First, let us run ldd ./GEOSgcm.x inside the install dir:

$ ldd ./GEOSgcm.x | grep mom
	libmom.so => /gpfsm/dnb02/mathomp4/SystemTests/builds/AGCM/CURRENT/GEOSgcm/install-Release/bin/./../lib/libmom.so (0x00002aaab204d000)
	libmom6.so => /gpfsm/dnb02/mathomp4/SystemTests/builds/AGCM/CURRENT/GEOSgcm/install-Release/bin/./../lib/libmom6.so (0x00002aaab7874000)

Now I'll create an experiment and try with that copy:

$ create_expt.py stock-gcm-2020Dec01-1day-c24 
...
$ ldd /discover/nobackup/mathomp4/Experiments/stock-gcm-2020Dec01-1day-c24/GEOSgcm.x | grep mom
	libmom.so => not found
	libmom6.so => not found

Now with the work that @tclune and @weiyuan-jiang are doing with DSOs maybe this is moot...or maybe it's more important? I don't know. But it would be fairly simple to instead of doing:

    $RUN_CMD $NPES ./GEOSgcm.x $IOSERVER_OPTIONS --logging_config 'logging.yaml'

we could just do:

    $RUN_CMD $NPES ${GEOSBIN}/GEOSgcm.x $IOSERVER_OPTIONS --logging_config 'logging.yaml'

just like how we run binarytile.x:

$GEOSBIN/binarytile.x tile.data tile.bin

However, perhaps @sdrabenh or @lltakacs have opinions on this?
@mathomp4 mathomp4 added the question Further information is requested label Dec 1, 2020
@mathomp4 mathomp4 self-assigned this Dec 1, 2020
@mathomp4
Copy link
Member Author

mathomp4 commented Dec 1, 2020

Note that if we want to keep the copies, I think then @tclune and I would need to look at hardcoding the RPATH in the executable. I think right now it uses $ORIGIN or something? CMake and RPATH is confusing at the best of time.

@lltakacs
Copy link
Contributor

lltakacs commented Dec 1, 2020 via email

@mathomp4
Copy link
Member Author

mathomp4 commented Dec 2, 2020

The reason we copied the executable into the experiment directory is due to the fact that we routinely make “simple” source code updates, recompile and relink, and then run a new test with the new executable. Basically, all experiments are identical except for a few simple source code updates. So, they were all run under a single experiment ID. Sometimes, these tests were done by simply changing AGCM.rc parameters, but not all tests were based on a simple parameter change. Actually, the DAS also has this ability. If you place a GEOSgcm.x in the run directory, it will over-ride the GEOSgcm.x in the Linux/bin directory. Generally, however, the DAS never updates the GEOSgcm.x.

Maybe that would be a good compromise? If there is a GEOSgcm.x in the experiment directory, we run from that. Otherwise, we run from the one in install/bin?

Though, I guess, if you make a simple source code change, you'll need to recompile anyway which means the one in install/bin is the one you'd want? Right?

@tclune
Copy link
Collaborator

tclune commented Dec 2, 2020

I think we should investigate how this relates with @weiyuan-jiang 's changes. Sounds like there is at least a possibility that the issue will largely evaporate. I'm sure we'll need the LD_LIBRARY_PATH to be set though.

@lltakacs
Copy link
Contributor

lltakacs commented Dec 2, 2020 via email

@mathomp4
Copy link
Member Author

mathomp4 commented Dec 2, 2020

Oh yeah. I guess I forget that some people run the model for more than a model day/15 wall-minutes! 😄

Hmm. Interesting. I wonder if you can specify a different install directory after changing one file and get a complete new install? If not, I might want to look at seeing if we can make that path "hardcoded" rather than relative.

@tclune
Copy link
Collaborator

tclune commented Dec 2, 2020

Yes - you can definitely alter the install directory late in the game like that. I've done it recently. One has to be careful though, as if you forget that step, you will overwrite the executable as Larry said. Making a copy provides insulation against such mistakes.

Flip side - what if one copies the entire install directory?

@weiyuan-jiang
Copy link
Contributor

Why do you oppose to set LD_LIBRARY_PATH? Any side effect? @mathomp4

@mathomp4
Copy link
Member Author

mathomp4 commented Dec 2, 2020

@weiyuan-jiang No real opposition, but before the mom5/mom6 shared object transition this was possible:

  1. User makes an experiment
  2. Run gcm_run.j and oops something crashed.
  3. Go into the scratch/ directory and, say, fiddle with AGCM.rc
  4. Run mpirun -np N ./GEOSgcm.x directly

Now with GEOSgcm.x this isn't possible unless LD_LIBRARY_PATH in your environment has install/lib because the libmom.so and libmom6.so will not be resolved. Running gcm_run.j again is of course doable, but the executable is now a little less portable. I'm sure I could figure out how to hard-code the paths in GEOSgcm.x though if we'd want to...

Though as Tom says, if we go full DSO, this might be all moot.

@mathomp4
Copy link
Member Author

mathomp4 commented Dec 2, 2020 •
edited

Yes - you can definitely alter the install directory late in the game like that. I've done it recently. One has to be careful though, as if you forget that step, you will overwrite the executable as Larry said. Making a copy provides insulation against such mistakes.

Flip side - what if one copies the entire install directory?

@tclune My thought (that I'm going to try now) is what does this do:

  1. Build GEOS as one normally does building in build/ installing into install/
  2. Make a change somewhere
  3. Go back to build/ and then run either
    1. make install DESTDIR=newinstalldir
    2. cmake -DCMAKE_INSTALL_PREFIX=newinstalldir && make install

I'm wondering: do you get a "full" install? Or does it only install the new files? I've never tried! I will find out in about 30 minutes...

@mathomp4
Copy link
Member Author

mathomp4 commented Dec 2, 2020

Update: DESTDIR is weird. This:

make install DESTDIR=../install-Debug-Again

installed to:

install-Debug-Again/discover/swdev/mathomp4/Models/GEOSgcm-InstallChangeInstall/GEOSgcm/install-Debug/

and it wasn't a complete install. So don't use.

But:

cmake .. -DCMAKE_INSTALL_PREFIX=../install-Debug-ReCmake
make install

worked as expected. Hmm.

@tclune
Copy link
Collaborator

tclune commented Dec 2, 2020

Yes the latter is all that I attempted. Have not played with DESTDIR in a while. Vague memory that I found it confusing at best. Maybe only suitable for system installs?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mathomp4 mathomp4

Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mathomp4 @tclune @weiyuan-jiang @lltakacs