Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: HDF5 testing logs #29

Merged
merged 2 commits into from
Mar 17, 2022
Merged

Conversation

nawtrey
Copy link
Contributor

@nawtrey nawtrey commented Mar 10, 2022

  • Add 5 darshan logs, each containing H5D and/or
    H5F data for testing HDF5 data aggregation tools
    in the new Python summary report

Targeted at darshan-hpc/darshan#663, these logs cover various cases that can be used to test some of the HDF5 aggregation.

For confirmation, here are some local prints of the relevant module data:

hdf5_file_opens_only.darshan
File:  hdf5_file_opens_only
Modules:  ['POSIX', 'H5F']

H5F DataFrame: 
   rank  H5F_OPENS  H5F_FLUSHES
0     0          1            0
1     1          1            0
2     2          1            0
hdf5_diagonal_write_1_byte_dxt.darshan
File:  hdf5_diagonal_write_1_byte_dxt
Modules:  ['POSIX', 'H5F', 'H5D', 'DXT_POSIX']

H5F DataFrame: 
   rank  H5F_OPENS  H5F_FLUSHES
0     0          1            0
1     1          1            0
2     2          1            0
3     3          1            0
4     4          1            0
5     5          1            0
6     6          1            0
7     7          1            0
8     8          1            0
9     9          1            0

H5D DataFrame: 
   rank  H5D_OPENS  H5D_FLUSHES  H5D_READS  H5D_WRITES  H5D_BYTES_READ  H5D_BYTES_WRITTEN
0     0          1            0          0           1               0                  1
1     1          1            0          0           1               0                  1
2     2          1            0          0           1               0                  1
3     3          1            0          0           1               0                  1
4     4          1            0          0           1               0                  1
5     5          1            0          0           1               0                  1
6     6          1            0          0           1               0                  1
7     7          1            0          0           1               0                  1
8     8          1            0          0           1               0                  1
9     9          1            0          0           1               0                  1
hdf5_diagonal_write_bytes_range_dxt.darshan
File:  hdf5_diagonal_write_bytes_range_dxt
Modules:  ['POSIX', 'H5F', 'H5D', 'DXT_POSIX']

H5F DataFrame: 
   rank  H5F_OPENS  H5F_FLUSHES
0     0          1            0
1     1          1            0
2     2          1            0
3     3          1            0
4     4          1            0
5     5          1            0
6     6          1            0
7     7          1            0
8     8          1            0
9     9          1            0

H5D DataFrame: 
   rank  H5D_OPENS  H5D_FLUSHES  H5D_READS  H5D_WRITES  H5D_BYTES_READ  H5D_BYTES_WRITTEN
0     0          1            0          0           1               0                 10
1     1          1            0          0           1               0                 20
2     2          1            0          0           1               0                 30
3     3          1            0          0           1               0                 40
4     4          1            0          0           1               0                 50
5     5          1            0          0           1               0                 60
6     6          1            0          0           1               0                 70
7     7          1            0          0           1               0                 80
8     8          1            0          0           1               0                 90
9     9          1            0          0           1               0                100
hdf5_diagonal_write_half_flush_dxt.darshan
File:  hdf5_diagonal_write_half_flush_dxt
Modules:  ['POSIX', 'H5F', 'H5D', 'DXT_POSIX']

H5F DataFrame: 
   rank  H5F_OPENS  H5F_FLUSHES
0     0          1            1
1     1          1            0
2     2          1            1
3     3          1            0
4     4          1            1
5     5          1            0
6     6          1            1
7     7          1            0
8     8          1            1
9     9          1            0

H5D DataFrame: 
   rank  H5D_OPENS  H5D_FLUSHES  H5D_READS  H5D_WRITES  H5D_BYTES_READ  H5D_BYTES_WRITTEN
0     0          1            0          0           1               0                  1
1     1          1            0          0           1               0                  1
2     2          1            0          0           1               0                  1
3     3          1            0          0           1               0                  1
4     4          1            0          0           1               0                  1
5     5          1            0          0           1               0                  1
6     6          1            0          0           1               0                  1
7     7          1            0          0           1               0                  1
8     8          1            0          0           1               0                  1
9     9          1            0          0           1               0                  1
hdf5_diagonal_write_half_ranks_dxt.darshan
File:  hdf5_diagonal_write_half_ranks_dxt
Modules:  ['POSIX', 'H5F', 'H5D', 'DXT_POSIX']

H5F DataFrame: 
   rank  H5F_OPENS  H5F_FLUSHES
0     0          1            0
1     1          1            0
2     2          1            0
3     3          1            0
4     4          1            0
5     5          1            0
6     6          1            0
7     7          1            0
8     8          1            0
9     9          1            0

H5D DataFrame: 
   rank  H5D_OPENS  H5D_FLUSHES  H5D_READS  H5D_WRITES  H5D_BYTES_READ  H5D_BYTES_WRITTEN
0     0          1            0          0           1               0                  1
1     2          1            0          0           1               0                  1
2     4          1            0          0           1               0                  1
3     6          1            0          0           1               0                  1
4     8          1            0          0           1               0                  1

All of the DXT logs have a diagonal IO pattern similar to the logs from gh-22. The heatmap for hdf5_diagonal_write_1_byte_dxt.darshan (only "write") data:
hdf5_testing

Not super important for HDF5 testing, but for hdf5_diagonal_write_half_ranks_dxt.darshan, you can actually observe a difference between ranks that wrote data out and ranks that didn't in the DXT data:
hdf5_testing_half_ranks

* Add 5 darshan logs, each containing `H5D` and/or
`H5F` data for testing HDF5 data aggregation tools
in the new Python summary report
Copy link
Collaborator

@tylerjereddy tylerjereddy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I checked the logs locally. The detail looks good for reproducing the results in the future, and for expanding when read support becomes available via Python bindings.

Copy link
Contributor

@carns carns left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The logs look great, I just requested a few clarifications in the description so that we can more quickly remember interesting details here when we look back at it later.

* Add comment explaining the `h5py` library was used to
generate the HDF5 logs, with an HDF5 configuration that
didn't use MPI-IO
@carns carns merged commit 1c2a0a7 into darshan-hpc:main Mar 17, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants