Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download check for cache directory #559

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gerhardol
Copy link
Contributor

Add a marker .download file to validate the contents in cache directories. Previously only the existence of the directory was used, so if the download was aborted the cache directory had to be deleted manually if this occurred (with a likely cryptic error message). If the .download check file does not exist, the directory will be deleted and downloaded again.

It is also possible to check the contents with a checksum. If not matching, the directory will be deleted and downloaded again.

For Git repos the repos can be deleted if the status is not clean, a checksum is not relevant (but used in the tests).

Note: The important part of this PR is the .download file, that will cover the majority of the issues.

Add a marker .download file to validate the contents in cache directories.
Previously only the existence of the directory was used, so if the
download was aborted the cache directory had to be deleted manually if
this occurred (with a likely cryptic error message).
If the .download check file does not exist, the directory will be deleted
and downloaded again.

It is also possible to check the contents with a checksum.
If not matching, the directory will be deleted and downloaded again.

For Git repos the repos can be deleted if the status is not clean,
a checksum is not relevant (but used in the tests).
@kyle-verdant
Copy link

Does this handle submodules well? The failures I most often saw were not that the main directory was empty, but that some required submodule didn't exist - most often with aws-sdk-cpp (don't use aws-sdk-cpp with CPM, it's too big).

@gerhardol
Copy link
Contributor Author

Does this handle submodules well? The failures I most often saw were not that the main directory was empty, but that some required submodule didn't exist - most often with aws-sdk-cpp (don't use aws-sdk-cpp with CPM, it's too big).

This will currently make no change if the download is aborted. CMake FetchContents makes some retries, then the job fails. If CMake reports this correctly, then the marker file is not created.
I do not know if that is an issue really, have not got good statistics why the core problem occurs.

The change is for cache and reusing the downloaded data. If no marker file, the directory is incomplete and will be deleted and downloaded again.
I assume CMake detects failures correctly, this should be the core change.

If checksum is used, it could handle also failed checksums.
With Git you do not need to use a checksum. If you set CPM_CHECK_CACHE_CHECKSUM then the directory is removed if git-status fail.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants