Implement recovery of HTCondor spool (job queue) #2500
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tpdownes
force-pushed
the
htcondor_ap_stateful
branch
from
818119c
to
1c19b66
Compare
tpdownes
added
release-module-improvements
rohitramu
approved these changes
May 9, 2024
tpdownes
force-pushed
the
htcondor_ap_stateful
branch
3 times, most recently
from
df5ca11
to
516f181
Compare
Implement high availability of the HTCondor job queue by using a managed instance group to provision a single VM with "stateful": - public or private IP address - disk containing the spool (job queue and related data) In a single zone, "HA" means reliability against a failure of the access point VM. In two zones, "HA" means reliability against a failure of the access points' zone. When configured to use two zones, the disk is replicated synchronously. Future work: when using 2 zones, identify solution for aligning zones between HTCondor pool components (Central Manager, Execute Points)
…mponents Use ANY_SINGLE_ZONE to increase the odds that HTCondor core components are provisioned in the same zone by the regional Managed Instance Group. Reasoning: 1. This target shape prioritizes the zone with the most reservations (therefore reservations can guide alignment) 2. The next priority is resource availability within the zone, which should also serve to align resources. 3. Align machine_type for Central Manager and Access Point
Replace a fixed-wait sleep with the built-in feature that Terraform offers to wait for instance creation in MIGs.
tpdownes
force-pushed
the
htcondor_ap_stateful
branch
from
9266a58
to
1eeab10
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Implement high availability of the HTCondor job queue by using a managed instance group to provision a single VM with "stateful":
In a single zone, this configuration provides reliability against a failure of the access point VM by enabling the stateful disk to be mounted by a new VM when the MIG replaces it. In two zones, this is a proper "HA" configuration meaning reliability against a failure of the access points' zone. When configured to use two zones, the disk is replicated synchronously.
In the 2nd commit, we promote alignment of zones between the HTCondor pool components by using a different MIG target shape. In practice, I observe higher alignment in manual testing. Future work will identify a more robust solution (likely by having htcondor-setup output 2 zones)
Submission Checklist
Please take the following actions before submitting this pull request.