Integration Test suite fails on modern versions of vault

[adammike]

The hvac pipeline runs all of the linting and documentation tests against vault 1.7.2+ent. The integration tests run on 1.4.7+ent, 1.5.9+ent, 1.6.5+ent, and 1.7.2+ent

Vault 1.9, 1.10, and 1.11 are supported. Hashicorp's official policy is that they support vault for the two previous major releases or 2 years, whichever is shorter. That means that our pipeline is currently testing on severely out of date versions of vault.

I've ran our test suite on every vault version from 1.6.0 to current and here are my findings:

• ✅ All tests pass on 1.6.0->1.7.10 on enterprise versions of vault

• ❌ Starting in 1.8.0 vault enterprise forces a license check on the dev server, which causes all of our tests to fail on any enterprise vault after 1.7.10.

• ✅ All tests pass on 1.6.0-1.8.12 on open-source versions of vault, but tests for enterprise features are skipped

• ❌ In 1.9.0 we start seeing test failures:

FAILED tests/integration_tests/api/auth_methods/test_oidc.py::TestOIDC::test_oidc_authorization_url_request_0_success - hvac.exceptions.InvalidRequest: cannot find key "oidc-test-key", on post https://localhost:8200/v1/identity/oidc/role/hvac-oidc-test
FAILED tests/integration_tests/api/secrets_engines/test_azure.py::TestAzure::test_delete_config_0_create_and_then_delete_config - AssertionError: '' != 0
FAILED tests/integration_tests/api/secrets_engines/test_identity.py::TestIdentity::test_create_or_update_group_3_create_success_with_group_type - AssertionError: None != []
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_auth_kubernetes - AssertionError: 'claim "iss" is invalid' not found in 'open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory, on post https://localhost:8200/v1/auth/k8s/login'

• ❌ In 1.10 there are more failed tests:

FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_0_just_organization - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_0_just_organization - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_1_different_base_url - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_2_custom_ttl_seconds - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_3_custom_ttl_minutes - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_4_custom_ttl_hours - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_5_custom_max_ttl - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_login_0_valid_token - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_login_1_invalid_token_not_in_org - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_oidc.py::TestOIDC::test_oidc_authorization_url_request_0_success - hvac.exceptions.InvalidRequest: cannot find key "oidc-test-key", on post https://localhost:8200/v1/identity/oidc/role/hvac-oidc-test
FAILED tests/integration_tests/api/secrets_engines/test_azure.py::TestAzure::test_delete_config_0_create_and_then_delete_config - AssertionError: '' != 0
FAILED tests/integration_tests/api/secrets_engines/test_identity.py::TestIdentity::test_create_or_update_group_3_create_success_with_group_type - AssertionError: None != []
FAILED tests/integration_tests/api/system_backend/test_key.py::TestKey::test_start_generate_root_then_cancel - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/api/system_backend/test_key.py::TestKey::test_start_generate_root_with_completion - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_auth_gcp_alternate_mount_point_with_no_client_token_exception - hvac.exceptions.Forbidden: permission denied, on post https://localhost:8200/v1/auth/gcp/login
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_auth_kubernetes - AssertionError: 'claim "iss" is invalid' not found in 'open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory, on post https://localhost:8200/v1/auth/k8s/login'
FAILED tests/integration_tests/v1/test_system_backend.py::TestSystemBackend::test_start_generate_root_then_cancel - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/v1/test_system_backend.py::TestSystemBackend::test_start_generate_root_with_completion - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt

• ❌ And many more failed tests for 1.11

FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_0_just_organization - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_0_just_organization - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_1_different_base_url - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_2_custom_ttl_seconds - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_3_custom_ttl_minutes - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_4_custom_ttl_hours - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_configure_and_read_configuration_5_custom_max_ttl - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_login_0_valid_token - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_github.py::TestGithub::test_login_1_invalid_token_not_in_org - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_jwt.py::TestJWT::test_create_role_0_success - AssertionError: 'data' not found in <Response [204]>
FAILED tests/integration_tests/api/auth_methods/test_kubernetes.py::TestKubernetes::test_configure_4_set_invalid_token_reviewer_jwt - AssertionError: '* not a compact JWS' not found in '1 error occurred:\n\t* square/go-jose: compact JWS format must have three parts\n\n, on post https://localhost:8200/v1/...
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_configure_0_enable_mfa_with_supported_auth_method - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_configure_4_enable_mfa_with_unsupported_auth_method_type_forced - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_configure_duo_access_0_configure_duo_access_success - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_configure_duo_behavior_0_enable_mfa_with_supported_auth_method - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_login_with_mfa_0_login_without_duo_access_configured - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_read_configuration_0_read_configured_path - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_mfa.py::TestMfa::test_read_duo_behavior_configuration_0_read_configured_path - hvac.exceptions.InvalidPath: 1 error occurred:
FAILED tests/integration_tests/api/auth_methods/test_oidc.py::TestOIDC::test_oidc_authorization_url_request_0_success - hvac.exceptions.InvalidRequest: cannot find key "oidc-test-key", on post https://localhost:8200/v1/identity/oidc/role/hvac-oidc-test
FAILED tests/integration_tests/api/secrets_engines/test_azure.py::TestAzure::test_delete_config_0_create_and_then_delete_config - AssertionError: '' != 0
FAILED tests/integration_tests/api/secrets_engines/test_identity.py::TestIdentity::test_create_or_update_group_3_create_success_with_group_type - AssertionError: None != []
FAILED tests/integration_tests/api/system_backend/test_key.py::TestKey::test_start_generate_root_then_cancel - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/api/system_backend/test_key.py::TestKey::test_start_generate_root_with_completion - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_auth_gcp_alternate_mount_point_with_no_client_token_exception - hvac.exceptions.Forbidden: permission denied, on post https://localhost:8200/v1/auth/gcp/login
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_auth_kubernetes - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_create_kubernetes_configuration - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_create_kubernetes_role - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_delete_kubernetes_role - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_get_kubernetes_configuration - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_get_kubernetes_role - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_integration.py::IntegrationTest::test_list_kubernetes_roles - hvac.exceptions.InternalServerError: 1 error occurred:
FAILED tests/integration_tests/v1/test_system_backend.py::TestSystemBackend::test_start_generate_root_then_cancel - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt
FAILED tests/integration_tests/v1/test_system_backend.py::TestSystemBackend::test_start_generate_root_with_completion - hvac.exceptions.InvalidRequest: OTP string is wrong length, on put https://localhost:8200/v1/sys/generate-root/attempt

[yan12125]

FWIW, I collected possible causes for failures with Vault 1.9 in #786.

[adammike]

@colin-pm @briantist most, if not all, of these failures are caused by breaking changes these features. Do we want version specific implementations of these features? Do we want to only support the current version? What are your thoughts on this?

One idea would be for the client object to do a version check pretty early on, and then choose feature implementations based on that version.

[briantist]

While it might be a pain, I think each failure/breaking change should be handled case-by-case,s o perhaps having an issue opened for each one would be best.

I think it's going to be important to recognize whether hvac functionality is currently broken completely (cannot be use used successfully), or if "default" configuration has stopped working but it's still possible to make it work, or whether it's our tests only that need to be updated.

My initial reaction is to avoid an implicit unavoidable version check by the library, in order to avoid extra round trips to the server. I hope in most cases we will not need that kind of workaround, but if we do, my general feeling is that the user of the library should specify the behavior in some way, with a parameter; one of the options for that parameter could be auto where the library will query the version and try to figure it out conditionally. But again I think our choices will vary a lot case by case.

I would also be surprised if Vault had true breaking changes without incrementing major version number, but maybe they aren't using semver? I was not able to find out with a quick search.

[adammike]

perhaps having an issue opened for each one would be best.

My thought was this issue would become a project, and one or more of the failing tests would be grouped into their own issues that are relevant to what is causing the failure.

[briantist]

perhaps having an issue opened for each one would be best.

My thought was this issue would become a project, and one or more of the failing tests would be grouped into their own issues that are relevant to what is causing the failure.

Ok that sounds good too.

---

If we find we truly have breaking changes where we need to change behavior conditionally, due to incompatibilities in Vault, I had another idea for the longer term: we could have multiple major versions of hvac that are under development and released.

Each new one has breaking changes, includes those needed to support newer Vault versions. The older ones keep compatibility with the Vault versions they maintain, and receive new non-breaking changes and bugfixes.

The older stable versions would not be supported forever; we will have to drop support for them on some schedule, probably closely aligned to Vault's support.

There are probably many ways to do this; the way I've seen it done in the world of Ansible collection projects is to have stable-X branches where X is a major version number.

Changes landed to main are applied to stable branches (could be manual cherry-picking but lately I've mostly seen use of the patchback app which handles putting up PRs via labels).

While specific to Ansible collections, this document describes the release branches process as it's done over there.

---

This does add some complication to releasing, but it has the advantage of being able to release new major versions sooner to respond to changes in Vault, without shutting the door on any other changes and improvements being added to a release that's still compatible with an older version.

Most importantly, it means we don't have to add a lot of conditionals or other workarounds to support multiple versions simultaneously. Those could be hard error prone, hard to test, and then there's additional work to remove them safely. With separate major versions, the older stable versions go out of support and we simply stop backporting and releasing to them.

We don't have to do this right away, or before 1.0.0 either, just something to think about.

[colin-pm]

Thanks @adammike and @yan12125 for documenting these test failures.

I created #856, #857, #858, #859, #860, #861, #862, #863, #864, and #865 to break up the test failures into individual issues, grouped by module. Since we would like to have these tests fixed by the 1.0.0 release of HVAC, all issues have been assigned to that work effort.

When I've looked at these tests failures previously, the failures in each module stem from a common issue, so I figure this will give us the granularity to address the failures on a case-by-case basis as @briantist suggested.