Migrate to MS Graph for AAD operations

[manicminer]

Overview

Migrate away from the Azure Active Directory Graph API and use Microsoft Graph for Azure Active Directory operations: creating and removing applications, passwords and service principals

This change is necessary as the AAD Graph API is deprecated and will no longer be supported from June 2022. Additionally, this API is feature frozen and exhibits longer delays due to eventual consistency.

Impact on practitioners will include:

• Having to set new permissions for the service principal which they provide to Vault to manage ephemeral service principals
  • These will include these permissions on Microsoft Graph: Applications.ReadWrite.All and Groups.ReadWrite.All
• Users should observe marginally faster provisioning of service principal credentials and encounter fewer consistency errors due to service-side replication delay

Design of Change

This makes use of manicminer/hamilton which is the same library being used by the Terraform AzureAD Provider.

There are some caveats, related to the API (not the library):

• It's no longer possible to specify Key IDs for passwords, so the current trick of using a prefixed GUID with leading ffffff does not work
• Passwords can no longer be specified, they are computed by Azure and returned in the response to /addPassword (and never subsequently)

Related Issues/Pull Requests

[ ] Issue #58 - resolved by this change because the new API uses a transactional /addPassword endpoint instead of racily retrieving existing credentials, inserting a new one client-side, and posting the resulting array
[ ] Issue #30 - should also resolve this issue by using the newer API which is easier to grant permissions for (AAD Graph is missing many permissions which forces practitioners to resort to directory roles)

Contributor Checklist

[ ] Add relevant docs to upstream Vault repository, or sufficient reasoning why docs won’t be added yet
My Docs PR Link - docs not yet added, kindly request code review first
[ ] Add output for any tests not ran in CI to the PR description (eg, acceptance tests)
[ ] Backwards compatible - whilst no compatibility issues exist, practitioners will need to review the API permissions given to Vault for Microsoft Graph

Test Results

Both the mocked and real tests are passing for me, using a MITM proxy to verify the API requests and responses are as expected.

make test

==> Checking that code complies with gofmt requirements...
go generate
VAULT_ACC=1 go test -tags='vault-plugin-secrets-azure' $(go list ./... | grep -v /vendor/) -v  -timeout 45m
=== RUN   TestRetry
=== PAUSE TestRetry
=== RUN   TestConfig
--- PASS: TestConfig (0.00s)
=== RUN   TestConfigDelete
--- PASS: TestConfigDelete (0.00s)
=== RUN   TestRoleCreate
=== RUN   TestRoleCreate/SP_role
=== RUN   TestRoleCreate/Static_SP_role
=== RUN   TestRoleCreate/Optional_role_TTLs
=== RUN   TestRoleCreate/Role_TTL_Checks
=== RUN   TestRoleCreate/Role_name_lookup
=== RUN   TestRoleCreate/Group_name_lookup
=== RUN   TestRoleCreate/Duplicate_role_name_and_scope
=== RUN   TestRoleCreate/Duplicate_role_name,_different_scope
=== RUN   TestRoleCreate/Duplicate_group_object_ID
=== RUN   TestRoleCreate/Role_name_lookup_(multiple_match)
=== RUN   TestRoleCreate/Group_name_lookup_(multiple_match)
--- PASS: TestRoleCreate (0.00s)
    --- PASS: TestRoleCreate/SP_role (0.00s)
    --- PASS: TestRoleCreate/Static_SP_role (0.00s)
    --- PASS: TestRoleCreate/Optional_role_TTLs (0.00s)
    --- PASS: TestRoleCreate/Role_TTL_Checks (0.00s)
    --- PASS: TestRoleCreate/Role_name_lookup (0.00s)
    --- PASS: TestRoleCreate/Group_name_lookup (0.00s)
    --- PASS: TestRoleCreate/Duplicate_role_name_and_scope (0.00s)
    --- PASS: TestRoleCreate/Duplicate_role_name,_different_scope (0.00s)
    --- PASS: TestRoleCreate/Duplicate_group_object_ID (0.00s)
    --- PASS: TestRoleCreate/Role_name_lookup_(multiple_match) (0.00s)
    --- PASS: TestRoleCreate/Group_name_lookup_(multiple_match) (0.00s)
=== RUN   TestRoleCreateBad
--- PASS: TestRoleCreateBad (0.00s)
=== RUN   TestRoleUpdateError
--- PASS: TestRoleUpdateError (0.00s)
=== RUN   TestRoleList
--- PASS: TestRoleList (0.00s)
=== RUN   TestRoleDelete
--- PASS: TestRoleDelete (0.00s)
=== RUN   TestSP_WAL_Cleanup
=== RUN   TestSP_WAL_Cleanup/Role_assign_fail
2021-07-29T02:54:22.767+0100 [DEBUG] rolling back service principal: objID=894740d9-654c-9c6f-b370-1fe682553cb9
2021-07-29T02:54:22.767+0100 [DEBUG] rolling back application: appID=b0ea0e65-d6d1-030a-b1e8-bc070b5a2298 appObjID=01672200-18fb-dce3-d901-00dc8b5cd194
--- PASS: TestSP_WAL_Cleanup (5.00s)
    --- PASS: TestSP_WAL_Cleanup/Role_assign_fail (5.00s)
=== RUN   TestSPRead
=== RUN   TestSPRead/Basic_Role
=== RUN   TestSPRead/Basic_Group
=== RUN   TestSPRead/TTLs
--- PASS: TestSPRead (0.00s)
    --- PASS: TestSPRead/Basic_Role (0.00s)
    --- PASS: TestSPRead/Basic_Group (0.00s)
    --- PASS: TestSPRead/TTLs (0.00s)
=== RUN   TestStaticSPRead
=== RUN   TestStaticSPRead/Basic
=== RUN   TestStaticSPRead/TTLs
--- PASS: TestStaticSPRead (0.00s)
    --- PASS: TestStaticSPRead/Basic (0.00s)
    --- PASS: TestStaticSPRead/TTLs (0.00s)
=== RUN   TestSPRevoke
=== RUN   TestSPRevoke/roles
=== RUN   TestSPRevoke/groups
--- PASS: TestSPRevoke (0.00s)
    --- PASS: TestSPRevoke/roles (0.00s)
    --- PASS: TestSPRevoke/groups (0.00s)
=== RUN   TestStaticSPRevoke
--- PASS: TestStaticSPRevoke (0.00s)
=== RUN   TestSPReadMissingRole
--- PASS: TestSPReadMissingRole (0.00s)
=== RUN   TestCredentialReadProviderError
--- PASS: TestCredentialReadProviderError (0.00s)
=== RUN   TestCredentialInteg
=== RUN   TestCredentialInteg/SP
=== PAUSE TestCredentialInteg/SP
=== RUN   TestCredentialInteg/Static_SP
=== PAUSE TestCredentialInteg/Static_SP
=== CONT  TestCredentialInteg/SP
=== CONT  TestCredentialInteg/Static_SP
--- PASS: TestCredentialInteg (0.00s)
    --- PASS: TestCredentialInteg/Static_SP (53.15s)
    --- PASS: TestCredentialInteg/SP (73.65s)
=== CONT  TestRetry
=== RUN   TestRetry/First_try_success
=== RUN   TestRetry/Three_retries
=== PAUSE TestRetry/Three_retries
=== RUN   TestRetry/Error_on_attempt
=== PAUSE TestRetry/Error_on_attempt
=== RUN   TestRetry/Timeout
=== PAUSE TestRetry/Timeout
=== RUN   TestRetry/Cancellation
=== PAUSE TestRetry/Cancellation
=== CONT  TestRetry/Three_retries
=== CONT  TestRetry/Timeout
=== CONT  TestRetry/Cancellation
=== CONT  TestRetry/Error_on_attempt
--- PASS: TestRetry (0.00s)
    --- PASS: TestRetry/First_try_success (0.00s)
    --- PASS: TestRetry/Error_on_attempt (0.00s)
    --- PASS: TestRetry/Cancellation (7.00s)
    --- PASS: TestRetry/Timeout (10.00s)
    --- PASS: TestRetry/Three_retries (11.49s)
PASS
ok  	github.com/hashicorp/vault-plugin-secrets-azure	90.420s
?   	github.com/hashicorp/vault-plugin-secrets-azure/cmd/vault-plugin-secrets-azure	[no test files]

[manicminer]

Just a note that this could be implemented progressively, supporting both APIs so users have the opportunity to opt in to MS Graph before AAD Graph support is removed (this is the approach we took in the Terraform AzureAD Provider).

[pcman312]

Closing this as MS Graph support was added in #67