Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG]: AzurePowershell@5 Decorator tasks fails to retrieve OIDC token on stage reruns #19758

Open
4 of 7 tasks
GABRIELNGBTUC opened this issue Apr 10, 2024 · 0 comments
Open
4 of 7 tasks

Comments

@GABRIELNGBTUC
Copy link

GABRIELNGBTUC commented Apr 10, 2024

New issue checklist

Task name

AzurePowershell

Task version

5.237.9

Issue Description

In our pipelines, a decorator runs after a deployment job ends to do some optional post deployment work.

This decorator is as follows:

- task: AzurePowerShell@5
      displayName: "dn"
      continueOnError: true
      retryCountOnTaskFailure: 3
      inputs:
        ScriptType: "InlineScript"
        azureSubscription: serviceConnection-$(System.TeamProject)"
        azurePowerShellVersion: LatestVersion
        Inline: |
           do work
          }
      env:
        JOB_STEPS: ${{ convertToJson(job.steps) }}

On the first run of the deployment job, the job succeed without any issue.

However on subsequent attempts of the same run (ie: if the deployment fails and "rerun failed job" is clicked. Or with the new "rerun only this stage" feature), this task fails with:

##[debug]Retrying OIDC token fetch. Retries left: 3
##[debug]Retrying OIDC token fetch. Retries left: 2
##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 689.00 MB out of 3377.00 MB, CPU: Usage 9.56%
##[debug]Retrying OIDC token fetch. Retries left: 1
##[debug]task result: Failed
##[error]run() failed

The service connection used for this task is a shared service connection using workload identity for the connection with ADO.

Environment type (Please select at least one enviroment where you face this issue)

  • Self-Hosted
  • Microsoft Hosted
  • VMSS Pool
  • Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Debian 12

Relevant log output

Generating script.
##[debug]System.TeamProjectId=id
##[debug]System.HostType=build
##[debug]System.PlanId=4a07c83d-742d-499e-a457-21c0a5727ee0
##[debug]System.JobId=108bf56a-fa83-5421-f47c-16e1929a4329
##[debug]System.CollectionUri=https://dev.azure.com/project/
##[debug]Getting credentials for local feeds
##[debug]SYSTEMVSSCONNECTION exists true
##[debug]Got auth token
##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 689.00 MB out of 3377.00 MB, CPU: Usage 10.50%
##[debug]Retrying OIDC token fetch. Retries left: 3
##[debug]Retrying OIDC token fetch. Retries left: 2
##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 689.00 MB out of 3377.00 MB, CPU: Usage 9.56%
##[debug]Retrying OIDC token fetch. Retries left: 1
##[debug]task result: Failed
##[error]run() failed

Full task logs with system.debug enabled

2024-04-10T08:32:48.5973498Z ##[debug]Evaluating condition for step: '[Decorator] ...'
2024-04-10T08:32:48.5974311Z ##[debug]Evaluating: SucceededNode()
2024-04-10T08:32:48.5974547Z ##[debug]Evaluating SucceededNode:
2024-04-10T08:32:48.5974917Z ##[debug]=> True
2024-04-10T08:32:48.5975158Z ##[debug]Result: True
2024-04-10T08:32:48.5975437Z ##[section]Starting: [Decorator] ...
2024-04-10T08:32:48.5982230Z ==============================================================================
2024-04-10T08:32:48.5982361Z Task         : Azure PowerShell
2024-04-10T08:32:48.5982429Z Description  : Run a PowerShell script within an Azure environment
2024-04-10T08:32:48.5982701Z Version      : 5.237.9
2024-04-10T08:32:48.5982763Z Author       : Microsoft Corporation
2024-04-10T08:32:48.5982836Z Help         : https://aka.ms/azurepowershelltroubleshooting
2024-04-10T08:32:48.5982925Z ==============================================================================
2024-04-10T08:32:49.5514788Z ##[debug]Invoking Method: System.Threading.Tasks.Task b__9(). Attempt count: 0
2024-04-10T08:32:50.5332951Z ##[debug]Using node path: /agent/externals/node20_1/bin/node
2024-04-10T08:32:50.6457623Z ##[debug]agent.TempDirectory=/agent/_work/_temp
2024-04-10T08:32:50.6493710Z ##[debug]loading inputs and endpoints
2024-04-10T08:32:50.6530269Z ##[debug]loading INPUT_CONNECTEDSERVICENAMEARM
2024-04-10T08:32:50.6551649Z ##[debug]loading INPUT_SCRIPTTYPE
2024-04-10T08:32:50.6569042Z ##[debug]loading INPUT_SCRIPTPATH
2024-04-10T08:32:50.6575030Z ##[debug]loading INPUT_INLINE
2024-04-10T08:32:50.6593270Z ##[debug]loading INPUT_ERRORACTIONPREFERENCE
2024-04-10T08:32:50.6601031Z ##[debug]loading INPUT_FAILONSTANDARDERROR
2024-04-10T08:32:50.6607045Z ##[debug]loading INPUT_TARGETAZUREPS
2024-04-10T08:32:50.6614014Z ##[debug]loading INPUT_PWSH
2024-04-10T08:32:50.6620616Z ##[debug]loading INPUT_VALIDATESCRIPTSIGNATURE
2024-04-10T08:32:50.6627915Z ##[debug]loading INPUT_WORKINGDIRECTORY
2024-04-10T08:32:50.6648348Z ##[debug]loading ENDPOINT_AUTH_appId
2024-04-10T08:32:50.6655365Z ##[debug]loading ENDPOINT_AUTH_SCHEME_appId
2024-04-10T08:32:50.6662959Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_appId_SERVICEPRINCIPALID
2024-04-10T08:32:50.6669085Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_appId_TENANTID
2024-04-10T08:32:50.6689194Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_appId_WORKLOADIDENTITYFEDERATIONSUBJECT
2024-04-10T08:32:50.6695660Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_appId_WORKLOADIDENTITYFEDERATIONISSUER
2024-04-10T08:32:50.6704232Z ##[debug]loading ENDPOINT_AUTH_SYSTEMVSSCONNECTION
2024-04-10T08:32:50.6710362Z ##[debug]loading ENDPOINT_AUTH_SCHEME_SYSTEMVSSCONNECTION
2024-04-10T08:32:50.6716071Z ##[debug]loading ENDPOINT_AUTH_PARAMETER_SYSTEMVSSCONNECTION_ACCESSTOKEN
2024-04-10T08:32:50.6746862Z ##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
2024-04-10T08:32:50.6753923Z ##[debug]loading SECRET_AZURE_APP_SERVICE_KUDU_PRODUCTION_PASSWORD
2024-04-10T08:32:50.6760327Z ##[debug]loaded 21
2024-04-10T08:32:50.6768525Z ##[debug]Agent.ProxyUrl=undefined
2024-04-10T08:32:50.6773610Z ##[debug]Agent.CAInfo=undefined
2024-04-10T08:32:50.6777927Z ##[debug]Agent.ClientCert=undefined
2024-04-10T08:32:50.6782724Z ##[debug]Agent.SkipCertValidation=undefined
2024-04-10T08:32:50.7185333Z ##[debug]agent.proxyurl=undefined
2024-04-10T08:32:50.7199801Z ##[debug]VSTS_ARM_REST_IGNORE_SSL_ERRORS=undefined
2024-04-10T08:32:50.7214144Z ##[debug]AZURE_HTTP_USER_AGENT=VSTS_89ffc636-9661-4fe1-9571-128fdd2c89a4_build_274_0
2024-04-10T08:32:50.9968285Z ##[debug]check path : /agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-04-10T08:32:50.9974392Z ##[debug]adding resource file: /agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-04-10T08:32:50.9977093Z ##[debug]system.culture=en-US
2024-04-10T08:32:50.9991300Z ##[debug]Agent.TempDirectory=/agent/_work/_temp
2024-04-10T08:32:51.0014166Z ##[debug]Resource file has already set to: /agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/node_modules/azure-pipelines-tasks-azure-arm-rest/module.json
2024-04-10T08:32:51.0030793Z ##[debug]workingDirectory=/agent/_work/1/s
2024-04-10T08:32:51.0033788Z ##[debug]check path : /agent/_work/1/s
2024-04-10T08:32:51.0047082Z ##[debug]agent.tempDirectory=/agent/_work/_temp
2024-04-10T08:32:51.0049214Z ##[debug]check path : /agent/_work/_temp
2024-04-10T08:32:51.0052244Z ##[debug]check path : /agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/task.json
2024-04-10T08:32:51.0054656Z ##[debug]adding resource file: /agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/task.json
2024-04-10T08:32:51.0061478Z ##[debug]system.culture=en-US
2024-04-10T08:32:51.0070822Z ##[debug]errorActionPreference=stop
2024-04-10T08:32:51.0082700Z ##[debug]ScriptType=InlineScript
2024-04-10T08:32:51.0086025Z ##[debug]ScriptPath=/agent/_work/1/s
2024-04-10T08:32:51.0103210Z ##[debug]Inline=scriptContent
2024-04-10T08:32:51.0106068Z ##[debug]ScriptArguments=undefined
2024-04-10T08:32:51.0109958Z ##[debug]FailOnStandardError=false
2024-04-10T08:32:51.0113442Z ##[debug]TargetAzurePs=LatestVersion
2024-04-10T08:32:51.0115376Z ##[debug]CustomTargetAzurePs=undefined
2024-04-10T08:32:51.0119363Z ##[debug]ConnectedServiceNameARM=appId
2024-04-10T08:32:51.0125650Z ##[debug]USE_MSAL=true
2024-04-10T08:32:51.0141352Z ##[debug]MSAL - USE_MSAL override is found: true
2024-04-10T08:32:51.0145675Z ##[debug]appId auth scheme = WorkloadIdentityFederation
2024-04-10T08:32:51.0147631Z ##[debug]Overriding useMSAL to true as workloadidentityfederation supports only MSAL
2024-04-10T08:32:51.0150420Z ##[debug]appId data subscriptionid = subId
2024-04-10T08:32:51.0152338Z ##[debug]appId data subscriptionname = subName
2024-04-10T08:32:51.0156411Z ##[debug]appId auth param serviceprincipalid = ***
2024-04-10T08:32:51.0158512Z ##[debug]appId data activeDirectoryAuthority = https://login.microsoftonline.com/
2024-04-10T08:32:51.0161638Z ##[debug]appId auth param tenantid = tenantId
2024-04-10T08:32:51.0164176Z ##[debug]appId=https://management.azure.com/
2024-04-10T08:32:51.0166048Z ##[debug]appId data environment = AzureCloud
2024-04-10T08:32:51.0169599Z ##[debug]appId auth scheme = WorkloadIdentityFederation
2024-04-10T08:32:51.0172154Z ##[debug]appId data msiclientId = undefined
2024-04-10T08:32:51.0174446Z ##[debug]appId data activeDirectoryServiceEndpointResourceId = https://management.core.windows.net/
2024-04-10T08:32:51.0176729Z ##[debug]appId data AzureKeyVaultServiceEndpointResourceId = https://vault.azure.net
2024-04-10T08:32:51.0179373Z ##[debug]appId data AzureKeyVaultDnsSuffix = vault.azure.net
2024-04-10T08:32:51.0181931Z ##[debug]appId data ScopeLevel = Subscription
2024-04-10T08:32:51.0192651Z ##[debug]MSAL - getEndpoint - useGraphActiveDirectoryResource=false
2024-04-10T08:32:51.0194400Z ##[debug]MSAL - getEndpoint - useMSAL=true
2024-04-10T08:32:51.0196110Z ##[debug]MSAL - getEndpoint - endpoint={"subscriptionID":"subId","subscriptionName":"subName","servicePrincipalClientID":"***","environmentAuthorityUrl":"https://login.microsoftonline.com/","tenantID":"tenantId","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"WorkloadIdentityFederation","activeDirectoryResourceID":"https://management.core.windows.net/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","scopeLevel":"Subscription"}
2024-04-10T08:32:51.0197829Z ##[debug]MSAL - getEndpoint - connectedServiceName=appId
2024-04-10T08:32:51.0198914Z ##[debug]appId auth param authenticationType = undefined
2024-04-10T08:32:51.0200130Z ##[debug]appId data EnableAdfsAuthentication = false
2024-04-10T08:32:51.0201258Z ##[debug]appId auth param apitoken = undefined
2024-04-10T08:32:51.0206293Z ##[debug]{"subscriptionID":"subId","subscriptionName":"subName","servicePrincipalClientID":"***","environmentAuthorityUrl":"https://login.microsoftonline.com/","tenantID":"tenantId","url":"https://management.azure.com/","environment":"AzureCloud","scheme":"WorkloadIdentityFederation","activeDirectoryResourceID":"https://management.azure.com/","azureKeyVaultServiceEndpointResourceId":"https://vault.azure.net","azureKeyVaultDnsSuffix":"vault.azure.net","scopeLevel":"Subscription","isADFSEnabled":false,"applicationTokenCredentials":{"connectedServiceName":"appId","clientId":"***","tenantId":"tenantId","baseUrl":"https://management.azure.com/","authorityUrl":"https://login.microsoftonline.com/","activeDirectoryResourceId":"https://management.azure.com/","isAzureStackEnvironment":false,"scheme":2,"isADFSEnabled":false,"useMSAL":true,"tokenMutex":{"_semaphore":{"_value":1,"_cancelError":{},"_weightedQueues":[],"_weightedWaiters":[]}}}}
2024-04-10T08:32:51.0213990Z Generating script.
2024-04-10T08:32:51.0228392Z ##[debug]System.TeamProjectId=91a91a79-7848-44b3-b477-2fa93928ecd4
2024-04-10T08:32:51.0234198Z ##[debug]System.HostType=build
2024-04-10T08:32:51.0251406Z ##[debug]System.PlanId=4a07c83d-742d-499e-a457-21c0a5727ee0
2024-04-10T08:32:51.0253557Z ##[debug]System.JobId=108bf56a-fa83-5421-f47c-16e1929a4329
2024-04-10T08:32:51.0255538Z ##[debug]System.CollectionUri=https://dev.azure.com/projectName/
2024-04-10T08:32:51.0257489Z ##[debug]Getting credentials for local feeds
2024-04-10T08:32:51.0262143Z ##[debug]SYSTEMVSSCONNECTION exists true
2024-04-10T08:32:51.0264109Z ##[debug]Got auth token
2024-04-10T08:32:52.6353402Z ##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 689.00 MB out of 3377.00 MB, CPU: Usage 10.50%
2024-04-10T08:32:54.0277530Z ##[debug]Retrying OIDC token fetch. Retries left: 3
2024-04-10T08:32:57.1474632Z ##[debug]Retrying OIDC token fetch. Retries left: 2
2024-04-10T08:32:57.6387469Z ##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 689.00 MB out of 3377.00 MB, CPU: Usage 9.56%
2024-04-10T08:32:59.2947957Z ##[debug]Retrying OIDC token fetch. Retries left: 1
2024-04-10T08:32:59.2958118Z ##[debug]task result: Failed
2024-04-10T08:32:59.2993358Z ##[error]run() failed
2024-04-10T08:32:59.3001258Z ##[debug]Processed: ##vso[task.issue type=error;source=TaskInternal;]run() failed
2024-04-10T08:32:59.3029216Z ##[debug]Processed: ##vso[task.complete result=Failed;]run() failed
2024-04-10T08:32:59.3029731Z ##[debug]which 'pwsh'
2024-04-10T08:32:59.3030052Z ##[debug]found: '/usr/bin/pwsh'
2024-04-10T08:32:59.3030389Z ##[debug]which '/usr/bin/pwsh'
2024-04-10T08:32:59.3030704Z ##[debug]found: '/usr/bin/pwsh'
2024-04-10T08:32:59.3031015Z ##[debug]/usr/bin/pwsh arg: -NoLogo
2024-04-10T08:32:59.3031337Z ##[debug]/usr/bin/pwsh arg: -NoProfile
2024-04-10T08:32:59.3031671Z ##[debug]/usr/bin/pwsh arg: -NonInteractive
2024-04-10T08:32:59.3032005Z ##[debug]/usr/bin/pwsh arg: -ExecutionPolicy
2024-04-10T08:32:59.3032619Z ##[debug]/usr/bin/pwsh arg: Unrestricted
2024-04-10T08:32:59.3032951Z ##[debug]/usr/bin/pwsh arg: -Command
2024-04-10T08:32:59.3033373Z ##[debug]/usr/bin/pwsh arg: . '/agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/RemoveAzContext.ps1'
2024-04-10T08:32:59.3033782Z ##[debug]exec tool: /usr/bin/pwsh
2024-04-10T08:32:59.3034102Z ##[debug]arguments:
2024-04-10T08:32:59.3034405Z ##[debug]   -NoLogo
2024-04-10T08:32:59.3034692Z ##[debug]   -NoProfile
2024-04-10T08:32:59.3034989Z ##[debug]   -NonInteractive
2024-04-10T08:32:59.3035295Z ##[debug]   -ExecutionPolicy
2024-04-10T08:32:59.3035595Z ##[debug]   Unrestricted
2024-04-10T08:32:59.3035890Z ##[debug]   -Command
2024-04-10T08:32:59.3036271Z ##[debug]   . '/agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/RemoveAzContext.ps1'
2024-04-10T08:32:59.3037126Z [command]/usr/bin/pwsh -NoLogo -NoProfile -NonInteractive -ExecutionPolicy Unrestricted -Command . '/agent/_work/_tasks/AzurePowerShell_72a1931b-effb-4d2e-8fd8-f8472a07cb62/5.237.9/RemoveAzContext.ps1'
2024-04-10T08:33:00.7495765Z ##[command]Disconnect-AzAccount -Scope CurrentUser -ErrorAction Stop
2024-04-10T08:33:00.8253864Z ##[command]Disconnect-AzAccount -Scope Process -ErrorAction Stop
2024-04-10T08:33:00.8315812Z ##[command]Clear-AzContext -Scope Process -ErrorAction Stop
2024-04-10T08:33:01.8778208Z 
2024-04-10T08:33:01.8822575Z ##[debug]Exit code 0 received from tool '/usr/bin/pwsh'
2024-04-10T08:33:01.8823193Z ##[debug]STDIO streams have closed for tool '/usr/bin/pwsh'
2024-04-10T08:33:01.8917266Z ##[warning]RetryHelper encountered task failure, will retry (attempt #: 1 out of 3) after 1000 ms
2024-04-10T08:33:02.6440550Z ##[debug]Agent environment resources - Disk: / Available 53241.00 MB out of 64201.00 MB, Memory: Used 656.00 MB out of 3377.00 MB, CPU: Usage 8.87%
2024-04-10T08:33:02.8920145Z ##[debug]PERF: RetryHelper Method:System.Threading.Tasks.Task b__9() : took 13340.4725 ms
2024-04-10T08:33:02.8920889Z ##[debug]PERF WARNING: RetryHelper Method:System.Threading.Tasks.Task b__9() : took 13340.4725 ms
... retries 2 and  3
2024-04-10T08:33:45.4103653Z ##[section]Finishing: [Decorator] ...

Repro steps

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants