Promptflow worker terminated due to signal 9

[jwk-qbllp]

We are having some strange behavior with a promptflow that is not completing due to a WORKER TIMEOUT. This flow has a handful of LLM tool steps that do not take longer than 180 seconds to run, but we have a node that always triggers a termination signal after 60 seconds. This happens after multiple successful LLM node runs prior to this step. This node always sends a signal 9 termination code after 60 seconds.

GenerateClaimsResponse Runs fine and no timeouts.
"system_metrics":{"completion_tokens":1838,"duration":139.077701,"prompt_tokens":4677,"total_tokens":6515},

Node ClaimsComparison this node always failed in a deployed endpoint but runs fine as a debug run.
"system_metrics":{"completion_tokens":1808, "duration":125.590981"prompt_tokens":4586,"total_tokens":6394}
This is the node that gets the termination signal after 60 seconds.

Here are the logs:
WARNING ClaimsComparison in line 0 has been running for 60 seconds
…
[CRITICAL] WORKER TIMEOUT (pid:25)
[WARNING] Worker with pid 25 was terminated due to signal 9

We have tried many sizes of compute nodes, and it does not appear to affect this timeout behavior.
Any ideas on this?

[brynn-code]

@thy09 Could you please help take a look on this?

[thy09]

@jwk-qbllp Can you provide more logs? The log "has been running for 60 seconds" is just a warning in a monitor thread, it doesn't affect the running of the node. As for the signal 9 problem, I have no idea why is it happen if we don't have more context.

[jwk-qbllp]

@thy09 Here are the complete logs:
Is there anything that stands out?

Instance status:
SystemSetup: Succeeded
UserContainerImagePull: Succeeded
ModelDownload: Succeeded
UserContainerStart: Succeeded

Container events:
Kind: Pod, Name: Downloading, Type: Normal, Time: 2024-03-15T20:33:24.543508Z, Message: Start downloading models
Kind: Pod, Name: Pulling, Type: Normal, Time: 2024-03-15T20:33:24.775201Z, Message: Start pulling container image
Kind: Pod, Name: Pulled, Type: Normal, Time: 2024-03-15T20:34:35.441587Z, Message: Container image is pulled successfully
Kind: Pod, Name: Downloaded, Type: Normal, Time: 2024-03-15T20:34:35.441587Z, Message: Models are downloaded successfully
Kind: Pod, Name: Created, Type: Normal, Time: 2024-03-15T20:34:35.461502Z, Message: Created container inference-server
Kind: Pod, Name: Started, Type: Normal, Time: 2024-03-15T20:34:35.488597Z, Message: Started container inference-server
Kind: Pod, Name: ContainerReady, Type: Normal, Time: 2024-03-15T20:34:51.147028231Z, Message: Container is ready

Container logs:
PROMPTFLOW_WORKER_NUM not set, setting PROMPTFLOW_WORKER_NUM to cpu core numner: 16
2024-03-15T20:34:35,492577471+00:00 - Starting runsvdir
build info: {"commit": "1a5d003ba26f039f0a2d8ca46de23e4255936681", "branch": "n/a", "build_number": "20240306.v5", "date": "2024-03-07 02:00:39"}
start promptflow runtime...
start promptflow serving with worker_num: 16, worker_threads: 1, app: promptflow._sdk._serving.app:create_app(extension_type='azureml')}
serving mode, skip invoking embedding store service
[2024-03-15 20:34:35 +0000] [16] [INFO] Starting gunicorn 20.1.0
[2024-03-15 20:34:35 +0000] [16] [INFO] Listening at: http://0.0.0.0:8080 (16)
[2024-03-15 20:34:35 +0000] [16] [INFO] Using worker: sync
[2024-03-15 20:34:35 +0000] [18] [INFO] Booting worker with pid: 18
[2024-03-15 20:34:35 +0000] [19] [INFO] Booting worker with pid: 19
[2024-03-15 20:34:35 +0000] [20] [INFO] Booting worker with pid: 20
[2024-03-15 20:34:35 +0000] [21] [INFO] Booting worker with pid: 21
[2024-03-15 20:34:35 +0000] [22] [INFO] Booting worker with pid: 22
[2024-03-15 20:34:35 +0000] [23] [INFO] Booting worker with pid: 23
[2024-03-15 20:34:35 +0000] [24] [INFO] Booting worker with pid: 24
[2024-03-15 20:34:35 +0000] [25] [INFO] Booting worker with pid: 25
[2024-03-15 20:34:35 +0000] [26] [INFO] Booting worker with pid: 26
[2024-03-15 20:34:36 +0000] [27] [INFO] Booting worker with pid: 27
[2024-03-15 20:34:36 +0000] [28] [INFO] Booting worker with pid: 28
[2024-03-15 20:34:36 +0000] [29] [INFO] Booting worker with pid: 29
[2024-03-15 20:34:36 +0000] [30] [INFO] Booting worker with pid: 30
[2024-03-15 20:34:36 +0000] [31] [INFO] Booting worker with pid: 31
[2024-03-15 20:34:36 +0000] [32] [INFO] Booting worker with pid: 32
[2024-03-15 20:34:36 +0000] [33] [INFO] Booting worker with pid: 33
[2024-03-15 20:34:36,817][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:36,878][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,041][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,046][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,120][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,125][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,127][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,255][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,277][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,294][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,364][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,428][pfserving-app][INFO] - Init mdc...
2024-03-15 20:34:37,558 | mdc | INFO | init data collector
2024-03-15 20:34:37,559 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:37,559 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:37,559 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:37,559 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,559 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,577 | mdc | INFO | data collector ready
[2024-03-15 20:34:37,577][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:37,577][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:37,617][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,679][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,723][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,723][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:37,723][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:37,724][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:37,724][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
2024-03-15 20:34:37,794 | mdc | INFO | init data collector
2024-03-15 20:34:37,794 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:37,794 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:37,794 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:37,795 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,795 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,795 | mdc | INFO | data collector ready
[2024-03-15 20:34:37,795][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:37,795][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:37,819][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:37,855][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,868][pfserving-app][INFO] - Enable 1 trace exporters.
2024-03-15 20:34:37,872 | mdc | INFO | init data collector
2024-03-15 20:34:37,873 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:37,873 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:37,873 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:37,873 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,873 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:37,883 | mdc | INFO | data collector ready
[2024-03-15 20:34:37,883][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:37,883][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:37,901][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:37,921][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,000][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,037][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,076][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,076][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,076][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,076][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
2024-03-15 20:34:38,087 | mdc | INFO | init data collector
2024-03-15 20:34:38,089 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,089 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,090 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,090 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,090 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
[2024-03-15 20:34:38,093][pfserving-app][INFO] - Init mdc...
2024-03-15 20:34:38,095 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,095][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,095][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,147][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,147][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,148][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,148][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,183][pfserving-app][INFO] - Enable AzureML extension.
2024-03-15 20:34:38,203 | mdc | INFO | init data collector
2024-03-15 20:34:38,204 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,204 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,204 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,204 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,204 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,236 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,236][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,237][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,252][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,319][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,320][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,320][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:38,331][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,332][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,332][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,332][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,379][pfserving-app][INFO] - Enable 1 trace exporters.
2024-03-15 20:34:38,390 | mdc | INFO | init data collector
2024-03-15 20:34:38,390 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,391 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,391 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,391 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,391 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,392 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,392][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,392][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,393][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,393][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,393][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,394][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,398][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:38,430][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,433][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,452][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
2024-03-15 20:34:38,464 | mdc | INFO | init data collector
2024-03-15 20:34:38,464 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,464 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,464 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,465 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,465 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,465 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,465][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,465][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,473][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:38,474][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,487][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,494][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,527][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,527][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,527][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,527][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,553][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,588][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:38,661][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,661][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,661][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:38,678 | mdc | INFO | init data collector
2024-03-15 20:34:38,679 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,679 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,679 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,679 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,679 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
[2024-03-15 20:34:38,690][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,691][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,691][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:38,696 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,696][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,696][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,723][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,723][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,724][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,724][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,732][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:34:38,745][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,746][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,746][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:38,775][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:38,777][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:38,793][pfserving-app][INFO] - Init mdc...
[2024-03-15 20:34:38,818][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,819][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,819][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:38,820 | mdc | INFO | init data collector
2024-03-15 20:34:38,821 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,821 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,821 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,821 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,821 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,831 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,831][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,831][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,835][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
2024-03-15 20:34:38,835 | mdc | INFO | init data collector
2024-03-15 20:34:38,836 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,836 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,836 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,836 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,836 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,841 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,841][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,841][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,861][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:38,875][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,876][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,876][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,876][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
2024-03-15 20:34:38,881 | mdc | INFO | init data collector
2024-03-15 20:34:38,881 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,881 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,881 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,882 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,882 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,882 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,882][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,882][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,950][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
2024-03-15 20:34:38,958 | mdc | INFO | init data collector
2024-03-15 20:34:38,958 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:38,958 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:38,959 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:38,959 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,959 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:38,959 | mdc | INFO | data collector ready
[2024-03-15 20:34:38,959][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:38,959][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:38,980][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:38,981][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:38,982][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:38,989][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:38,989][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:38,989][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:38,989][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:38,990][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,029][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,029][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,029][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,029][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,047][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,052][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,052][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,052][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,063][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,064][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,064][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,064][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,071][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,071][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,071][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,071][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,104][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,122][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,170][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,171][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
2024-03-15 20:34:39,186 | mdc | INFO | init data collector
2024-03-15 20:34:39,186 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:39,186 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:39,186 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:39,186 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,186 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,187 | mdc | INFO | data collector ready
[2024-03-15 20:34:39,187][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:39,187][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:39,191][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,204][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,249][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,249][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,249][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:39,256 | mdc | INFO | init data collector
2024-03-15 20:34:39,256 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:39,256 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:39,256 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:39,256 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,257 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,257 | mdc | INFO | data collector ready
[2024-03-15 20:34:39,257][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:39,257][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
2024-03-15 20:34:39,286 | mdc | INFO | init data collector
2024-03-15 20:34:39,286 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:39,286 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:39,287 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:39,287 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,287 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,287 | mdc | INFO | data collector ready
[2024-03-15 20:34:39,287][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:39,287][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:39,291][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,292][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,292][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,340][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,340][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,340][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,340][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,344][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,344][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,344][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,366][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,367][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,367][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,367][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,377][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,383][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,383][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,383][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:39,390 | mdc | INFO | init data collector
2024-03-15 20:34:39,390 | mdc | INFO | mdc enabled: True
2024-03-15 20:34:39,390 | mdc | INFO | mdc collections count 3
2024-03-15 20:34:39,390 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:34:39,390 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:34:39,391 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
[2024-03-15 20:34:39,392][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,392][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,392][pfserving-app][INFO] - Promptflow executor starts initializing...
2024-03-15 20:34:39,408 | mdc | INFO | data collector ready
[2024-03-15 20:34:39,408][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:34:39,408][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:34:39,426][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,426][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,426][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,426][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,434][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,463][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,475][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,487][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,498][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,534][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,538][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:34:39,538][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:34:39,538][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:34:39,538][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:34:39,568][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,674][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:34:39,675][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,675][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,675][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,682][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,682][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,682][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,743][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,743][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,743][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,788][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,836][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,851][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:34:39,851][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:39,852][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:39,883][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:39,986][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:34:42,181][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,181][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,197][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,197][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,252][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,252][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,278][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,278][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,309][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,309][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,316][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,316][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,402][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,402][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,408][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,408][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,408][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,424][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,425][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,425][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,496][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,496][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,496][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,504][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,504][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,504][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,533][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,534][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,534][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,540][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,540][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,540][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,584][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,584][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,606][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:42,607][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:42,625][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,626][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,626][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,809][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,809][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,809][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:42,832][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:42,833][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:42,833][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:43,100][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:43,100][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:43,324][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:43,324][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:43,324][pfserving-app][INFO] - Promptflow executor initializing succeed!
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
[2024-03-15 20:34:49,411][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:49,448][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:49,572][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:49,594][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:49,654][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:49,662][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 6 seconds...
[2024-03-15 20:34:50,163][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:50,163][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:50,387][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:50,387][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:34:50,387][flowinvoker][INFO] - Promptflow executor initiated successfully.
AppServiceCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
[2024-03-15 20:34:54,936][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:34:54,936][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:34:55,166][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:34:55,166][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:34:55,166][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:35:00,423][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 12 seconds...
[2024-03-15 20:35:00,423][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 12 seconds...
[2024-03-15 20:35:00,472][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 12 seconds...
[2024-03-15 20:35:00,475][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 12 seconds...
[2024-03-15 20:35:01,026][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:35:01,026][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:35:01,256][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:35:01,256][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:35:01,256][pfserving-app][INFO] - Promptflow executor initializing succeed!
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
AppServiceCredential.get_token failed: No token received.
ManagedIdentityCredential.get_token failed: No token received.
[2024-03-15 20:35:06,451][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:35:06,451][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:35:06,674][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:35:06,674][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:35:06,674][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:35:17,296][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 24 seconds...
[2024-03-15 20:35:17,299][promptflow._utils.retry_utils][WARNING] - <class 'Exception'>, Retrying in 24 seconds...
[2024-03-15 20:35:17,979][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:35:17,979][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:35:17,986][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:35:17,986][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:35:18,218][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:35:18,218][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:35:18,218][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:35:18,218][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:35:18,218][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:35:18,218][pfserving-app][INFO] - Promptflow executor initializing succeed!
[2024-03-15 20:36:35,295][pfserving-app][INFO] - Start monitoring new request, request_id: d0c15ccf-1aee-4a37-a28f-d420675e7f59, client_request_id: d0c15ccf-1aee-4a37-a28f-d420675e7f59
[2024-03-15 20:36:35,295][pfserving-app][INFO] - Start loading request data...
[2024-03-15 20:36:35,295][flowinvoker][INFO] - Validating flow input with data {'application_number': '18/303,073', 'rejection_date': '2023-11-09', 'rejection_code': 'CTNF', 'matter_id': '125141.04281', 'search_index_name': 'projection-index-office-action-simple', 'job_tracker_url': 'https://logic-pai-prd-wu-001.azurewebsites.net:443/api/wf-oar-jobtracker/triggers/Trigger/invoke?api-version=2022-05-01&sp=%2Ftriggers%2FTrigger%2Frun&sv=1.0&sig=UFmRAR7ew5PykhlwAQqTimIDA-zZKVoBKfxcjQ9YsWg', 'job_runid': 'd5054f8f-09a5-4248-965b-f72445a34282'}
[2024-03-15 20:36:35,295][flowinvoker][INFO] - Execute flow with data {'application_number': '18/303,073', 'rejection_date': '2023-11-09', 'rejection_code': 'CTNF', 'matter_id': '125141.04281', 'search_index_name': 'projection-index-office-action-simple', 'job_tracker_url': 'https://logic-pai-prd-wu-001.azurewebsites.net:443/api/wf-oar-jobtracker/triggers/Trigger/invoke?api-version=2022-05-01&sp=%2Ftriggers%2FTrigger%2Frun&sv=1.0&sig=UFmRAR7ew5PykhlwAQqTimIDA-zZKVoBKfxcjQ9YsWg', 'job_runid': 'd5054f8f-09a5-4248-965b-f72445a34282'}
2024-03-15 20:36:35 +0000 25 execution.flow INFO Start executing nodes in thread pool mode.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Start to run 29 nodes with concurrency level 16.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node StartJobRun. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_StartJobRun_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node RetrieveRejection. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_RetrieveRejection_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node RetrieveTemplate. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_RetrieveTemplate_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Node RetrieveTemplate completes.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node FormatTemplateDoc. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_FormatTemplateDoc_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Node FormatTemplateDoc completes.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Node RetrieveRejection completes.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node FormatRejectionDocs. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_FormatRejectionDocs_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Node FormatRejectionDocs completes.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Node StartJobRun completes.
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node RetrieveApplication. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_RetrieveApplication_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node RetrieveClaims. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_RetrieveClaims_0
2024-03-15 20:36:35 +0000 25 execution.flow INFO Executing node GetClientID. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_GetClientID_0
2024-03-15 20:36:36 +0000 25 execution.flow INFO Node RetrieveClaims completes.
2024-03-15 20:36:36 +0000 25 execution.flow INFO Node RetrieveApplication completes.
2024-03-15 20:36:36 +0000 25 execution.flow INFO Executing node FormatClaims. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_FormatClaims_0
2024-03-15 20:36:36 +0000 25 execution.flow INFO Executing node FormatAppDocs. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_FormatAppDocs_0
2024-03-15 20:36:36 +0000 25 execution.flow INFO Node FormatClaims completes.
2024-03-15 20:36:36 +0000 25 execution.flow INFO Node FormatAppDocs completes.
2024-03-15 20:36:36 +0000 25 execution.flow INFO Executing node GenerateBeginningJSON. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_GenerateBeginningJSON_0
2024-03-15 20:36:37 +0000 25 execution.flow INFO Node GetClientID completes.
2024-03-15 20:36:37 +0000 25 execution.flow INFO Executing node RetrieveClientReportingTemplate. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_RetrieveClientReportingTemplate_0
2024-03-15 20:36:37 +0000 25 execution.flow INFO Node RetrieveClientReportingTemplate completes.
2024-03-15 20:36:37 +0000 25 execution.flow INFO Executing node FormatClientReportingTemplate. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_FormatClientReportingTemplate_0
2024-03-15 20:36:37 +0000 25 execution.flow INFO Node FormatClientReportingTemplate completes.
2024-03-15 20:37:07 +0000 25 execution.flow INFO Node GenerateBeginningJSON completes.
2024-03-15 20:37:07 +0000 25 execution.flow INFO Executing node MakeSequential1. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_MakeSequential1_0
2024-03-15 20:37:07 +0000 25 execution.flow INFO Node MakeSequential1 completes.
2024-03-15 20:37:07 +0000 25 execution.flow INFO Executing node GenerateClaimsResponse. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_GenerateClaimsResponse_0
2024-03-15 20:38:07 +0000 25 execution WARNING GenerateClaimsResponse in line 0 has been running for 60 seconds, stacktrace of thread 139993131493120:
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/common.py", line 207, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/aoai.py", line 147, in chat
completion = self._client.chat.completions.create(**params)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/openai_injector.py", line 88, in wrapper
return f(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
output = func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_utils/_utils.py", line 275, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 663, in create
return self._post(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 1200, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 889, in request
return self._request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 918, in _request
response = self._client.send(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 1015, in _send_single_request
response = transport.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
return self._connection.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
) = self._receive_response_headers(**kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
event = self._receive_event(timeout=timeout)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
data = self._network_stream.read(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 126, in read
return self._sock.recv(max_bytes)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1227, in recv
return self.read(buflen)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1102, in read
return self._sslobj.read(len)

2024-03-15 20:39:07 +0000 25 execution WARNING GenerateClaimsResponse in line 0 has been running for 120 seconds, stacktrace of thread 139993131493120:
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/common.py", line 207, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/aoai.py", line 147, in chat
completion = self._client.chat.completions.create(**params)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/openai_injector.py", line 88, in wrapper
return f(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
output = func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_utils/_utils.py", line 275, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 663, in create
return self._post(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 1200, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 889, in request
return self._request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 918, in _request
response = self._client.send(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 1015, in _send_single_request
response = transport.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
return self._connection.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
) = self._receive_response_headers(**kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
event = self._receive_event(timeout=timeout)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
data = self._network_stream.read(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 126, in read
return self._sock.recv(max_bytes)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1227, in recv
return self.read(buflen)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1102, in read
return self._sslobj.read(len)

2024-03-15 20:39:36 +0000 25 execution.flow INFO Node GenerateClaimsResponse completes.
2024-03-15 20:39:36 +0000 25 execution.flow INFO Executing node AddWait1. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_AddWait1_0
2024-03-15 20:40:01 +0000 25 execution.flow INFO Node AddWait1 completes.
2024-03-15 20:40:01 +0000 25 execution.flow INFO Executing node ClaimsComparison. node run id: d0c15ccf-1aee-4a37-a28f-d420675e7f59_ClaimsComparison_0
2024-03-15 20:41:01 +0000 25 execution WARNING ClaimsComparison in line 0 has been running for 60 seconds, stacktrace of thread 139993131493120:
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/common.py", line 207, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/tools/aoai.py", line 147, in chat
completion = self._client.chat.completions.create(**params)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/openai_injector.py", line 88, in wrapper
return f(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/promptflow/_core/tracer.py", line 528, in wrapped
output = func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_utils/_utils.py", line 275, in wrapper
return func(*args, **kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/resources/chat/completions.py", line 663, in create
return self._post(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 1200, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 889, in request
return self._request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/openai/_base_client.py", line 918, in _request
response = self._client.send(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 914, in send
response = self._send_handling_auth(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 942, in _send_handling_auth
response = self._send_handling_redirects(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
response = self._send_single_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_client.py", line 1015, in _send_single_request
response = transport.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpx/_transports/default.py", line 233, in handle_request
resp = self._pool.handle_request(req)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection_pool.py", line 196, in handle_request
response = connection.handle_request(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/connection.py", line 101, in handle_request
return self._connection.handle_request(request)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 113, in handle_request
) = self._receive_response_headers(**kwargs)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 186, in _receive_response_headers
event = self._receive_event(timeout=timeout)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_sync/http11.py", line 224, in _receive_event
data = self._network_stream.read(
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/site-packages/httpcore/_backends/sync.py", line 126, in read
return self._sock.recv(max_bytes)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1227, in recv
return self.read(buflen)
File "/azureml-envs/prompt-flow/runtime/lib/python3.9/ssl.py", line 1102, in read
return self._sslobj.read(len)

[2024-03-15 20:41:35 +0000] [16] [CRITICAL] WORKER TIMEOUT (pid:25)
[2024-03-15 20:41:37 +0000] [16] [WARNING] Worker with pid 25 was terminated due to signal 9
[2024-03-15 20:41:37 +0000] [615] [INFO] Booting worker with pid: 615
[2024-03-15 20:41:37,914][pfserving-app][INFO] - Enable AzureML extension.
[2024-03-15 20:41:37,953][pfserving-app][INFO] - Init mdc...
2024-03-15 20:41:38,286 | mdc | INFO | init data collector
2024-03-15 20:41:38,286 | mdc | INFO | mdc enabled: True
2024-03-15 20:41:38,287 | mdc | INFO | mdc collections count 3
2024-03-15 20:41:38,287 | mdc | INFO | mdc collection app_traces enabled:True,sample_percentage:100
2024-03-15 20:41:38,287 | mdc | INFO | mdc collection model_inputs enabled:True,sample_percentage:100
2024-03-15 20:41:38,287 | mdc | INFO | mdc collection model_outputs enabled:True,sample_percentage:100
2024-03-15 20:41:38,287 | mdc | INFO | data collector ready
[2024-03-15 20:41:38,287][pfserving-app][INFO] - Mdc init status: True
[2024-03-15 20:41:38,287][pfserving-app][INFO] - Initialize metrics recorder with azure monitor metrics exporter...
[2024-03-15 20:41:38,357][pfserving-app][INFO] - OpenTelemetry metric is enabled, metrics will be recorded.
[2024-03-15 20:41:38,357][pfserving-app][INFO] - Init mdc for app_traces...
[2024-03-15 20:41:38,357][pfserving-app][INFO] - Mdc exporter init status: True
[2024-03-15 20:41:38,357][pfserving-app][INFO] - Project path: /var/azureml-app/azureml-models/endpoint-oar-1-6-arm01/5/model
[2024-03-15 20:41:38,432][pfserving-app][INFO] - Enable 1 trace exporters.
[2024-03-15 20:41:38,535][pfserving-app][INFO] - Promptflow sample file detected.
[2024-03-15 20:41:38,535][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:41:38,535][pfserving-app][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:41:38,607][flowinvoker][INFO] - Getting connections from pf client with provider azureml:/subscriptions/f058f18b-e29b-47c4-ac91-e31e67998f58/resourceGroups/rg-pai-prd-wu-001/providers/Microsoft.MachineLearningServices/workspaces/copilot-testing...
[2024-03-15 20:41:40,867][flowinvoker][INFO] - Promptflow get connections successfully. keys: dict_keys(['conn-srch-main', 'stpaiprdwu001', 'conn-oai-main', 'conn-client-data-api'])
[2024-03-15 20:41:40,867][flowinvoker][INFO] - Promptflow executor starts initializing...
[2024-03-15 20:41:41,098][flowinvoker][INFO] - Promptflow executor initiated successfully.
[2024-03-15 20:41:41,098][pfserving-app][INFO] - Response included fields: ["responseURL","reportingURL"]
[2024-03-15 20:41:41,098][pfserving-app][INFO] - Promptflow executor initializing succeed!

[wxpjimmy]

so far the maximum timeout of a single request is 5 min for MIR deployment, and seems your flow contains many nodes and runs longer than 5min, is this long request processing time expected in your scenario?

[jwk-qbllp]

Yes our run might take longer than 5 minutes. Do we have any options to increase this timeout?