AWS STS API - Expiration field misunderstood - Timestamps are formatted according to the ISO 8601 not unix timestamp

[MalibuKoKo]

A warning explain than Expiration value is missing or not an integer.
I think it's a misunderstood about Expiration field, we can see an example on API documentation.
Timestamps are formatted according to the ISO 8601 standard.

You can find the impacted code here, and could be patched quickly with the same logic we can found here

Patch

diff --git a/source/extensions/common/aws/credentials_provider_impl.cc b/source/extensions/common/aws/credentials_provider_impl.cc
index 139d53f177..6cec8e96cd 100644
--- a/source/extensions/common/aws/credentials_provider_impl.cc
+++ b/source/extensions/common/aws/credentials_provider_impl.cc
@@ -688,12 +688,15 @@ void WebIdentityCredentialsProvider::extractCredentials(
   setCredentialsToAllThreads(
       std::make_unique<Credentials>(access_key_id, secret_access_key, session_token));
 
-  const auto expiration = Utility::getIntegerFromJsonOrDefault(credentials.value(), EXPIRATION, 0);
+  const auto expiration =
+      Utility::getStringFromJsonOrDefault(credentials.value(), EXPIRATION, "");
 
-  if (expiration != 0) {
-    expiration_time_ =
-        std::chrono::time_point<std::chrono::system_clock>(std::chrono::seconds(expiration));
-    ENVOY_LOG(debug, "AWS STS credentials expiration time (unix timestamp): {}", expiration);
+  if (!expiration.empty()) {
+    absl::Time expiration_time;
+    if (absl::ParseTime(EXPIRATION_FORMAT, expiration, &expiration_time, nullptr)) {
+      ENVOY_LOG(debug, "Container role AWS credentials expiration time: {}", expiration);
+      expiration_time_ = absl::ToChronoTime(expiration_time);
+    }
   } else {
     expiration_time_ = api_.timeSource().systemTime() + REFRESH_INTERVAL;
     ENVOY_LOG(warn, "Could not get Expiration value of AWS credentials document from STS, so "

Repro steps:

• EKS 1.28
• ArgoCD
• AWS Controllers for Kubernetes (ACK) - IAM
• AWS Controllers for Kubernetes (ACK) - Lambda

Manifest

apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-3"
    eks.amazonaws.com/audience: sts.amazonaws.com
    eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/k8s-dev-app
    eks.amazonaws.com/sts-regional-endpoints: "true"
    eks.amazonaws.com/token-expiration: "43200"
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app
  namespace: default
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  annotations:
    argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
    argocd.argoproj.io/sync-wave: "-3"
  labels:
    app.kubernetes.io/part-of: ack
  name: envoy-invoke-lambda
spec:
  assumeRolePolicyDocument: |
    {
      "Statement": [
        {
          "Action": "sts:AssumeRole",
          "Condition": {
            "Bool": {
              "aws:MultiFactorAuthPresent": "true"
            }
          },
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::791324568:root"
          },
          "Sid": "root"
        },
        {
          "Action": "sts:AssumeRoleWithWebIdentity",
          "Condition": {
            "StringEquals": {
              "oidc.eks.eu-central-1.amazonaws.com/id/456789123abcdef:sub": "system:serviceaccount:default:app"
            }
          },
          "Effect": "Allow",
          "Principal": {
            "Federated": "arn:aws:iam::123456789:oidc-provider/oidc.eks.eu-central-1.amazonaws.com/id/456789123abcdef"
          },
          "Sid": "AssumeRoleWithWebIdentity"
        }
      ],
      "Version": "2012-10-17"
    }
  description: envoy-invoke-lambda
  maxSessionDuration: 43200
  name: k8s-dev-app
  permissionsBoundary: arn:aws:iam::123456789:policy/ack/boundary@dev
  policies:
    - arn:aws:iam::123456789:policy/k8s-dev-app
  tags:
    - key: AppName
      value: app
    - key: CostCenter
      value: IT
    - key: EnvironmentType
      value: DEV
    - key: ProjectName
      value: project
    - key: StackSource
      value: https://github.com/envoyproxy/envoy
    - key: TeamContact
      value: foo@bar.baz
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Role
metadata:
  annotations:
    argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
    argocd.argoproj.io/sync-wave: "-3"
  labels:
    app.kubernetes.io/part-of: ack
  name: hello-world
spec:
  assumeRolePolicyDocument: |
    {
      "Statement": [
        {
          "Action": "sts:AssumeRole",
          "Condition": {
            "Bool": {
              "aws:MultiFactorAuthPresent": "true"
            }
          },
          "Effect": "Allow",
          "Principal": {
            "AWS": "arn:aws:iam::791324568:root"
          },
          "Sid": "root"
        },
        {
          "Action": "sts:AssumeRole",
          "Effect": "Allow",
          "Principal": {
            "Service": "lambda.amazonaws.com"
          }
        }
      ],
      "Version": "2012-10-17"
    }
  description: hello-world
  maxSessionDuration: 3600
  name: k8s-hello-world
  permissionsBoundary: arn:aws:iam::123456789:policy/ack/boundary@dev
  policies:
    - arn:aws:iam::123456789:policy/k8s-hello-world
  tags:
    - key: AppName
      value: app
    - key: CostCenter
      value: IT
    - key: EnvironmentType
      value: DEV
    - key: ProjectName
      value: project
    - key: StackSource
      value: https://github.com/envoyproxy/envoy
    - key: TeamContact
      value: foo@bar.baz
---
apiVersion: v1
data:
  envoy.yaml: |-
    admin:
      access_log_path: /dev/stdout
      address:
        socket_address:
          address: 0.0.0.0
          port_value: 9901
    layered_runtime:
      layers:
      - name: static_layer
        static_layer:
          envoy.reloadable_features.use_http_client_to_fetch_aws_credentials: true
    static_resources:
      listeners:
      - name: envoy-http-listener
        address:
          socket_address:
            address: 0.0.0.0
            port_value: 8080
            protocol: TCP
        filter_chains:
        - name: httphost-shared
          filter_chain_match: {}
          filters:
          - name: envoy.filters.network.http_connection_manager
            typed_config:
              '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              stat_prefix: ingress_http
              access_log:
                - name: envoy.access_loggers.file
                  typed_config:
                    '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                    path: /dev/fd/1
                    log_format:
                      text_format_source:
                        inline_string: |
                          ACCESS [%START_TIME%] "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%UPSTREAM_HOST%"
              http_filters:
              - name: envoy.filters.http.aws_lambda
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.aws_lambda.v3.Config
                  arn: "arn:aws:lambda:eu-central-1:*:function:*"
                  payload_passthrough: false
              - name: envoy.filters.http.router
                typed_config:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
                  suppress_envoy_headers: true
              normalize_path: true
              use_remote_address: true
              server_name: envoy
              http_protocol_options:
                accept_http_10: false
              preserve_external_request_id: false
              route_config:
                name: local_route
                virtual_hosts:
                - name: lambda
                  domains: ["*"]
                  routes:
                  - match:
                      prefix: /hello-world/
                    route:
                      cluster: lambda
                    typed_per_filter_config:
                      envoy.filters.http.aws_lambda:
                        '@type': type.googleapis.com/envoy.extensions.filters.http.aws_lambda.v3.PerRouteConfig
                        invoke_config:
                          arn: "arn:aws:lambda:eu-central-1:123456789:function:k8s-hello-world"
                          payload_passthrough: false
        traffic_direction: UNSPECIFIED
      clusters:
      - name: lambda
        connect_timeout: 3s
        type: LOGICAL_DNS
        dns_lookup_family: V4_ONLY
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: lambda
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address: { address: lambda.eu-central-1.amazonaws.com, port_value: 443, protocol: TCP }
        metadata:
          filter_metadata:
            com.amazonaws.lambda:
              egress_gateway: true
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            sni: "*.amazonaws.com"
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app
---
apiVersion: v1
data: null
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app-xds
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app
spec:
  ports:
    - name: http
      port: 80
      protocol: TCP
      targetPort: http
  selector:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/name: envoy
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "0"
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: envoy
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: envoy
      app.kubernetes.io/version: 1.30.1
      helm.sh/chart: envoy-1.10.0
  strategy:
    rollingUpdate:
      maxSurge: 2
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "-3"
        checksum/config: a61c9bb99ad3b4688ee6510fab20bf88f3166192adac4105ec3ad63b9ad6a73a
        eks.amazonaws.com/audience: sts.amazonaws.com
        eks.amazonaws.com/role-arn: arn:aws:iam::123456789:role/k8s-dev-app
        eks.amazonaws.com/sts-regional-endpoints: "true"
        eks.amazonaws.com/token-expiration: "43200"
      labels:
        app.kubernetes.io/instance: envoy
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: envoy
        app.kubernetes.io/version: 1.30.1
        helm.sh/chart: envoy-1.10.0
    spec:
      containers:
        - args:
            - --log-level
            - info
            - --config-path
            - /config/envoy.yaml
          command:
            - /docker-entrypoint.sh
          env: null
          image: envoyproxy/envoy:v1.30.1
          imagePullPolicy: IfNotPresent
          lifecycle: {}
          livenessProbe:
            initialDelaySeconds: 30
            tcpSocket:
              port: admin
          name: envoy
          ports:
            - containerPort: 9901
              name: admin
              protocol: TCP
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            initialDelaySeconds: 30
            tcpSocket:
              port: admin
          resources:
            limits:
              cpu: 500m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 256Mi
          volumeMounts:
            - mountPath: /config
              name: config
      initContainers:
        - command:
            - sh
            - -c
            - |
              /bin/sh <<'EOF'
                set -e
                export $(printf "AWS_ACCESS_KEY_ID=%s AWS_SECRET_ACCESS_KEY=%s AWS_SESSION_TOKEN=%s" \
                $(aws sts assume-role-with-web-identity \
                --role-arn $AWS_ROLE_ARN \
                --role-session-name ${POD_NAME} \
                --web-identity-token file://$AWS_WEB_IDENTITY_TOKEN_FILE \
                --duration-seconds 900 \
                --query "Credentials.[AccessKeyId,SecretAccessKey,SessionToken]" \
                --output text))
                aws lambda invoke --region eu-central-1 --function-name k8s-hello-world /dev/stdout
              EOF
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: public.ecr.aws/aws-cli/aws-cli:2.15.41
          imagePullPolicy: IfNotPresent
          name: aws-cli
      securityContext: {}
      serviceAccountName: app
      terminationGracePeriodSeconds: 30
      topologySpreadConstraints:
        - labelSelector:
            matchLabels:
              app.kubernetes.io/instance: envoy
          maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
        - labelSelector:
            matchLabels:
              app.kubernetes.io/instance: envoy
          maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
      volumes:
        - configMap:
            name: app
          name: config
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  labels:
    app.kubernetes.io/instance: envoy
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: envoy
    app.kubernetes.io/version: 1.30.1
    helm.sh/chart: envoy-1.10.0
  name: app
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/instance: envoy
      app.kubernetes.io/managed-by: Helm
      app.kubernetes.io/name: envoy
      app.kubernetes.io/version: 1.30.1
      helm.sh/chart: envoy-1.10.0
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  annotations:
    argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
    argocd.argoproj.io/sync-wave: "-3"
  labels:
    app.kubernetes.io/part-of: ack
  name: envoy-invoke-lambda
spec:
  description: envoy-invoke-lambda
  name: k8s-dev-app
  path: /
  policyDocument: |
    {
      "Statement": [
        {
          "Action": [
            "lambda:InvokeFunction"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:lambda:eu-central-1:123456789:function:k8s-hello-world"
          ],
          "Sid": "invoke"
        }
      ],
      "Version": "2012-10-17"
    }
  tags:
    - key: AppName
      value: app
    - key: CostCenter
      value: IT
    - key: EnvironmentType
      value: DEV
    - key: ProjectName
      value: project
    - key: StackSource
      value: https://github.com/envoyproxy/envoy
    - key: TeamContact
      value: foo@bar.baz
---
apiVersion: iam.services.k8s.aws/v1alpha1
kind: Policy
metadata:
  annotations:
    argocd.argoproj.io/sync-options: PrunePropagationPolicy=background
    argocd.argoproj.io/sync-wave: "-3"
  labels:
    app.kubernetes.io/part-of: ack
  name: hello-world
spec:
  description: hello-world
  name: k8s-hello-world
  path: /
  policyDocument: |
    {
      "Statement": [
        {
          "Action": "logs:CreateLogGroup",
          "Effect": "Allow",
          "Resource": "arn:aws:logs:eu-central-1:123456789:*"
        },
        {
          "Action": [
            "logs:CreateLogStream",
            "logs:PutLogEvents"
          ],
          "Effect": "Allow",
          "Resource": [
            "arn:aws:logs:eu-central-1:123456789:log-group:/aws/lambda/k8s-hello-world:*"
          ]
        }
      ],
      "Version": "2012-10-17"
    }
  tags:
    - key: AppName
      value: app
    - key: CostCenter
      value: IT
    - key: EnvironmentType
      value: DEV
    - key: ProjectName
      value: project
    - key: StackSource
      value: https://github.com/envoyproxy/envoy
    - key: TeamContact
      value: foo@bar.baz
---
apiVersion: lambda.services.k8s.aws/v1alpha1
kind: Function
metadata:
  annotations:
    argocd.argoproj.io/sync-options: PruneLast=false,PrunePropagationPolicy=background
    argocd.argoproj.io/sync-wave: "-3"
    services.k8s.aws/region: eu-central-1
  name: hello-world
spec:
  architectures:
    - x86_64
  code:
    imageURI: 123456789.dkr.ecr.eu-central-1.amazonaws.com/lambda-hello-world:202311022305
  description: function created by ACK lambda-controller
  environment:
    variables:
      LOGGING_LEVEL: INFO
  ephemeralStorage:
    size: 512
  memorySize: 128
  name: k8s-hello-world
  packageType: Image
  publish: true
  role: arn:aws:iam::123456789:role/k8s-hello-world
  tags:
    - key: AppName
      value: app
    - key: CostCenter
      value: IT
    - key: EnvironmentType
      value: DEV
    - key: ProjectName
      value: project
    - key: StackSource
      value: https://github.com/envoyproxy/envoy
    - key: TeamContact
      value: foo@bar.baz
  timeout: 300

Logs:

[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:428] initializing epoch 0 (base id=0, hot restart version=11.104)
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:430] statically linked extensions:
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.connection_handler: envoy.connection_handler.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.resolvers: envoy.ip
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   filter_state.object: envoy.filters.listener.original_dst.local_ip, envoy.filters.listener.original_dst.remote_ip, envoy.network.application_protocols, envoy.network.transport_socket.original_dst_address, envoy.network.upstream_server_name, envoy.network.upstream_subject_alt_names, envoy.string, envoy.tcp_proxy.cluster, envoy.tcp_proxy.disable_tunneling, envoy.tcp_proxy.per_connection_idle_timeout_ms, envoy.upstream.dynamic_host, envoy.upstream.dynamic_port
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.action: envoy.matching.actions.format_string, filter-chain-name
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.common_inputs: envoy.matching.common_inputs.environment_variable
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.network.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.retry_host_predicates: envoy.retry_host_predicates.omit_canary_hosts, envoy.retry_host_predicates.omit_host_metadata, envoy.retry_host_predicates.previous_hosts
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.compression.decompressor: envoy.compression.brotli.decompressor, envoy.compression.gzip.decompressor, envoy.compression.zstd.decompressor
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.original_ip_detection: envoy.http.original_ip_detection.custom_header, envoy.http.original_ip_detection.xff
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.custom_response: envoy.extensions.http.custom_response.local_response_policy, envoy.extensions.http.custom_response.redirect_policy
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.input_matchers: envoy.matching.matchers.cel_matcher, envoy.matching.matchers.consistent_hashing, envoy.matching.matchers.ip, envoy.matching.matchers.runtime_fraction
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.header_validators: envoy.http.header_validators.envoy_default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.injected_credentials: envoy.http.injected_credentials.generic
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.network.dns_resolver: envoy.network.dns_resolver.cares, envoy.network.dns_resolver.getaddrinfo
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.quic.connection_id_generator: envoy.quic.deterministic_connection_id_generator
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.early_header_mutation: envoy.http.early_header_mutation.header_mutation
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.access_loggers: envoy.access_loggers.file, envoy.access_loggers.fluentd, envoy.access_loggers.http_grpc, envoy.access_loggers.open_telemetry, envoy.access_loggers.stderr, envoy.access_loggers.stdout, envoy.access_loggers.tcp_grpc, envoy.access_loggers.wasm, envoy.file_access_log, envoy.fluentd_access_log, envoy.http_grpc_access_log, envoy.open_telemetry_access_log, envoy.stderr_access_log, envoy.stdout_access_log, envoy.tcp_grpc_access_log, envoy.wasm_access_log
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.compression.compressor: envoy.compression.brotli.compressor, envoy.compression.gzip.compressor, envoy.compression.zstd.compressor
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.filters: envoy.filters.thrift.header_to_metadata, envoy.filters.thrift.payload_to_metadata, envoy.filters.thrift.rate_limit, envoy.filters.thrift.router
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   quic.http_server_connection: quic.http_server_connection.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.upstreams: envoy.filters.connection_pools.tcp.generic
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.filters.http: envoy.bandwidth_limit, envoy.buffer, envoy.cors, envoy.csrf, envoy.ext_authz, envoy.ext_proc, envoy.fault, envoy.filters.http.adaptive_concurrency, envoy.filters.http.admission_control, envoy.filters.http.alternate_protocols_cache, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.bandwidth_limit, envoy.filters.http.basic_auth, envoy.filters.http.buffer, envoy.filters.http.cache, envoy.filters.http.cdn_loop, envoy.filters.http.composite, envoy.filters.http.compressor, envoy.filters.http.connect_grpc_bridge, envoy.filters.http.cors, envoy.filters.http.credential_injector, envoy.filters.http.csrf, envoy.filters.http.custom_response, envoy.filters.http.decompressor, envoy.filters.http.dynamic_forward_proxy, envoy.filters.http.ext_authz, envoy.filters.http.ext_proc, envoy.filters.http.fault, envoy.filters.http.file_system_buffer, envoy.filters.http.gcp_authn, envoy.filters.http.geoip, envoy.filters.http.grpc_field_extraction, envoy.filters.http.grpc_http1_bridge, envoy.filters.http.grpc_http1_reverse_bridge, envoy.filters.http.grpc_json_transcoder, envoy.filters.http.grpc_stats, envoy.filters.http.grpc_web, envoy.filters.http.header_mutation, envoy.filters.http.header_to_metadata, envoy.filters.http.health_check, envoy.filters.http.ip_tagging, envoy.filters.http.json_to_metadata, envoy.filters.http.jwt_authn, envoy.filters.http.local_ratelimit, envoy.filters.http.lua, envoy.filters.http.match_delegate, envoy.filters.http.oauth2, envoy.filters.http.on_demand, envoy.filters.http.original_src, envoy.filters.http.rate_limit_quota, envoy.filters.http.ratelimit, envoy.filters.http.rbac, envoy.filters.http.router, envoy.filters.http.set_filter_state, envoy.filters.http.set_metadata, envoy.filters.http.stateful_session, envoy.filters.http.tap, envoy.filters.http.wasm, envoy.geoip, envoy.grpc_http1_bridge, envoy.grpc_json_transcoder, envoy.grpc_web, envoy.health_check, envoy.ip_tagging, envoy.local_rate_limit, envoy.lua, envoy.rate_limit, envoy.router
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.protocols: auto, binary, binary/non-strict, compact, twitter
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.transport_sockets.upstream: envoy.transport_sockets.alts, envoy.transport_sockets.http_11_proxy, envoy.transport_sockets.internal_upstream, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, envoy.transport_sockets.upstream_proxy_protocol, raw_buffer, starttls, tls
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.stateful_header_formatters: envoy.http.stateful_header_formatters.preserve_case, preserve_case
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.quic.proof_source: envoy.quic.proof_source.filter_chain
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.stats_sinks: envoy.dog_statsd, envoy.graphite_statsd, envoy.metrics_service, envoy.open_telemetry_stat_sink, envoy.stat_sinks.dog_statsd, envoy.stat_sinks.graphite_statsd, envoy.stat_sinks.hystrix, envoy.stat_sinks.metrics_service, envoy.stat_sinks.open_telemetry, envoy.stat_sinks.statsd, envoy.stat_sinks.wasm, envoy.statsd
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.tracers.opentelemetry.samplers: envoy.tracers.opentelemetry.samplers.always_on, envoy.tracers.opentelemetry.samplers.dynatrace
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.health_checkers: envoy.health_checkers.grpc, envoy.health_checkers.http, envoy.health_checkers.redis, envoy.health_checkers.tcp, envoy.health_checkers.thrift
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.protocols: dubbo
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.router.cluster_specifier_plugin: envoy.router.cluster_specifier_plugin.lua
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.route.early_data_policy: envoy.route.early_data_policy.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.access_loggers.extension_filters: envoy.access_loggers.extension_filters.cel
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.network.input: envoy.matching.inputs.application_protocol, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.filter_state, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.subject, envoy.matching.inputs.transport_protocol, envoy.matching.inputs.uri_san
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.guarddog_actions: envoy.watchdog.abort_action, envoy.watchdog.profile_action
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.transport_sockets.downstream: envoy.transport_sockets.alts, envoy.transport_sockets.quic, envoy.transport_sockets.raw_buffer, envoy.transport_sockets.starttls, envoy.transport_sockets.tap, envoy.transport_sockets.tcp_stats, envoy.transport_sockets.tls, raw_buffer, starttls, tls
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.filters.network: envoy.echo, envoy.ext_authz, envoy.filters.network.connection_limit, envoy.filters.network.direct_response, envoy.filters.network.dubbo_proxy, envoy.filters.network.echo, envoy.filters.network.ext_authz, envoy.filters.network.http_connection_manager, envoy.filters.network.local_ratelimit, envoy.filters.network.mongo_proxy, envoy.filters.network.ratelimit, envoy.filters.network.rbac, envoy.filters.network.redis_proxy, envoy.filters.network.set_filter_state, envoy.filters.network.sni_cluster, envoy.filters.network.sni_dynamic_forward_proxy, envoy.filters.network.tcp_proxy, envoy.filters.network.thrift_proxy, envoy.filters.network.wasm, envoy.filters.network.zookeeper_proxy, envoy.http_connection_manager, envoy.mongo_proxy, envoy.ratelimit, envoy.redis_proxy, envoy.tcp_proxy
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.load_balancing_policies: envoy.load_balancing_policies.cluster_provided, envoy.load_balancing_policies.least_request, envoy.load_balancing_policies.maglev, envoy.load_balancing_policies.random, envoy.load_balancing_policies.ring_hash, envoy.load_balancing_policies.round_robin, envoy.load_balancing_policies.subset
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.upstream.local_address_selector: envoy.upstream.local_address_selector.default_local_address_selector
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.udp_packet_writer: envoy.udp_packet_writer.default, envoy.udp_packet_writer.gso
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.route_config_update_requester: envoy.route_config_update_requester.default
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.retry_priorities: envoy.retry_priorities.previous_priorities
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.upstream_options: envoy.extensions.upstreams.http.v3.HttpProtocolOptions, envoy.extensions.upstreams.tcp.v3.TcpProtocolOptions, envoy.upstreams.http.http_protocol_options, envoy.upstreams.tcp.tcp_protocol_options
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   network.connection.client: default, envoy_internal
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.matching.http.input: envoy.matching.inputs.cel_data_input, envoy.matching.inputs.destination_ip, envoy.matching.inputs.destination_port, envoy.matching.inputs.direct_source_ip, envoy.matching.inputs.dns_san, envoy.matching.inputs.request_headers, envoy.matching.inputs.request_trailers, envoy.matching.inputs.response_headers, envoy.matching.inputs.response_trailers, envoy.matching.inputs.server_name, envoy.matching.inputs.source_ip, envoy.matching.inputs.source_port, envoy.matching.inputs.source_type, envoy.matching.inputs.status_code_class_input, envoy.matching.inputs.status_code_input, envoy.matching.inputs.subject, envoy.matching.inputs.uri_san, query_params
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.geoip_providers: envoy.geoip_providers.maxmind
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.path.match: envoy.path.match.uri_template.uri_template_matcher
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.rate_limit_descriptors: envoy.rate_limit_descriptors.expr
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.grpc_credentials: envoy.grpc_credentials.aws_iam, envoy.grpc_credentials.default, envoy.grpc_credentials.file_based_metadata
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.clusters: envoy.cluster.eds, envoy.cluster.logical_dns, envoy.cluster.original_dst, envoy.cluster.static, envoy.cluster.strict_dns, envoy.clusters.aggregate, envoy.clusters.dynamic_forward_proxy, envoy.clusters.redis
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.request_id: envoy.request_id.uuid
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.tracers: envoy.dynamic.ot, envoy.tracers.datadog, envoy.tracers.dynamic_ot, envoy.tracers.opencensus, envoy.tracers.opentelemetry, envoy.tracers.skywalking, envoy.tracers.xray, envoy.tracers.zipkin, envoy.zipkin
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.regex_engines: envoy.regex_engines.google_re2
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.wasm.runtime: envoy.wasm.runtime.null, envoy.wasm.runtime.v8
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.resource_monitors: envoy.resource_monitors.fixed_heap, envoy.resource_monitors.injected_resource
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.formatter: envoy.formatter.cel, envoy.formatter.metadata, envoy.formatter.req_without_query
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.bootstrap: envoy.bootstrap.internal_listener, envoy.bootstrap.wasm, envoy.extensions.network.socket_interface.default_socket_interface
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.filters.udp.session: envoy.filters.udp.session.dynamic_forward_proxy, envoy.filters.udp.session.http_capsule
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.serializers: dubbo.hessian2
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.thrift_proxy.transports: auto, framed, header, unframed
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.internal_redirect_predicates: envoy.internal_redirect_predicates.allow_listed_routes, envoy.internal_redirect_predicates.previous_routes, envoy.internal_redirect_predicates.safe_cross_scheme
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.filters.http.upstream: envoy.buffer, envoy.ext_proc, envoy.filters.http.admission_control, envoy.filters.http.aws_lambda, envoy.filters.http.aws_request_signing, envoy.filters.http.buffer, envoy.filters.http.composite, envoy.filters.http.ext_proc, envoy.filters.http.header_mutation, envoy.filters.http.match_delegate, envoy.filters.http.upstream_codec
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.health_check.event_sinks: envoy.health_check.event_sink.file
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.filters.listener: envoy.filters.listener.http_inspector, envoy.filters.listener.local_ratelimit, envoy.filters.listener.original_dst, envoy.filters.listener.original_src, envoy.filters.listener.proxy_protocol, envoy.filters.listener.tls_inspector, envoy.listener.http_inspector, envoy.listener.original_dst, envoy.listener.original_src, envoy.listener.proxy_protocol, envoy.listener.tls_inspector
[2024-04-26 02:51:18.027][1][info][main] [source/server/server.cc:432]   envoy.http.stateful_session: envoy.http.stateful_session.cookie, envoy.http.stateful_session.header
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.common.key_value: envoy.key_value.file_based
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.dubbo_proxy.filters: envoy.filters.dubbo.router
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.quic.server_preferred_address: quic.server_preferred_address.fixed
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.tls.cert_validator: envoy.tls.cert_validator.default, envoy.tls.cert_validator.spiffe
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.rbac.matchers: envoy.rbac.matchers.upstream_ip_port
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.config_subscription: envoy.config_subscription.ads, envoy.config_subscription.ads_collection, envoy.config_subscription.aggregated_grpc_collection, envoy.config_subscription.delta_grpc, envoy.config_subscription.delta_grpc_collection, envoy.config_subscription.filesystem, envoy.config_subscription.filesystem_collection, envoy.config_subscription.grpc, envoy.config_subscription.rest
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.config_mux: envoy.config_mux.delta_grpc_mux_factory, envoy.config_mux.grpc_mux_factory, envoy.config_mux.new_grpc_mux_factory, envoy.config_mux.sotw_grpc_mux_factory
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.string_matcher: envoy.string_matcher.lua
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.matching.http.custom_matchers: envoy.matching.custom_matchers.trie_matcher
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.filters.udp_listener: envoy.filters.udp.dns_filter, envoy.filters.udp_listener.udp_proxy
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.listener_manager_impl: envoy.listener_manager_impl.default, envoy.listener_manager_impl.validation
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.quic.server.crypto_stream: envoy.quic.crypto_stream.server.quiche
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.path.rewrite: envoy.path.rewrite.uri_template.uri_template_rewriter
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.http.cache: envoy.extensions.http.cache.file_system_http_cache, envoy.extensions.http.cache.simple
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.config.validators: envoy.config.validators.minimum_clusters, envoy.config.validators.minimum_clusters_validator
[2024-04-26 02:51:18.028][1][info][main] [source/server/server.cc:432]   envoy.tracers.opentelemetry.resource_detectors: envoy.tracers.opentelemetry.resource_detectors.dynatrace, envoy.tracers.opentelemetry.resource_detectors.environment
[2024-04-26 02:51:18.031][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.config.bootstrap.v3.Admin Using deprecated option 'envoy.config.bootstrap.v3.Admin.access_log_path' from file bootstrap.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-04-26 02:51:18.031][1][info][main] [source/server/server.cc:486] HTTP header map info:
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489]   request header map: 664 bytes: :authority,:method,:path,:protocol,:scheme,accept,accept-encoding,access-control-request-headers,access-control-request-method,access-control-request-private-network,authentication,authorization,cache-control,cdn-loop,connection,content-encoding,content-length,content-type,expect,grpc-accept-encoding,grpc-timeout,if-match,if-modified-since,if-none-match,if-range,if-unmodified-since,keep-alive,origin,pragma,proxy-connection,proxy-status,referer,te,transfer-encoding,upgrade,user-agent,via,x-client-trace-id,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-downstream-service-cluster,x-envoy-downstream-service-node,x-envoy-expected-rq-timeout-ms,x-envoy-external-address,x-envoy-force-trace,x-envoy-hedge-on-per-try-timeout,x-envoy-internal,x-envoy-ip-tags,x-envoy-is-timeout-retry,x-envoy-max-retries,x-envoy-original-path,x-envoy-original-url,x-envoy-retriable-header-names,x-envoy-retriable-status-codes,x-envoy-retry-grpc-on,x-envoy-retry-on,x-envoy-upstream-alt-stat-name,x-envoy-upstream-rq-per-try-timeout-ms,x-envoy-upstream-rq-timeout-alt-response,x-envoy-upstream-rq-timeout-ms,x-envoy-upstream-stream-duration-ms,x-forwarded-client-cert,x-forwarded-for,x-forwarded-host,x-forwarded-port,x-forwarded-proto,x-ot-span-context,x-request-id
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489]   request trailer map: 120 bytes:
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489]   response header map: 432 bytes: :status,access-control-allow-credentials,access-control-allow-headers,access-control-allow-methods,access-control-allow-origin,access-control-allow-private-network,access-control-expose-headers,access-control-max-age,age,cache-control,connection,content-encoding,content-length,content-type,date,etag,expires,grpc-message,grpc-status,keep-alive,last-modified,location,proxy-connection,proxy-status,server,transfer-encoding,upgrade,vary,via,x-envoy-attempt-count,x-envoy-decorator-operation,x-envoy-degraded,x-envoy-immediate-health-check-fail,x-envoy-ratelimited,x-envoy-upstream-canary,x-envoy-upstream-healthchecked-cluster,x-envoy-upstream-service-time,x-request-id
[2024-04-26 02:51:18.032][1][info][main] [source/server/server.cc:489]   response trailer map: 144 bytes: grpc-message,grpc-status
[2024-04-26 02:51:18.037][1][info][main] [source/server/server.cc:861] runtime: layers:
  - name: static_layer
    static_layer:
      envoy.reloadable_features.use_http_client_to_fetch_aws_credentials: true
[2024-04-26 02:51:18.037][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.access_loggers.file.v3.FileAccessLog Using deprecated option 'envoy.extensions.access_loggers.file.v3.FileAccessLog.format' from file file.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2024-04-26 02:51:18.037][1][info][admin] [source/server/admin/admin.cc:66] admin address: 0.0.0.0:9901
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:168] loading tracing configuration
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:124] loading 0 static secret(s)
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:130] loading 1 cluster(s)
[2024-04-26 02:51:18.038][1][info][config] [source/server/configuration_impl.cc:138] loading 1 listener(s)
[2024-04-26 02:51:18.040][1][info][config] [source/server/configuration_impl.cc:154] loading stats configuration
[2024-04-26 02:51:18.040][1][warning][main] [source/server/server.cc:928] There is no configured limit to the number of allowed active downstream connections. Configure a limit in `envoy.resource_monitors.downstream_connections` resource monitor.
[2024-04-26 02:51:18.040][1][info][main] [source/server/server.cc:969] starting main dispatch loop
[2024-04-26 02:51:18.040][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a LOGICAL_DNS internal cluster [name: sts_token_service_internal, address:sts.eu-central-1.amazonaws.com:443] to fetch aws credentials
[2024-04-26 02:51:18.041][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a STATIC internal cluster [name: ec2_instance_metadata_server_internal, address:169.254.169.254:80] to fetch aws credentials
[2024-04-26 02:51:18.041][1][info][misc] [source/extensions/common/aws/utility.cc:381] Added a LOGICAL_DNS internal cluster [name: sts_token_service_internal, address:sts.eu-central-1.amazonaws.com:443] to fetch aws credentials
[2024-04-26 02:51:18.042][1][info][runtime] [source/common/runtime/runtime_impl.cc:614] RTDS has finished initialization
[2024-04-26 02:51:18.042][1][info][upstream] [source/common/upstream/cluster_manager_impl.cc:240] cm init: all clusters initialized
[2024-04-26 02:51:18.042][1][info][main] [source/server/server.cc:950] all clusters initialized. initializing init manager
[2024-04-26 02:51:18.071][1][error][misc] [source/extensions/common/aws/utility.cc:501] Unable to retrieve integer value from json: Expiration
[2024-04-26 02:51:18.071][1][warning][aws] [source/extensions/common/aws/credentials_provider_impl.cc:700] Could not get Expiration value of AWS credentials document from STS, so setting expiration to 1 hour in future
[2024-04-26 02:51:18.133][1][error][misc] [source/extensions/common/aws/utility.cc:501] Unable to retrieve integer value from json: Expiration
[2024-04-26 02:51:18.133][1][warning][aws] [source/extensions/common/aws/credentials_provider_impl.cc:700] Could not get Expiration value of AWS credentials document from STS, so setting expiration to 1 hour in future
[2024-04-26 02:51:18.133][1][info][config] [source/common/listener_manager/listener_manager_impl.cc:930] all dependencies initialized. starting workers
[2024-04-26 03:06:18.132][1][info][main] [source/server/drain_manager_impl.cc:208] shutting down parent after drain```

[wbpcode]

cc @mattklein123

[nbaws]

I'll pick this one up

[nbaws]

@MalibuKoKo We retrieve this from STS using json format which does in fact use the unix timestamp. However STS is returning a scientific notation version in the json payload. I'm investigating with the STS team as to whether this is accurate and then can propose a fix.

[nbaws]

It's not expected behaviour from STS, but valid json nontheless. I have a fix for this that will go to PR shortly.