What cause so many "Fail to process task" in cadence history service?

[zedongh]

Here is error info:

{
    "level": "error",
    "ts": "2023-04-17T20:35:28.840+0800",
    "msg": "Fail to process task",
    "service": "cadence-history",
    "shard-id": 562,
    "address": "10.247.3.47:7934",
    "component": "transfer-queue-processor",
    "cluster-name": "primary",
    "queue-task-id": 22758295475,
    "queue-task-visibility-timestamp": 1681734926405195130,
    "xdc-failover-version": -24,
    "queue-task-type": 1,
    "wf-domain-id": "a3f89a74-0ee1-4b2d-aa2d-8201a0c2d4bb",
    "wf-id": "v3^test-workflow^standard_workflow_2:8b55cb9b-3b81-43a1-be2e-ab49af6907dc",
    "wf-run-id": "228ebd0b-b5b9-49b9-baf2-5e679c78dec8",
    "error": "code:deadline-exceeded message:timeout",
    "lifecycle": "ProcessingFailed",
    "logging-call-at": "task.go:306",
    "stacktrace": "github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/cadence/common/log/loggerimpl/logger.go:131\ngithub.com/uber/cadence/service/history/task.(*taskImpl).HandleErr\n\t/cadence/service/history/task/task.go:306\ngithub.com/uber/cadence/common/task.(*parallelTaskProcessorImpl).executeTask.func1\n\t/cadence/common/task/parallelTaskProcessor.go:156\ngithub.com/uber/cadence/common/backoff.Retry\n\t/cadence/common/backoff/retry.go:105\ngithub.com/uber/cadence/common/task.(*parallelTaskProcessorImpl).executeTask\n\t/cadence/common/task/parallelTaskProcessor.go:171\ngithub.com/uber/cadence/common/task.(*parallelTaskProcessorImpl).taskWorker\n\t/cadence/common/task/parallelTaskProcessor.go:145"
}

And I found a similar question in here.

I deploy two cadence service by helm charts without connection by modify service port.
Both server work very well expect some strange errors in history service.