I found a work around. by using kubectl patch.

in the sts.tf

resource "kubernetes_stateful_set" "prometheus" {
  ...
          volume_mount {
            name             = "data"
            mount_path       = "/data"
            //sub_path_expr  = "$(POD_NAME)"
          }
  ...
  provisioner "local-exec" {
    command = "kubectl -n ${var.namespace} patch sts prometheus --patch \"$(cat ${path.module}/patch/subpathexpr.yaml)\""
  }
}

in the patch/subpathexpr.yaml

spec:
  template:
    spec:
      containers:
        - name: prometheus
          volumeMounts:
            - mountPath: "/data"
              subPathExpr: $(POD_NAME)

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.