Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Object Gateway panel does not work in Ceph Dashboard (KeyError: 'id') #14058

Closed
hemantsonawane95 opened this issue Apr 11, 2024 · 10 comments
Closed

Object Gateway panel does not work in Ceph Dashboard (KeyError: 'id') #14058

hemantsonawane95 opened this issue Apr 11, 2024 · 10 comments
Assignees
@rkachach
Labels
bug

Comments

@hemantsonawane95
Copy link

hemantsonawane95 commented Apr 11, 2024 •
edited

Is this a bug report or feature request?

  • Bug Report

Deviation from expected behavior:

In ceph dashboard the Object Gateway panel having 500 "Internal Server Error". None of the pages of this panel works. After digging in I managed to gather some logs:

rook-ceph-mgr-a-6664b67588-9nbqt mgr debug 2024-04-11T09:09:54.481+0000 7f5181386700  0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=192.168.101.57, port=8080, ssl=False
rook-ceph-mgr-a-6664b67588-9nbqt mgr debug 2024-04-11T09:09:54.485+0000 7f5181386700  0 [dashboard INFO rgw_client] Found RGW daemon with configuration: host=192.168.101.49, port=8080, ssl=False
rook-ceph-mgr-a-6664b67588-9nbqt mgr debug 2024-04-11T09:09:54.485+0000 7f5181386700  0 [dashboard ERROR exception] Internal Server Error
rook-ceph-mgr-a-6664b67588-9nbqt mgr Traceback (most recent call last):
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/services/exception.py", line 47, in dashboard_exception_handler
rook-ceph-mgr-a-6664b67588-9nbqt mgr     return handler(*args, **kwargs)
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/lib/python3.6/site-packages/cherrypy/_cpdispatch.py", line 54, in __call__
rook-ceph-mgr-a-6664b67588-9nbqt mgr     return self.callable(*self.args, **self.kwargs)
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 263, in inner
rook-ceph-mgr-a-6664b67588-9nbqt mgr     ret = func(*args, **kwargs)
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/controllers/rgw.py", line 56, in status
rook-ceph-mgr-a-6664b67588-9nbqt mgr     instance = RgwClient.admin_instance()
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 382, in admin_instance
rook-ceph-mgr-a-6664b67588-9nbqt mgr     return RgwClient.instance(daemon_name=daemon_name)
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 340, in instance
rook-ceph-mgr-a-6664b67588-9nbqt mgr     RgwClient._daemons = _get_daemons()
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 72, in _get_daemons
rook-ceph-mgr-a-6664b67588-9nbqt mgr     daemon = _determine_rgw_addr(daemon_map[key])
rook-ceph-mgr-a-6664b67588-9nbqt mgr   File "/usr/share/ceph/mgr/dashboard/services/rgw_client.py", line 92, in _determine_rgw_addr
rook-ceph-mgr-a-6664b67588-9nbqt mgr     who=name_to_config_section('rgw.' + daemon_info['metadata']['id']),  # noqa E501 #pylint: disable=line-too-long
rook-ceph-mgr-a-6664b67588-9nbqt mgr KeyError: 'id'
rook-ceph-mgr-a-6664b67588-9nbqt mgr debug 2024-04-11T09:09:54.485+0000 7f5181386700  0 [dashboard ERROR request] [::ffff:85.163.56.57:51329] [GET] [500] [0.005s] [admin] [513.0B] /ui-api/rgw/status
rook-ceph-mgr-a-6664b67588-9nbqt mgr debug 2024-04-11T09:09:54.485+0000 7f5181386700  0 [dashboard ERROR request] [b'{"status": "500 Internal Server Error", "detail": "The server encountered an unexpected condition which prevented it from fulfilling the request.", "request_id": "3a084733-be95-4543-839d-4907baa936da"}

Ther error refers to KeyError: 'id'.

This is happening whenever I click on any tab in Object Gateway panel in Ceph Dashboard.

Screenshot 2024-04-11 at 11 15 13

I have also referred issue #9660 which was already reported earlier and it was looking like similar to mine. But referring it and trying steps explained there did not helped me. Any idea about this issue and possible solution for it ?

Expected behavior:

Object Gateway panel should be enabled by default as radosgw is working fine and running.

How to reproduce it (minimal and precise):

  • Once ceph cluster is deployed expose ceph dashboard to access it in browser.
  • Click on the "Object Gateway" panel and then click on the any options underneath of it for e.g Overview, Gateways, Users, Buckets, Multi-Site
  • You will see "Page Not Found" error exactly same as above attached image.
@travisn
Copy link
Member

travisn commented Apr 11, 2024

@hemantsonawane95 What version of Rook and Ceph are you using?

@hemantsonawane95
Copy link
Author

@travisn
ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)
rook-version: v1.13.7

@rkachach
Copy link
Contributor

@hemantsonawane95 please, can you share the specs you used to create our object store?

@hemantsonawane95
Copy link
Author

hemantsonawane95 commented Apr 11, 2024 •
edited

@rkachach I used default

object store created in rook-ceph namespace

apiVersion: v1
items:
- apiVersion: ceph.rook.io/v1
  kind: CephObjectStore
  metadata:
    annotations:
      meta.helm.sh/release-name: rook-ceph-rook-ceph-cluster
      meta.helm.sh/release-namespace: rook-ceph
    creationTimestamp: "2024-04-03T09:52:02Z"
    finalizers:
    - cephobjectstore.ceph.rook.io
    generation: 2
    labels:
      app.kubernetes.io/managed-by: Helm
      helm.toolkit.fluxcd.io/name: rook-ceph-cluster
      helm.toolkit.fluxcd.io/namespace: rook-ceph
    name: ceph-objectstore
    namespace: rook-ceph
    resourceVersion: "1427260"
  spec:
    dataPool:
      erasureCoded:
        codingChunks: 0
        dataChunks: 0
      failureDomain: host
      mirroring: {}
      quotas: {}
      replicated:
        size: 3
      statusCheck:
        mirror: {}
    gateway:
      instances: 2
      placement: {}
      port: 8080
      priorityClassName: system-cluster-critical
      resources:
        limits:
          cpu: "2"
          memory: 2Gi
        requests:
          cpu: "1"
          memory: 1Gi
    healthCheck: {}
    metadataPool:
      erasureCoded:
        codingChunks: 0
        dataChunks: 0
      failureDomain: host
      mirroring: {}
      quotas: {}
      replicated:
        size: 3
      statusCheck:
        mirror: {}
    preservePoolsOnDelete: true
    zone:
      name: ""
  status:
    endpoints:
      insecure:
      - http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:8080
      secure: []
    info:
      endpoint: http://rook-ceph-rgw-ceph-objectstore.rook-ceph.svc:8080
    observedGeneration: 2
    phase: Ready
kind: List
metadata:
  resourceVersion: ""

config:

    # ceph object storage configuration
    cephObjectStores:
      - name: ceph-objectstore
        spec:
          metadataPool:
            failureDomain: host
            replicated:
              size: 3
          dataPool:
            failureDomain: host
            replicated:
              size: 3
          preservePoolsOnDelete: true
          gateway:
            port: 8080
            resources:
              limits:
                cpu: "2000m"
                memory: "2Gi"
              requests:
                cpu: "1000m"
                memory: "1Gi"
            # securePort: 443
            # sslCertificateRef:
            instances: 2
            priorityClassName: system-cluster-critical
        storageClass:
          enabled: true
          name: ceph-bucket
          reclaimPolicy: Delete

@hemantsonawane95
Copy link
Author

@rkachach Hi did you try to reproduce it ?

@rkachach
Copy link
Contributor

@hemantsonawane95 I tried to reproduce the issue by I couldn't. In my case rgws using a similar spec are created correctly. Please, in case you are able to reproduce the issue (or still running on the same) it will be good to see what's the output of the following command (run from the cmd tools):

ceph status --format json | jq

@hemantsonawane95
Copy link
Author

@rkachach Yeah still the same issue.

ceph status --format json | jq
{
  "fsid": "bcf036df-6da1-473c-a38a-jhgfkop",
  "health": {
    "status": "HEALTH_OK",
    "checks": {},
    "mutes": []
  },
  "election_epoch": 6,
  "quorum": [
    0,
    1,
    2
  ],
  "quorum_names": [
    "c",
    "a",
    "b"
  ],
  "quorum_age": 2240362,
  "monmap": {
    "epoch": 1,
    "min_mon_release_name": "reef",
    "num_mons": 3
  },
  "osdmap": {
    "epoch": 2873,
    "num_osds": 84,
    "num_up_osds": 84,
    "osd_up_since": 1713259521,
    "num_in_osds": 84,
    "osd_in_since": 1712229618,
    "num_remapped_pgs": 0
  },
  "pgmap": {
    "pgs_by_state": [
      {
        "state_name": "active+clean",
        "count": 1735
      },
      {
        "state_name": "active+clean+scrubbing+deep",
        "count": 3
      },
      {
        "state_name": "active+clean+scrubbing",
        "count": 2
      }
    ],
    "num_pgs": 1740,
    "num_pools": 17,
    "num_objects": 2937887,
    "data_bytes": 12234357780523,
    "bytes_used": 35751625273344,
    "bytes_avail": 469714502701056,
    "bytes_total": 505466127974400,
    "read_bytes_sec": 6069121,
    "write_bytes_sec": 45239941,
    "read_op_per_sec": 237,
    "write_op_per_sec": 438
  },
  "fsmap": {
    "epoch": 1,
    "by_rank": [],
    "up:standby": 0
  },
  "mgrmap": {
    "available": true,
    "num_standbys": 1,
    "modules": [
      "dashboard",
      "iostat",
      "nfs",
      "prometheus",
      "restful",
      "rook"
    ],
    "services": {
      "dashboard": "https://10.xx.x.xxx:8443/",
      "prometheus": "http://10.xx.x.xxx:9283/"
    }
  },
  "servicemap": {
    "epoch": 21516,
    "modified": "2024-04-29T08:07:51.204083+0000",
    "services": {
      "rgw": {
        "daemons": {
          "summary": "",
          "10725": {
            "start_epoch": 4,
            "start_stamp": "2024-04-03T10:02:24.121835+0000",
            "gid": 10725,
            "addr": "192.xxx.x.xxx:0/2702331193",
            "metadata": {
              "arch": "x86_64",
              "ceph_release": "reef",
              "ceph_version": "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)",
              "ceph_version_short": "18.2.2",
              "container_hostname": "ceph01",
              "container_image": "quay.io/ceph/ceph:v18.2.2",
              "cpu": "AMD EPYC 7452 32-Core Processor",
              "distro": "centos",
              "distro_description": "CentOS Stream 8",
              "distro_version": "8",
              "frontend_config#0": "beast port=8080",
              "frontend_type#0": "beast",
              "hostname": "ceph01",
              "id": "ceph.objectstore.a",
              "kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
              "kernel_version": "6.5.0-26-generic",
              "mem_swap_kb": "0",
              "mem_total_kb": "528182824",
              "num_handles": "1",
              "os": "Linux",
              "pid": "13",
              "pod_name": "rook-ceph-rgw-ceph-objectstore-a-65d5cb88d4-dlj97",
              "pod_namespace": "rook-ceph",
              "realm_id": "be275a37-0428-47a2-a490-sdsadfasfacs2334",
              "realm_name": "ceph-objectstore",
              "zone_id": "d6c15726-9083-4efa-a46f-8adrgyjjm2380",
              "zone_name": "ceph-objectstore",
              "zonegroup_id": "946d3b80-5c8b-48fc-9a0a-sg5th67jukjs",
              "zonegroup_name": "ceph-objectstore"
            },
            "task_status": {}
          },
          "18959": {
            "start_epoch": 4,
            "start_stamp": "2024-04-03T10:02:24.127984+0000",
            "gid": 18959,
            "addr": "192.xxx.x.xxx:0/936826137",
            "metadata": {
              "arch": "x86_64",
              "ceph_release": "reef",
              "ceph_version": "ceph version 18.2.2 (531c0d11a1c5d39fbfe6aa8a521f023abf3bf3e2) reef (stable)",
              "ceph_version_short": "18.2.2",
              "container_hostname": "ceph02",
              "container_image": "quay.io/ceph/ceph:v18.2.2",
              "cpu": "AMD EPYC 7452 32-Core Processor",
              "distro": "centos",
              "distro_description": "CentOS Stream 8",
              "distro_version": "8",
              "frontend_config#0": "beast port=8080",
              "frontend_type#0": "beast",
              "hostname": "ceph02",
              "id": "ceph.objectstore.a",
              "kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
              "kernel_version": "6.5.0-26-generic",
              "mem_swap_kb": "0",
              "mem_total_kb": "528182812",
              "num_handles": "1",
              "os": "Linux",
              "pid": "13",
              "pod_name": "rook-ceph-rgw-ceph-objectstore-a-65d5cb88d4-blrrr",
              "pod_namespace": "rook-ceph",
              "realm_id": "be275a37-0428-75h2-a490-224ebef4b996",
              "realm_name": "ceph-objectstore",
              "zone_id": "d6c15726-9083-3fg6-a46f-85d68637d4ef",
              "zone_name": "ceph-objectstore",
              "zonegroup_id": "946d3b80-7jl9-48fc-9a0a-659a4ffeb07d",
              "zonegroup_name": "ceph-objectstore"
            },
            "task_status": {}
          },
          "ceph-rgw-6c995b947b-l5pbr": {
            "start_epoch": 6133,
            "start_stamp": "2024-04-10T16:47:56.079373+0000",
            "gid": 2238000,
            "addr": "10.233.68.242:0/3542550434",
            "metadata": {
              "arch": "x86_64",
              "ceph_release": "octopus",
              "ceph_version": "ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)",
              "ceph_version_short": "15.2.15",
              "cpu": "AMD EPYC 7452 32-Core Processor",
              "distro": "ubuntu",
              "distro_description": "Ubuntu 18.04.6 LTS",
              "distro_version": "18.04",
              "frontend_config#0": "civetweb port=8088",
              "frontend_type#0": "civetweb",
              "hostname": "ceph-rgw-6c995b947b-l5pbr",
              "kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
              "kernel_version": "6.5.0-26-generic",
              "mem_swap_kb": "0",
              "mem_total_kb": "528182824",
              "num_handles": "1",
              "os": "Linux",
              "pid": "61",
              "zone_id": "d6c15726-9083-4efa-a46f-8hd68534y4ef",
              "zone_name": "ceph-objectstore",
              "zonegroup_id": "946d3b80-5c8b-48fc-7hj8-6586gjle807f",
              "zonegroup_name": "ceph-objectstore"
            },
            "task_status": {}
          },
          "ceph-rgw-6c995b947b-mmt78": {
            "start_epoch": 6133,
            "start_stamp": "2024-04-10T16:47:56.068111+0000",
            "gid": 1827607,
            "addr": "10.233.83.147:0/816919461",
            "metadata": {
              "arch": "x86_64",
              "ceph_release": "octopus",
              "ceph_version": "ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)",
              "ceph_version_short": "15.2.15",
              "cpu": "AMD EPYC 7452 32-Core Processor",
              "distro": "ubuntu",
              "distro_description": "Ubuntu 18.04.6 LTS",
              "distro_version": "18.04",
              "frontend_config#0": "civetweb port=8088",
              "frontend_type#0": "civetweb",
              "hostname": "ceph-rgw-6c995b947b-mmt78",
              "kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
              "kernel_version": "6.5.0-26-generic",
              "mem_swap_kb": "0",
              "mem_total_kb": "528182800",
              "num_handles": "1",
              "os": "Linux",
              "pid": "61",
              "zone_id": "d6c15726-9083-4efa-a0k8-85d68637d4ef",
              "zone_name": "ceph-objectstore",
              "zonegroup_id": "946d3b80-5c8b-09l6-9a0a-659a4ffeb07d",
              "zonegroup_name": "ceph-objectstore"
            },
            "task_status": {}
          }
        }
      }
    }
  },
  "progress_events": {}
}

@rkachach
Copy link
Contributor

rkachach commented Apr 29, 2024 •
edited

@hemantsonawane95 thanks for the info. The error you are experimenting is caused by the two rgw daemons listed bellow ceph-rgw-6c995b947b-l5pbr and ceph-rgw-6c995b947b-mmt78. Both of them have a very old ceph version "ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)" hence they lack the new field 'id' that's why you get the exception.. seems like a left-over from some an upgrade but TBH I'm not sure how could they be still there. Maybe @travisn or @BlaineEXE can have some idea on how could we reach this situation.

		"ceph-rgw-6c995b947b-l5pbr": {
		    "start_epoch": 6133,
		    "start_stamp": "2024-04-10T16:47:56.079373+0000",
		    "gid": 2238000,
		    "addr": "10.233.68.242:0/3542550434",
		    "metadata": {
			"arch": "x86_64",
			"ceph_release": "octopus",
			"ceph_version": "ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)",
			"ceph_version_short": "15.2.15",
			"cpu": "AMD EPYC 7452 32-Core Processor",
			"distro": "ubuntu",
			"distro_description": "Ubuntu 18.04.6 LTS",
			"distro_version": "18.04",
			"frontend_config#0": "civetweb port=8088",
			"frontend_type#0": "civetweb",
			"hostname": "ceph-rgw-6c995b947b-l5pbr",
			"kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
			"kernel_version": "6.5.0-26-generic",
			"mem_swap_kb": "0",
			"mem_total_kb": "528182824",
			"num_handles": "1",
			"os": "Linux",
			"pid": "61",
			"zone_id": "d6c15726-9083-4efa-a46f-8hd68534y4ef",
			"zone_name": "ceph-objectstore",
			"zonegroup_id": "946d3b80-5c8b-48fc-7hj8-6586gjle807f",
			"zonegroup_name": "ceph-objectstore"
		    },
		    "task_status": {}
		},
		"ceph-rgw-6c995b947b-mmt78": {
		    "start_epoch": 6133,
		    "start_stamp": "2024-04-10T16:47:56.068111+0000",
		    "gid": 1827607,
		    "addr": "10.233.83.147:0/816919461",
		    "metadata": {
			"arch": "x86_64",
			"ceph_release": "octopus",
			"ceph_version": "ceph version 15.2.15 (2dfb18841cfecc2f7eb7eb2afd65986ca4d95985) octopus (stable)",
			"ceph_version_short": "15.2.15",
			"cpu": "AMD EPYC 7452 32-Core Processor",
			"distro": "ubuntu",
			"distro_description": "Ubuntu 18.04.6 LTS",
			"distro_version": "18.04",
			"frontend_config#0": "civetweb port=8088",
			"frontend_type#0": "civetweb",
			"hostname": "ceph-rgw-6c995b947b-mmt78",
			"kernel_description": "#26~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 12 10:22:43 UTC 2",
			"kernel_version": "6.5.0-26-generic",
			"mem_swap_kb": "0",
			"mem_total_kb": "528182800",
			"num_handles": "1",
			"os": "Linux",
			"pid": "61",
			"zone_id": "d6c15726-9083-4efa-a0k8-85d68637d4ef",
			"zone_name": "ceph-objectstore",
			"zonegroup_id": "946d3b80-5c8b-09l6-9a0a-659a4ffeb07d",
			"zonegroup_name": "ceph-objectstore"
		    },
		    "task_status": {}
		}
            }

@hemantsonawane95
Copy link
Author

Hey @rkachach Thank you a lot for pointing out the rgw daemons version issue. I managed to fix the issue by upgrading them to pacific version. I think what you have explained clarifies everything and we can close this issue.

@rkachach
Copy link
Contributor

@hemantsonawane95 thanks for getting back. I'm closing the issue 👍

@rkachach rkachach closed this as completed May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@rkachach rkachach

Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@travisn @rkachach @hemantsonawane95