diff --git a/Documentation/README.md b/Documentation/README.md index 12aaf91933a3..e57ddd93a84c 100644 --- a/Documentation/README.md +++ b/Documentation/README.md @@ -6,23 +6,18 @@ Rook turns storage software into self-managing, self-scaling, and self-healing s Rook integrates deeply into cloud native environments leveraging extension points and providing a seamless experience for scheduling, lifecycle management, resource management, security, monitoring, and user experience. -For more details about the status of storage solutions currently supported by Rook, please refer to the [project status section](https://github.com/rook/rook/blob/master/README.md#project-status) of the Rook repository. -We plan to continue adding support for other storage systems and environments based on community demand and engagement in future releases. +The Ceph operator was declared stable in December 2018 in the Rook v0.9 release, providing a production storage platform for several years already. -## Quick Start Guides +## Quick Start Guide -Starting Rook in your cluster is as simple as a few `kubectl` commands depending on the storage provider. -See our [Quickstart](quickstart.md) guide list for the detailed instructions for each storage provider. +Starting Ceph in your cluster is as simple as a few `kubectl` commands. +See our [Quickstart](quickstart.md) guide to get started with the Ceph operator! -## Storage Provider Designs +## Designs -High-level Storage Provider design documents: +[Ceph](https://docs.ceph.com/en/latest/) is a highly scalable distributed storage solution for block storage, object storage, and shared filesystems with years of production deployments. See the [Ceph overview](ceph-storage.md). -| Storage Provider | Status | Description | -| ----------------------- | ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [Ceph](ceph-storage.md) | Stable | Ceph is a highly scalable distributed storage solution for block storage, object storage, and shared filesystems with years of production deployments. | - -Low level design documentation for supported list of storage systems collected at [design docs](https://github.com/rook/rook/tree/master/design) section. +For detailed design documentation, see also the [design docs](https://github.com/rook/rook/tree/master/design). ## Need help? Be sure to join the Rook Slack diff --git a/Documentation/authenticated-registry.md b/Documentation/authenticated-registry.md new file mode 100644 index 000000000000..f9903e8cc96f --- /dev/null +++ b/Documentation/authenticated-registry.md @@ -0,0 +1,65 @@ +--- +title: Authenticated Registries +weight: 1100 +indent: true +--- + +## Authenticated docker registries + +If you want to use an image from authenticated docker registry (e.g. for image cache/mirror), you'll need to +add an `imagePullSecret` to all relevant service accounts. This way all pods created by the operator (for service account: +`rook-ceph-system`) or all new pods in the namespace (for service account: `default`) will have the `imagePullSecret` added +to their spec. + +The whole process is described in the [official kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account). + +### Example setup for a ceph cluster + +To get you started, here's a quick rundown for the ceph example from the [quickstart guide](/Documentation/quickstart.md). + +First, we'll create the secret for our registry as described [here](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod): + +```console +# for namespace rook-ceph +$ kubectl -n rook-ceph create secret docker-registry my-registry-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL + +# and for namespace rook-ceph (cluster) +$ kubectl -n rook-ceph create secret docker-registry my-registry-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL +``` + +Next we'll add the following snippet to all relevant service accounts as described [here](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account): + +```yaml +imagePullSecrets: +- name: my-registry-secret +``` + +The service accounts are: + +* `rook-ceph-system` (namespace: `rook-ceph`): Will affect all pods created by the rook operator in the `rook-ceph` namespace. +* `default` (namespace: `rook-ceph`): Will affect most pods in the `rook-ceph` namespace. +* `rook-ceph-mgr` (namespace: `rook-ceph`): Will affect the MGR pods in the `rook-ceph` namespace. +* `rook-ceph-osd` (namespace: `rook-ceph`): Will affect the OSD pods in the `rook-ceph` namespace. + +You can do it either via e.g. `kubectl -n edit serviceaccount default` or by modifying the [`operator.yaml`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/operator.yaml) +and [`cluster.yaml`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/cluster.yaml) before deploying them. + +Since it's the same procedure for all service accounts, here is just one example: + +```console +kubectl -n rook-ceph edit serviceaccount default +``` + +```yaml +apiVersion: v1 +kind: ServiceAccount +metadata: + name: default + namespace: rook-ceph +secrets: +- name: default-token-12345 +imagePullSecrets: # here are the new +- name: my-registry-secret # parts +``` + +After doing this for all service accounts all pods should be able to pull the image from your registry. diff --git a/Documentation/ceph-block.md b/Documentation/ceph-block.md index 62920ece6fcf..62615a2231f3 100644 --- a/Documentation/ceph-block.md +++ b/Documentation/ceph-block.md @@ -11,7 +11,7 @@ Block storage allows a single pod to mount storage. This guide shows how to crea ## Prerequisites -This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md). +This guide assumes a Rook cluster as explained in the [Quickstart](quickstart.md). ## Provision Storage diff --git a/Documentation/ceph-client-crd.md b/Documentation/ceph-client-crd.md index 6cae6ffcd3da..7a8fccd67b63 100644 --- a/Documentation/ceph-client-crd.md +++ b/Documentation/ceph-client-crd.md @@ -44,4 +44,4 @@ spec: ### Prerequisites -This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](ceph-quickstart.md) +This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](quickstart.md) diff --git a/Documentation/ceph-cluster-crd.md b/Documentation/ceph-cluster-crd.md index 392d86608c2e..404d86eb2830 100755 --- a/Documentation/ceph-cluster-crd.md +++ b/Documentation/ceph-cluster-crd.md @@ -451,7 +451,7 @@ Below are the settings for host-based cluster. This type of cluster can specify * `config`: Device-specific config settings. See the [config settings](#osd-configuration-settings) below Host-based cluster only supports raw device and partition. Be sure to see the -[Ceph quickstart doc prerequisites](ceph-quickstart.md#prerequisites) for additional considerations. +[Ceph quickstart doc prerequisites](quickstart.md#prerequisites) for additional considerations. Below are the settings for a PVC-based cluster. diff --git a/Documentation/ceph-common-issues.md b/Documentation/ceph-common-issues.md index beedb1693888..d8c8960cfe21 100644 --- a/Documentation/ceph-common-issues.md +++ b/Documentation/ceph-common-issues.md @@ -92,7 +92,7 @@ If you see that the PVC remains in **pending** state, see the topic [PVCs stay i ### Possible Solutions Summary -* `rook-ceph-agent` pod is in a `CrashLoopBackOff` status because it cannot deploy its driver on a read-only filesystem: [Flexvolume configuration pre-reqs](./ceph-prerequisites.md#ceph-flexvolume-configuration) +* `rook-ceph-agent` pod is in a `CrashLoopBackOff` status because it cannot deploy its driver on a read-only filesystem: [Flexvolume configuration pre-reqs](./prerequisites.md#ceph-flexvolume-configuration) * Persistent Volume and/or Claim are failing to be created and bound: [Volume Creation](#volume-creation) * `rook-ceph-agent` pod is failing to mount and format the volume: [Rook Agent Mounting](#volume-mounting) @@ -165,7 +165,7 @@ First, clean up the agent deployment with: kubectl -n rook-ceph delete daemonset rook-ceph-agent ``` -Once the `rook-ceph-agent` pods are gone, **follow the instructions in the [Flexvolume configuration pre-reqs](./ceph-prerequisites.md#ceph-flexvolume-configuration)** to ensure a good value for `--volume-plugin-dir` has been provided to the Kubelet. +Once the `rook-ceph-agent` pods are gone, **follow the instructions in the [Flexvolume configuration pre-reqs](./prerequisites.md#ceph-flexvolume-configuration)** to ensure a good value for `--volume-plugin-dir` has been provided to the Kubelet. After that has been configured, and the Kubelet has been restarted, start the agent pods up again by restarting `rook-operator`: ```console diff --git a/Documentation/ceph-filesystem.md b/Documentation/ceph-filesystem.md index 0988ae4dd144..70c336d02e43 100644 --- a/Documentation/ceph-filesystem.md +++ b/Documentation/ceph-filesystem.md @@ -13,7 +13,7 @@ This example runs a shared filesystem for the [kube-registry](https://github.com ## Prerequisites -This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](ceph-quickstart.md) +This guide assumes you have created a Rook cluster as explained in the main [Kubernetes guide](quickstart.md) ### Multiple Filesystems Support diff --git a/Documentation/ceph-fs-mirror-crd.md b/Documentation/ceph-fs-mirror-crd.md index 192ea65009d8..a5d05afabd20 100644 --- a/Documentation/ceph-fs-mirror-crd.md +++ b/Documentation/ceph-fs-mirror-crd.md @@ -5,7 +5,7 @@ indent: true --- {% include_relative branch.liquid %} -This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](ceph-quickstart.md) +This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](quickstart.md) # Ceph FilesystemMirror CRD @@ -97,4 +97,3 @@ If any setting is unspecified, a suitable default will be used automatically. * `labels`: Key value pair list of labels to add. * `resources`: The resource requirements for the cephfs-mirror pods. * `priorityClassName`: The priority class to set on the cephfs-mirror pods. - diff --git a/Documentation/ceph-object-multisite.md b/Documentation/ceph-object-multisite.md index 97e397141c88..db9061b0ab77 100644 --- a/Documentation/ceph-object-multisite.md +++ b/Documentation/ceph-object-multisite.md @@ -22,7 +22,7 @@ To review core multisite concepts please read the [ceph-multisite design overvie ## Prerequisites -This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md). +This guide assumes a Rook cluster as explained in the [Quickstart](quickstart.md). # Creating Object Multisite diff --git a/Documentation/ceph-object.md b/Documentation/ceph-object.md index 6a9270425b0f..d03e18e7699d 100644 --- a/Documentation/ceph-object.md +++ b/Documentation/ceph-object.md @@ -10,7 +10,7 @@ Object storage exposes an S3 API to the storage cluster for applications to put ## Prerequisites -This guide assumes a Rook cluster as explained in the [Quickstart](ceph-quickstart.md). +This guide assumes a Rook cluster as explained in the [Quickstart](quickstart.md). ## Configure an Object Store diff --git a/Documentation/ceph-osd-mgmt.md b/Documentation/ceph-osd-mgmt.md index d12f7eb07d73..d528dc4abe24 100644 --- a/Documentation/ceph-osd-mgmt.md +++ b/Documentation/ceph-osd-mgmt.md @@ -33,7 +33,7 @@ kubectl -n rook-ceph exec -it $(kubectl -n rook-ceph get pod -l "app=rook-ceph-t ## Add an OSD -The [QuickStart Guide](ceph-quickstart.md) will provide the basic steps to create a cluster and start some OSDs. For more details on the OSD +The [QuickStart Guide](quickstart.md) will provide the basic steps to create a cluster and start some OSDs. For more details on the OSD settings also see the [Cluster CRD](ceph-cluster-crd.md) documentation. If you are not seeing OSDs created, see the [Ceph Troubleshooting Guide](ceph-common-issues.md). To add more OSDs, Rook will automatically watch for new nodes and devices being added to your cluster. @@ -70,10 +70,10 @@ If you are using `useAllDevices: true`, no change to the CR is necessary. removal steps in order to prevent Rook from detecting the old OSD and trying to re-create it before the disk is wiped or removed.** -To stop the Rook Operator, run +To stop the Rook Operator, run `kubectl -n rook-ceph scale deployment rook-ceph-operator --replicas=0`. -You must perform steps below to (1) purge the OSD and either (2.a) delete the underlying data or +You must perform steps below to (1) purge the OSD and either (2.a) delete the underlying data or (2.b)replace the disk before starting the Rook Operator again. Once you have done that, you can start the Rook operator again with diff --git a/Documentation/ceph-quickstart.md b/Documentation/ceph-quickstart.md deleted file mode 100644 index 545f1a5a5e43..000000000000 --- a/Documentation/ceph-quickstart.md +++ /dev/null @@ -1,189 +0,0 @@ ---- -title: Ceph Storage -weight: 300 -indent: true ---- - -{% include_relative branch.liquid %} - - -# Ceph Storage Quickstart - -This guide will walk you through the basic setup of a Ceph cluster and enable you to consume block, object, and file storage -from other pods running in your cluster. - -## Minimum Version - -Kubernetes **v1.11** or higher is supported by Rook. - -**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Rook CRDs. Create the `crds.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. - -## Prerequisites - -To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](k8s-pre-reqs.md). - -In order to configure the Ceph storage cluster, at least one of these local storage options are required: -- Raw devices (no partitions or formatted filesystems) - - This requires `lvm2` to be installed on the host. - To avoid this dependency, you can create a single full-disk partition on the disk (see below) -- Raw partitions (no formatted filesystem) -- Persistent Volumes available from a storage class in `block` mode - -You can confirm whether your partitions or devices are formatted filesystems with the following command. - -```console -lsblk -f -``` ->``` ->NAME FSTYPE LABEL UUID MOUNTPOINT ->vda ->└─vda1 LVM2_member >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB -> ├─ubuntu--vg-root ext4 c2366f76-6e21-4f10-a8f3-6776212e2fe4 / -> └─ubuntu--vg-swap_1 swap 9492a3dc-ad75-47cd-9596-678e8cf17ff9 [SWAP] ->vdb ->``` - -If the `FSTYPE` field is not empty, there is a filesystem on top of the corresponding device. In this case, you can use vdb for Ceph and can't use vda and its partitions. - -## TL;DR - -If you're feeling lucky, a simple Rook cluster can be created with the following kubectl commands and [example yaml files](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph). For the more detailed install, skip to the next section to [deploy the Rook operator](#deploy-the-rook-operator). - -```console -$ git clone --single-branch --branch v1.7.3 https://github.com/rook/rook.git -cd rook/cluster/examples/kubernetes/ceph -kubectl create -f crds.yaml -f common.yaml -f operator.yaml -kubectl create -f cluster.yaml -``` - -After the cluster is running, you can create [block, object, or file](#storage) storage to be consumed by other applications in your cluster. - -### Cluster Environments - -The Rook documentation is focused around starting Rook in a production environment. Examples are also -provided to relax some settings for test environments. When creating the cluster later in this guide, consider these example cluster manifests: -- [cluster.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster.yaml): Cluster settings for a production cluster running on bare metal. Requires at least three worker nodes. -- [cluster-on-pvc.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml): Cluster settings for a production cluster running in a dynamic cloud environment. -- [cluster-test.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-test.yaml): Cluster settings for a test environment such as minikube. - -See the [Ceph examples](ceph-examples.md) for more details. - -## Deploy the Rook Operator - -The first step is to deploy the Rook operator. Check that you are using the [example yaml files](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph) that correspond to your release of Rook. For more options, see the [examples documentation](ceph-examples.md). - -```console -cd cluster/examples/kubernetes/ceph -kubectl create -f crds.yaml -f common.yaml -f operator.yaml - -# verify the rook-ceph-operator is in the `Running` state before proceeding -kubectl -n rook-ceph get pod -``` - -You can also deploy the operator with the [Rook Helm Chart](helm-operator.md). - -Before you start the operator in production, there are some settings that you may want to consider: -1. If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml`. - The apiextension v1beta1 version of CustomResourceDefinition was deprecated in Kubernetes v1.16. -2. Consider if you want to enable certain Rook features that are disabled by default. See the [operator.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator.yaml) for these and other advanced settings. - 1. Device discovery: Rook will watch for new devices to configure if the `ROOK_ENABLE_DISCOVERY_DAEMON` setting is enabled, commonly used in bare metal clusters. - 2. Flex driver: The flex driver is deprecated in favor of the CSI driver, but can still be enabled with the `ROOK_ENABLE_FLEX_DRIVER` setting. - 3. Node affinity and tolerations: The CSI driver by default will run on any node in the cluster. To configure the CSI driver affinity, several settings are available. - -If you wish to deploy into a namespace other than the default `rook-ceph`, see the -[Ceph advanced configuration section](ceph-advanced-configuration.md#using-alternate-namespaces) on the topic. - -## Create a Rook Ceph Cluster - -Now that the Rook operator is running we can create the Ceph cluster. For the cluster to survive reboots, -make sure you set the `dataDirHostPath` property that is valid for your hosts. For more settings, see the documentation on [configuring the cluster](ceph-cluster-crd.md). - -Create the cluster: - -```console -kubectl create -f cluster.yaml -``` - -Use `kubectl` to list pods in the `rook-ceph` namespace. You should be able to see the following pods once they are all running. -The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured. -If you did not modify the `cluster.yaml` above, it is expected that one OSD will be created per node. -The CSI, `rook-ceph-agent` (flex driver), and `rook-discover` pods are also optional depending on your settings. - -> If the `rook-ceph-mon`, `rook-ceph-mgr`, or `rook-ceph-osd` pods are not created, please refer to the -> [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. - -```console -kubectl -n rook-ceph get pod -``` - ->``` ->NAME READY STATUS RESTARTS AGE ->csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s ->csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s ->csi-cephfsplugin-rthrp 3/3 Running 0 140s ->csi-rbdplugin-hbsm7 3/3 Running 0 140s ->csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s ->csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s ->rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s ->rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s ->rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s ->rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s ->rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s ->rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m ->rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s ->rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s ->rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s ->rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s ->rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s ->rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s ->``` - -To verify that the cluster is in a healthy state, connect to the [Rook toolbox](ceph-toolbox.md) and run the -`ceph status` command. - -* All mons should be in quorum -* A mgr should be active -* At least one OSD should be active -* If the health is not `HEALTH_OK`, the warnings or errors should be investigated - -```console -ceph status -``` ->``` -> cluster: -> id: a0452c76-30d9-4c1a-a948-5d8405f19a7c -> health: HEALTH_OK -> -> services: -> mon: 3 daemons, quorum a,b,c (age 3m) -> mgr: a(active, since 2m) -> osd: 3 osds: 3 up (since 1m), 3 in (since 1m) ->... ->``` - -If the cluster is not healthy, please refer to the [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. - -## Storage - -For a walkthrough of the three types of storage exposed by Rook, see the guides for: - -* **[Block](ceph-block.md)**: Create block storage to be consumed by a pod -* **[Object](ceph-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster -* **[Shared Filesystem](ceph-filesystem.md)**: Create a filesystem to be shared across multiple pods - -## Ceph Dashboard - -Ceph has a dashboard in which you can view the status of your cluster. Please see the [dashboard guide](ceph-dashboard.md) for more details. - -## Tools - -We have created a toolbox container that contains the full suite of Ceph clients for debugging and troubleshooting your Rook cluster. Please see the [toolbox readme](ceph-toolbox.md) for setup and usage information. Also see our [advanced configuration](ceph-advanced-configuration.md) document for helpful maintenance and tuning examples. - -## Monitoring - -Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/). -To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./ceph-monitoring.md). - -## Teardown - -When you are done with the test cluster, see [these instructions](ceph-teardown.md) to clean up the cluster. diff --git a/Documentation/ceph-rbd-mirror-crd.md b/Documentation/ceph-rbd-mirror-crd.md index 1820f6bb5353..9213554a85d9 100644 --- a/Documentation/ceph-rbd-mirror-crd.md +++ b/Documentation/ceph-rbd-mirror-crd.md @@ -27,7 +27,7 @@ spec: ### Prerequisites -This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](ceph-quickstart.md) +This guide assumes you have created a Rook cluster as explained in the main [Quickstart guide](quickstart.md) ## Settings diff --git a/Documentation/ceph-storage.md b/Documentation/ceph-storage.md index 28dc963f108d..19e632f836a6 100644 --- a/Documentation/ceph-storage.md +++ b/Documentation/ceph-storage.md @@ -9,28 +9,26 @@ Ceph is a highly scalable distributed storage solution for **block storage**, ** ## Design -Rook enables Ceph storage systems to run on Kubernetes using Kubernetes primitives. The following image illustrates how Ceph Rook integrates with Kubernetes: - -![Rook Architecture on Kubernetes](media/rook-architecture.png) +Rook enables Ceph storage to run on Kubernetes using Kubernetes primitives. With Ceph running in the Kubernetes cluster, Kubernetes applications can mount block devices and filesystems managed by Rook, or can use the S3/Swift API for object storage. The Rook operator automates configuration of storage components and monitors the cluster to ensure the storage remains available and healthy. The Rook operator is a simple container that has all that is needed to bootstrap -and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](ceph-mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other artifacts necessary to run the services. +and monitor the storage cluster. The operator will start and monitor [Ceph monitor pods](ceph-mon-health.md), the Ceph OSD daemons to provide RADOS storage, as well as start and manage other Ceph daemons. The operator manages CRDs for pools, object stores (S3/Swift), and filesystems by initializing the pods and other resources necessary to run the services. The operator will monitor the storage daemons to ensure the cluster is healthy. Ceph mons will be started or failed over when necessary, and other adjustments are made as the cluster grows or shrinks. The operator will also watch for desired state changes -requested by the api service and apply the changes. +specified in the Ceph custom resources (CRs) and apply the changes. -The Rook operator also initializes the agents that are needed for consuming the storage. Rook automatically configures the Ceph-CSI driver to mount the storage to your pods. Rook's flex driver is also available, though it is not enabled by default and will soon be deprecated in favor of the CSI driver. +Rook automatically configures the Ceph-CSI driver to mount the storage to your pods. ![Rook Components on Kubernetes](media/kubernetes.png) -The `rook/ceph` image includes all necessary tools to manage the cluster -- there is no change to the data path. -Rook does not attempt to maintain full fidelity with Ceph. Many of the Ceph concepts like placement groups and crush maps -are hidden so you don't have to worry about them. Instead Rook creates a much simplified user experience for admins that is in terms +The `rook/ceph` image includes all necessary tools to manage the cluster. Rook is not in the Ceph data path. +Many of the Ceph concepts like placement groups and crush maps +are hidden so you don't have to worry about them. Instead Rook creates a simplified user experience for admins that is in terms of physical resources, pools, volumes, filesystems, and buckets. At the same time, advanced configuration can be applied when needed with the Ceph tools. Rook is implemented in golang. Ceph is implemented in C++ where the data path is highly optimized. We believe diff --git a/Documentation/ceph-toolbox.md b/Documentation/ceph-toolbox.md index 0b8d5a809bbe..b78ddbddcfc1 100644 --- a/Documentation/ceph-toolbox.md +++ b/Documentation/ceph-toolbox.md @@ -13,7 +13,7 @@ The toolbox can be run in two modes: 1. [Interactive](#interactive-toolbox): Start a toolbox pod where you can connect and execute Ceph commands from a shell 2. [One-time job](#toolbox-job): Run a script with Ceph commands and collect the results from the job log -> Prerequisite: Before running the toolbox you should have a running Rook cluster deployed (see the [Quickstart Guide](ceph-quickstart.md)). +> Prerequisite: Before running the toolbox you should have a running Rook cluster deployed (see the [Quickstart Guide](quickstart.md)). ## Interactive Toolbox diff --git a/Documentation/development-flow.md b/Documentation/development-flow.md index 41718a73e8eb..2e80caf309e2 100644 --- a/Documentation/development-flow.md +++ b/Documentation/development-flow.md @@ -99,6 +99,7 @@ rook ├── cluster │   ├── charts # Helm charts │   │   └── rook-ceph +│   │   └── rook-ceph-cluster │   └── examples # Sample yaml files for Rook cluster │ ├── cmd # Binaries with main entrypoint @@ -134,7 +135,6 @@ rook    │   ├── installer # installs Rook and its supported storage providers into integration tests environments    │   └── utils    ├── integration # all test cases that will be invoked during integration testing -    ├── longhaul # longhaul tests    └── scripts # scripts for setting up integration and manual testing environments ``` @@ -148,7 +148,6 @@ To add a feature or to make a bug fix, you will need to create a branch in your For new features of significant scope and complexity, a design document is recommended before work begins on the implementation. So create a design document if: -* Adding a new storage provider * Adding a new CRD * Adding a significant feature to an existing storage provider. If the design is simple enough to describe in a github issue, you likely don't need a full design doc. @@ -350,18 +349,7 @@ By default, you should always open a pull request against master. The flow for getting a fix into a release branch is: 1. Open a PR to merge the changes to master following the process outlined above. -2. Add the backport label to that PR such as backport-release-1.1 +2. Add the backport label to that PR such as backport-release-1.7 3. After your PR is merged to master, the mergify bot will automatically open a PR with your commits backported to the release branch 4. If there are any conflicts you will need to resolve them by pulling the branch, resolving the conflicts and force push back the branch 5. After the CI is green, the bot will automatically merge the backport PR. - -## Debugging operators locally - -Operators are meant to be run inside a Kubernetes cluster. However, this makes it harder to use debugging tools and slows down the developer cycle of edit-build-test since testing requires to build a container image, push to the cluster, restart the pods, get logs, etc. - -A common operator developer practice is to run the operator locally on the developer machine in order to leverage the developer tools and comfort. - -In order to support this external operator mode, rook detects if the operator is running outside of the cluster (using standard cluster env) and changes the behavior as follows: - -* Connecting to Kubernetes API will load the config from the user `~/.kube/config`. -* Instead of the default [CommandExecutor](../pkg/util/exec/exec.go) this mode uses a [TranslateCommandExecutor](../pkg/util/exec/translate_exec.go) that executes every command issued by the operator to run as a Kubernetes job inside the cluster, so that any tools that the operator needs from its image can be called. diff --git a/Documentation/flexvolume.md b/Documentation/flexvolume.md index 58af53e38a55..f45ac23bbf75 100644 --- a/Documentation/flexvolume.md +++ b/Documentation/flexvolume.md @@ -23,7 +23,6 @@ Platform-specific instructions for the following Kubernetes deployment platforms * [OpenShift](#openshift) * [OpenStack Magnum](#openstack-magnum) * [Rancher](#rancher) -* [Tectonic](#tectonic) * [Custom containerized kubelet](#custom-containerized-kubelet) * [Configuring the FlexVolume path](#configuring-the-flexvolume-path) @@ -132,13 +131,6 @@ FlexVolume path for the Rook operator. If the default path as above is used no further configuration is required, otherwise if a different path is used the Rook operator will need to be reconfigured, to do this continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. -## Tectonic - -Follow [these instructions](tectonic.md) to configure the Flexvolume plugin for Rook on Tectonic during ContainerLinux node ignition file provisioning. -If you want to use Rook with an already provisioned Tectonic cluster, please refer to the [ContainerLinux](#containerlinux) section. - -Continue with [configuring the FlexVolume path](#configuring-the-flexvolume-path) to configure Rook to use the FlexVolume path. - ## Custom containerized kubelet Use the [most common read/write FlexVolume path](#most-common-readwrite-flexvolume-path) for the next steps. diff --git a/Documentation/helm-operator.md b/Documentation/helm-operator.md index 7049ca665095..3c5ba03243b4 100644 --- a/Documentation/helm-operator.md +++ b/Documentation/helm-operator.md @@ -25,7 +25,7 @@ See the [Helm support matrix](https://helm.sh/docs/topics/version_skew/) for mor The Ceph Operator helm chart will install the basic components necessary to create a storage platform for your Kubernetes cluster. 1. Install the Helm chart -1. [Create a Rook cluster](ceph-quickstart.md#create-a-rook-cluster). +1. [Create a Rook cluster](quickstart.md#create-a-rook-cluster). The `helm install` command deploys rook on the Kubernetes cluster in the default configuration. The [configuration](#configuration) section lists the parameters that can be configured during installation. It is recommended that the rook operator be installed into the `rook-ceph` namespace (you will install your clusters into separate namespaces). @@ -106,7 +106,7 @@ The following tables lists the configurable parameters of the rook-operator char | `csi.provisionerPriorityClassName` | PriorityClassName to be set on csi driver provisioner pods. | | | `csi.enableOMAPGenerator` | EnableOMAP generator deploys omap sidecar in CSI provisioner pod, to enable it set it to true | `false` | | `csi.rbdFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the RBD PVC is being mounted | ReadWriteOnceWithFSType | -| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted | `None` | +| `csi.cephFSFSGroupPolicy` | Policy for modifying a volume's ownership or permissions when the CephFS PVC is being mounted | `None` | | `csi.logLevel` | Set logging level for csi containers. Supported values from 0 to 5. 0 for general useful logs, 5 for trace level verbosity. | `0` | | `csi.enableGrpcMetrics` | Enable Ceph CSI GRPC Metrics. | `false` | | `csi.enableCSIHostNetwork` | Enable Host Networking for Ceph CSI nodeplugins. | `false` | diff --git a/Documentation/k8s-pre-reqs.md b/Documentation/k8s-pre-reqs.md deleted file mode 100644 index df45dc19fa87..000000000000 --- a/Documentation/k8s-pre-reqs.md +++ /dev/null @@ -1,163 +0,0 @@ ---- -title: Prerequisites -weight: 1000 ---- -{% include_relative branch.liquid %} - -# Prerequisites - -Rook can be installed on any existing Kubernetes cluster as long as it meets the minimum version -and Rook is granted the required privileges (see below for more information). If you don't have a Kubernetes cluster, -you can quickly set one up using [Minikube](#minikube), [Kubeadm](#kubeadm) or [CoreOS/Vagrant](#new-local-kubernetes-cluster-with-vagrant). - -## Minimum Version - -Kubernetes **v1.11** or higher is supported for the Ceph operator. -Kubernetes **v1.16** or higher is supported for the Cassandra and NFS operators. - -**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Ceph CRDs. Create the `crds.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. - -## Ceph Prerequisites - -See also **[Ceph Prerequisites](ceph-prerequisites.md)**. - -## Pod Security Policies - -Rook requires privileges to manage the storage in your cluster. If you have Pod Security Policies enabled -please review this section. By default, Kubernetes clusters do not have PSPs enabled so you may -be able to skip this section. - -If you are configuring Ceph on OpenShift, the Ceph walkthrough will configure the PSPs as well -when you start the operator with [operator-openshift.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator-openshift.yaml). - -### Cluster Role - -> **NOTE**: Cluster role configuration is only needed when you are not already `cluster-admin` in your Kubernetes cluster! - -Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`. - -One simple way to achieve it is to assign your certificate with the `system:masters` group: - -```console --subj "/CN=admin/O=system:masters" -``` - -`system:masters` is a special group that is bound to `cluster-admin` ClusterRole, but it can't be easily revoked so be careful with taking that route in a production setting. -Binding individual certificate to ClusterRole `cluster-admin` is revocable by deleting the ClusterRoleBinding. - -### RBAC for PodSecurityPolicies - -If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are -using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings` -for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods. - -Security policies will differ for different backends. See Ceph's Pod Security Policies set up in -[common.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/common.yaml) -for an example of how this is done in practice. - -### PodSecurityPolicy - -You need at least one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example -which should be more permissive than is needed for any backend: - -```yaml -apiVersion: policy/v1beta1 -kind: PodSecurityPolicy -metadata: - name: privileged -spec: - fsGroup: - rule: RunAsAny - privileged: true - runAsUser: - rule: RunAsAny - seLinux: - rule: RunAsAny - supplementalGroups: - rule: RunAsAny - volumes: - - '*' - allowedCapabilities: - - '*' - hostPID: true - # hostNetwork is required for using host networking - hostNetwork: false -``` - -**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in a Cluster `CustomResourceDefinition`! -You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given -port range will allow all ports: - -```yaml - hostPorts: - # Ceph msgr2 port - - min: 1 - max: 65535 -``` - -## Authenticated docker registries - -If you want to use an image from authenticated docker registry (e.g. for image cache/mirror), you'll need to -add an `imagePullSecret` to all relevant service accounts. This way all pods created by the operator (for service account: -`rook-ceph-system`) or all new pods in the namespace (for service account: `default`) will have the `imagePullSecret` added -to their spec. - -The whole process is described in the [official kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account). - -### Example setup for a ceph cluster - -To get you started, here's a quick rundown for the ceph example from the [quickstart guide](/Documentation/ceph-quickstart.md). - -First, we'll create the secret for our registry as described [here](https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod): - -```console -# for namespace rook-ceph -$ kubectl -n rook-ceph create secret docker-registry my-registry-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL - -# and for namespace rook-ceph (cluster) -$ kubectl -n rook-ceph create secret docker-registry my-registry-secret --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL -``` - -Next we'll add the following snippet to all relevant service accounts as described [here](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#add-imagepullsecrets-to-a-service-account): - -```yaml -imagePullSecrets: -- name: my-registry-secret -``` - -The service accounts are: - -* `rook-ceph-system` (namespace: `rook-ceph`): Will affect all pods created by the rook operator in the `rook-ceph` namespace. -* `default` (namespace: `rook-ceph`): Will affect most pods in the `rook-ceph` namespace. -* `rook-ceph-mgr` (namespace: `rook-ceph`): Will affect the MGR pods in the `rook-ceph` namespace. -* `rook-ceph-osd` (namespace: `rook-ceph`): Will affect the OSD pods in the `rook-ceph` namespace. - -You can do it either via e.g. `kubectl -n edit serviceaccount default` or by modifying the [`operator.yaml`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/operator.yaml) -and [`cluster.yaml`](https://github.com/rook/rook/blob/master/cluster/examples/kubernetes/ceph/cluster.yaml) before deploying them. - -Since it's the same procedure for all service accounts, here is just one example: - -```console -kubectl -n rook-ceph edit serviceaccount default -``` - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: default - namespace: rook-ceph -secrets: -- name: default-token-12345 -imagePullSecrets: # here are the new -- name: my-registry-secret # parts -``` - -After doing this for all service accounts all pods should be able to pull the image from your registry. - -## Bootstrapping Kubernetes - -Rook will run wherever Kubernetes is running. Here are a couple of simple environments to help you get started with Rook. - -* [Minikube](https://github.com/kubernetes/minikube/releases): A single-node cluster, simplest to get started -* [Kubeadm](https://kubernetes.io/docs/setup/independent/install-kubeadm/): One or more nodes for more comprehensive deployments diff --git a/Documentation/media/edgefs-isgw-edit.png b/Documentation/media/edgefs-isgw-edit.png deleted file mode 100644 index 8bbce92f1893..000000000000 Binary files a/Documentation/media/edgefs-isgw-edit.png and /dev/null differ diff --git a/Documentation/media/edgefs-isgw.png b/Documentation/media/edgefs-isgw.png deleted file mode 100644 index af413c7418da..000000000000 Binary files a/Documentation/media/edgefs-isgw.png and /dev/null differ diff --git a/Documentation/media/edgefs-rook.png b/Documentation/media/edgefs-rook.png deleted file mode 100644 index a62a6f9d96cf..000000000000 Binary files a/Documentation/media/edgefs-rook.png and /dev/null differ diff --git a/Documentation/media/edgefs-ui-dashboard.png b/Documentation/media/edgefs-ui-dashboard.png deleted file mode 100644 index 31c589b4e7bd..000000000000 Binary files a/Documentation/media/edgefs-ui-dashboard.png and /dev/null differ diff --git a/Documentation/media/edgefs-ui-nfs-edit.png b/Documentation/media/edgefs-ui-nfs-edit.png deleted file mode 100644 index fad41d819c07..000000000000 Binary files a/Documentation/media/edgefs-ui-nfs-edit.png and /dev/null differ diff --git a/Documentation/media/minio_demo.png b/Documentation/media/minio_demo.png deleted file mode 100644 index 64c6dfad00e4..000000000000 Binary files a/Documentation/media/minio_demo.png and /dev/null differ diff --git a/Documentation/media/nfs-webhook-deployment.png b/Documentation/media/nfs-webhook-deployment.png deleted file mode 100644 index df3cfe4b40f3..000000000000 Binary files a/Documentation/media/nfs-webhook-deployment.png and /dev/null differ diff --git a/Documentation/media/nfs-webhook-validation-flow.png b/Documentation/media/nfs-webhook-validation-flow.png deleted file mode 100644 index 67f7c8a6116a..000000000000 Binary files a/Documentation/media/nfs-webhook-validation-flow.png and /dev/null differ diff --git a/Documentation/media/rook-architecture.png b/Documentation/media/rook-architecture.png deleted file mode 100644 index 2118717cf361..000000000000 Binary files a/Documentation/media/rook-architecture.png and /dev/null differ diff --git a/Documentation/pod-security-policies.md b/Documentation/pod-security-policies.md new file mode 100644 index 000000000000..9062ab2a2a34 --- /dev/null +++ b/Documentation/pod-security-policies.md @@ -0,0 +1,67 @@ +--- +title: Pod Security Policies +weight: 1300 +indent: true +--- +{% include_relative branch.liquid %} + +## Pod Security Policies + +Rook requires privileges to manage the storage in your cluster. If you have Pod Security Policies enabled +please review this document. By default, Kubernetes clusters do not have PSPs enabled so you may +be able to skip this document. + +If you are configuring Ceph on OpenShift, the Ceph walkthrough will configure the PSPs as well +when you start the operator with [operator-openshift.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator-openshift.yaml). + +Creating the Rook operator requires privileges for setting up RBAC. To launch the operator you need to have created your user certificate that is bound to ClusterRole `cluster-admin`. + +### RBAC for PodSecurityPolicies + +If you have activated the [PodSecurityPolicy Admission Controller](https://kubernetes.io/docs/admin/admission-controllers/#podsecuritypolicy) and thus are +using [PodSecurityPolicies](https://kubernetes.io/docs/concepts/policy/pod-security-policy/), you will require additional `(Cluster)RoleBindings` +for the different `ServiceAccounts` Rook uses to start the Rook Storage Pods. + +Security policies will differ for different backends. See Ceph's Pod Security Policies set up in +[common.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/common.yaml) +for an example of how this is done in practice. + +### PodSecurityPolicy + +You need at least one `PodSecurityPolicy` that allows privileged `Pod` execution. Here is an example +which should be more permissive than is needed for any backend: + +```yaml +apiVersion: policy/v1beta1 +kind: PodSecurityPolicy +metadata: + name: privileged +spec: + fsGroup: + rule: RunAsAny + privileged: true + runAsUser: + rule: RunAsAny + seLinux: + rule: RunAsAny + supplementalGroups: + rule: RunAsAny + volumes: + - '*' + allowedCapabilities: + - '*' + hostPID: true + # hostNetwork is required for using host networking + hostNetwork: false +``` + +**Hint**: Allowing `hostNetwork` usage is required when using `hostNetwork: true` in a Cluster `CustomResourceDefinition`! +You are then also required to allow the usage of `hostPorts` in the `PodSecurityPolicy`. The given +port range will allow all ports: + +```yaml + hostPorts: + # Ceph msgr2 port + - min: 1 + max: 65535 +``` diff --git a/Documentation/ceph-prerequisites.md b/Documentation/pre-reqs.md similarity index 59% rename from Documentation/ceph-prerequisites.md rename to Documentation/pre-reqs.md index 5e9819aa6deb..a991a3452553 100644 --- a/Documentation/ceph-prerequisites.md +++ b/Documentation/pre-reqs.md @@ -1,18 +1,43 @@ --- title: Prerequisites -weight: 2010 -indent: true +weight: 1000 --- +{% include_relative branch.liquid %} -# Ceph Prerequisites +# Prerequisites -To make sure you have a Kubernetes cluster that is ready for `Rook`, review the general [Rook Prerequisites](k8s-pre-reqs.md). +Rook can be installed on any existing Kubernetes cluster as long as it meets the minimum version +and Rook is granted the required privileges (see below for more information). + +## Minimum Version + +Kubernetes **v1.11** or higher is supported for the Ceph operator. + +**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Ceph CRDs. Create the `crds.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. + +## Ceph Prerequisites In order to configure the Ceph storage cluster, at least one of these local storage options are required: - Raw devices (no partitions or formatted filesystems) - Raw partitions (no formatted filesystem) - PVs available from a storage class in `block` mode +You can confirm whether your partitions or devices are formatted with filesystems with the following command. + +```console +lsblk -f +``` +>``` +>NAME FSTYPE LABEL UUID MOUNTPOINT +>vda +>└─vda1 LVM2_member >eSO50t-GkUV-YKTH-WsGq-hNJY-eKNf-3i07IB +> ├─ubuntu--vg-root ext4 c2366f76-6e21-4f10-a8f3-6776212e2fe4 / +> └─ubuntu--vg-swap_1 swap 9492a3dc-ad75-47cd-9596-678e8cf17ff9 [SWAP] +>vdb +>``` + +If the `FSTYPE` field is not empty, there is a filesystem on top of the corresponding device. In this example, you can use `vdb` for Ceph and can't use `vda` or its partitions. + ## LVM package Ceph OSDs have a dependency on LVM in the following scenarios: @@ -50,17 +75,6 @@ runcmd: - [ vgchange, -ay ] ``` -## Ceph Flexvolume Configuration - -**NOTE** This configuration is only needed when using the FlexVolume driver (required for Kubernetes 1.12 or earlier). The Ceph-CSI RBD driver or the Ceph-CSI CephFS driver are recommended for Kubernetes 1.13 and newer, making FlexVolume configuration redundant. - -If you want to configure volumes with the Flex driver instead of CSI, the Rook agent requires setup as a Flex volume plugin to manage the storage attachments in your cluster. -See the [Flex Volume Configuration](flexvolume.md) topic to configure your Kubernetes deployment to load the Rook volume plugin. - -### Extra agent mounts - -On certain distributions it may be necessary to mount additional directories into the agent container. That is what the environment variable `AGENT_MOUNTS` is for. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.mounts`. The format of the variable content should be `mountname1=/host/path1:/container/path1,mountname2=/host/path2:/container/path2`. - ## Kernel ### RBD @@ -77,7 +91,3 @@ or choose a different Linux distribution. If you will be creating volumes from a Ceph shared file system (CephFS), the recommended minimum kernel version is **4.17**. If you have a kernel version less than 4.17, the requested PVC sizes will not be enforced. Storage quotas will only be enforced on newer kernels. - -## Kernel modules directory configuration - -Normally, on Linux, kernel modules can be found in `/lib/modules`. However, there are some distributions that put them elsewhere. In that case the environment variable `LIB_MODULES_DIR_PATH` can be used to override the default. Also see the documentation in [helm-operator](helm-operator.md) on the parameter `agent.libModulesDirPath`. One notable distribution where this setting is useful would be [NixOS](https://nixos.org). diff --git a/Documentation/quickstart.md b/Documentation/quickstart.md index 308290eae849..4874dfad2831 100644 --- a/Documentation/quickstart.md +++ b/Documentation/quickstart.md @@ -1,19 +1,174 @@ --- title: Quickstart -weight: 200 +weight: 300 --- -# Quickstart Guides +{% include_relative branch.liquid %} -Welcome to Rook! We hope you have a great experience installing the Rook **cloud-native storage orchestrator** platform to enable highly available, durable storage -in your Kubernetes cluster. +# Ceph Quickstart + +Welcome to Rook! We hope you have a great experience installing the Rook **cloud-native storage orchestrator** platform to enable highly available, durable Ceph storage in your Kubernetes cluster. If you have any questions along the way, please don't hesitate to ask us in our [Slack channel](https://rook-io.slack.com). You can sign up for our Slack [here](https://slack.rook.io). -Rook provides a growing number of storage providers to a Kubernetes cluster, each with its own operator to deploy and manage the resources for the storage provider. +This guide will walk you through the basic setup of a Ceph cluster and enable you to consume block, object, and file storage +from other pods running in your cluster. + +## Minimum Version + +Kubernetes **v1.11** or higher is supported by Rook. + +**Important** If you are using K8s 1.15 or older, you will need to create a different version of the Rook CRDs. Create the `crds.yaml` found in the [pre-k8s-1.16](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/pre-k8s-1.16) subfolder of the example manifests. + +## Prerequisites + +To make sure you have a Kubernetes cluster that is ready for `Rook`, you can [follow these instructions](pre-reqs.md). + +In order to configure the Ceph storage cluster, at least one of these local storage options are required: +- Raw devices (no partitions or formatted filesystems) + - This requires `lvm2` to be installed on the host. + To avoid this dependency, you can create a single full-disk partition on the disk (see below) +- Raw partitions (no formatted filesystem) +- Persistent Volumes available from a storage class in `block` mode + +## TL;DR + +A simple Rook cluster can be created with the following kubectl commands and [example manifests](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph). + +```console +$ git clone --single-branch --branch {{ branchName }} https://github.com/rook/rook.git +cd rook/cluster/examples/kubernetes/ceph +kubectl create -f crds.yaml -f common.yaml -f operator.yaml +kubectl create -f cluster.yaml +``` + +After the cluster is running, you can create [block, object, or file](#storage) storage to be consumed by other applications in your cluster. + +## Deploy the Rook Operator + +The first step is to deploy the Rook operator. Check that you are using the [example yaml files](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph) that correspond to your release of Rook. For more options, see the [examples documentation](ceph-examples.md). + +```console +cd cluster/examples/kubernetes/ceph +kubectl create -f crds.yaml -f common.yaml -f operator.yaml + +# verify the rook-ceph-operator is in the `Running` state before proceeding +kubectl -n rook-ceph get pod +``` + +You can also deploy the operator with the [Rook Helm Chart](helm-operator.md). + +Before you start the operator in production, there are some settings that you may want to consider: +1. If you are using kubernetes v1.15 or older you need to create CRDs found here `/cluster/examples/kubernetes/ceph/pre-k8s-1.16/crd.yaml`. + The apiextension v1beta1 version of CustomResourceDefinition was deprecated in Kubernetes v1.16. +2. Consider if you want to enable certain Rook features that are disabled by default. See the [operator.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/operator.yaml) for these and other advanced settings. + 1. Device discovery: Rook will watch for new devices to configure if the `ROOK_ENABLE_DISCOVERY_DAEMON` setting is enabled, commonly used in bare metal clusters. + 2. Flex driver: The flex driver is deprecated in favor of the CSI driver, but can still be enabled with the `ROOK_ENABLE_FLEX_DRIVER` setting. + 3. Node affinity and tolerations: The CSI driver by default will run on any node in the cluster. To configure the CSI driver affinity, several settings are available. + +If you wish to deploy into a namespace other than the default `rook-ceph`, see the +[Ceph advanced configuration section](ceph-advanced-configuration.md#using-alternate-namespaces) on the topic. + +## Cluster Environments + +The Rook documentation is focused around starting Rook in a production environment. Examples are also +provided to relax some settings for test environments. When creating the cluster later in this guide, consider these example cluster manifests: +- [cluster.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster.yaml): Cluster settings for a production cluster running on bare metal. Requires at least three worker nodes. +- [cluster-on-pvc.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-on-pvc.yaml): Cluster settings for a production cluster running in a dynamic cloud environment. +- [cluster-test.yaml](https://github.com/rook/rook/blob/{{ branchName }}/cluster/examples/kubernetes/ceph/cluster-test.yaml): Cluster settings for a test environment such as minikube. + +See the [Ceph examples](ceph-examples.md) for more details. + +## Create a Ceph Cluster + +Now that the Rook operator is running we can create the Ceph cluster. For the cluster to survive reboots, +make sure you set the `dataDirHostPath` property that is valid for your hosts. For more settings, see the documentation on [configuring the cluster](ceph-cluster-crd.md). + +Create the cluster: + +```console +kubectl create -f cluster.yaml +``` + +Use `kubectl` to list pods in the `rook-ceph` namespace. You should be able to see the following pods once they are all running. +The number of osd pods will depend on the number of nodes in the cluster and the number of devices configured. +If you did not modify the `cluster.yaml` above, it is expected that one OSD will be created per node. + +> If the `rook-ceph-mon`, `rook-ceph-mgr`, or `rook-ceph-osd` pods are not created, please refer to the +> [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. + +```console +kubectl -n rook-ceph get pod +``` + +>``` +>NAME READY STATUS RESTARTS AGE +>csi-cephfsplugin-provisioner-d77bb49c6-n5tgs 5/5 Running 0 140s +>csi-cephfsplugin-provisioner-d77bb49c6-v9rvn 5/5 Running 0 140s +>csi-cephfsplugin-rthrp 3/3 Running 0 140s +>csi-rbdplugin-hbsm7 3/3 Running 0 140s +>csi-rbdplugin-provisioner-5b5cd64fd-nvk6c 6/6 Running 0 140s +>csi-rbdplugin-provisioner-5b5cd64fd-q7bxl 6/6 Running 0 140s +>rook-ceph-crashcollector-minikube-5b57b7c5d4-hfldl 1/1 Running 0 105s +>rook-ceph-mgr-a-64cd7cdf54-j8b5p 1/1 Running 0 77s +>rook-ceph-mon-a-694bb7987d-fp9w7 1/1 Running 0 105s +>rook-ceph-mon-b-856fdd5cb9-5h2qk 1/1 Running 0 94s +>rook-ceph-mon-c-57545897fc-j576h 1/1 Running 0 85s +>rook-ceph-operator-85f5b946bd-s8grz 1/1 Running 0 92m +>rook-ceph-osd-0-6bb747b6c5-lnvb6 1/1 Running 0 23s +>rook-ceph-osd-1-7f67f9646d-44p7v 1/1 Running 0 24s +>rook-ceph-osd-2-6cd4b776ff-v4d68 1/1 Running 0 25s +>rook-ceph-osd-prepare-node1-vx2rz 0/2 Completed 0 60s +>rook-ceph-osd-prepare-node2-ab3fd 0/2 Completed 0 60s +>rook-ceph-osd-prepare-node3-w4xyz 0/2 Completed 0 60s +>``` + +To verify that the cluster is in a healthy state, connect to the [Rook toolbox](ceph-toolbox.md) and run the +`ceph status` command. + +* All mons should be in quorum +* A mgr should be active +* At least one OSD should be active +* If the health is not `HEALTH_OK`, the warnings or errors should be investigated + +```console +ceph status +``` +>``` +> cluster: +> id: a0452c76-30d9-4c1a-a948-5d8405f19a7c +> health: HEALTH_OK +> +> services: +> mon: 3 daemons, quorum a,b,c (age 3m) +> mgr: a(active, since 2m) +> osd: 3 osds: 3 up (since 1m), 3 in (since 1m) +>... +>``` + +If the cluster is not healthy, please refer to the [Ceph common issues](ceph-common-issues.md) for more details and potential solutions. + +## Storage + +For a walkthrough of the three types of storage exposed by Rook, see the guides for: + +* **[Block](ceph-block.md)**: Create block storage to be consumed by a pod (RWO) +* **[Shared Filesystem](ceph-filesystem.md)**: Create a filesystem to be shared across multiple pods (RWX) +* **[Object](ceph-object.md)**: Create an object store that is accessible inside or outside the Kubernetes cluster + +## Ceph Dashboard + +Ceph has a dashboard in which you can view the status of your cluster. Please see the [dashboard guide](ceph-dashboard.md) for more details. + +## Tools + +Create a toolbox pod for full access to a ceph admin client for debugging and troubleshooting your Rook cluster. Please see the [toolbox documentation](ceph-toolbox.md) for setup and usage information. Also see our [advanced configuration](ceph-advanced-configuration.md) document for helpful maintenance and tuning examples. + +## Monitoring + +Each Rook cluster has some built in metrics collectors/exporters for monitoring with [Prometheus](https://prometheus.io/). +To learn how to set up monitoring for your Rook cluster, you can follow the steps in the [monitoring guide](./ceph-monitoring.md). -**Follow these guides to get started with each provider**: +## Teardown -| Storage Provider | Status | Description | -| -------------------------- | ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | -| [Ceph](ceph-quickstart.md) | Stable / V1 | Ceph is a highly scalable distributed storage solution for block storage, object storage, and shared filesystems with years of production deployments. | +When you are done with the test cluster, see [these instructions](ceph-teardown.md) to clean up the cluster. diff --git a/Documentation/tectonic.md b/Documentation/tectonic.md deleted file mode 100644 index 23ec6440dd88..000000000000 --- a/Documentation/tectonic.md +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: Tectonic Configuration -weight: 11800 -indent: true ---- - -# Tectonic Configuration - -Here is a running guide on how to implement Rook on Tectonic. A complete guide on how to install Tectonic is out of the scope of the Rook project. More info can be found on the [Tectonic website](https://coreos.com/tectonic/docs/latest/) - -## Prerequisites - -* An installed tectonic-installer. These steps are described on [the Tectonic website](https://coreos.com/tectonic/docs/latest/install/bare-metal/#4-tectonic-installer) -* A running matchbox node which will do the provisioning (Matchbox is only required if you are running Tectonic on Bare metal) -* You can run through all steps of the GUI installer, but in the last step, choose `Boot manually`. This way we can make the necessary changes first. - -## Edit the kubelet.service file -We need to make a few adaptions to the Kubelet systemd service file generated by the Tectonic-installer. - -First change to the directory in which you untarred the tectonic installer and find your newly generated cluster configuration files. - -```console -cd ~/tectonic/tectonic-installer/LINUX-OR-DARWIN/clusters -``` - - -Open the file `modules/ignition/resources/services/kubelet.service` in your favorite editor and after the last line containing `ExecStartPre=...`, paste the following extra lines: - -```console -ExecStartPre=/bin/mkdir -p /var/lib/kubelet/volumeplugins -ExecStartPre=/bin/mkdir -p /var/lib/rook -``` - -And after the `ExecStart=/usr/lib/coreos/kubelet-wrapper \` line, insert the following flag for the kubelet-wrapper to point to a path reachable outside of the Kubelet rkt container: - -```console ---volume-plugin-dir=/var/lib/kubelet/volumeplugins \ -``` - -Save and close the file. - -### Boot your Tectonic cluster - -All the preparations are ready for Tectonic to boot now. We will use `terraform` to start the cluster. -Visit the official [Tectonic manual boot](https://coreos.com/tectonic/docs/latest/install/aws/manual-boot.html#deploy-the-cluster) page for the commands to use. - -**Remark:** The Tectonic installer contains the correct terraform binary out of the box. This terraform binary can be found in following directory `~/tectonic/tectonic-installer/linux`. - -## Start Rook - -After the Tectonic Installer ran and the Kubernetes cluster is started and ready, you can follow the [Rook installation guide](ceph-quickstart.md). -If you want to specify which disks Rook uses, follow the instructions in [creating Rook clusters](ceph-cluster-crd.md)