Rook-Ceph
Rook is a Kubernetes operator that turns distributed storage systems like Ceph into self-managing, self-scaling, and self-healing storage services integrated deeply with Kubernetes.
Architecture Deep Dive
Custom Resources
Rook extends Kubernetes with Custom Resource Definitions (CRDs) to manage Ceph:
CephCluster
The primary resource defining the Ceph cluster:
apiVersion: ceph.rook.io/v1
kind: CephCluster
metadata:
name: rook-ceph
namespace: storage
spec:
cephVersion:
image: quay.io/ceph/ceph:v19.2.3
allowUnsupported: false
dataDirHostPath: /var/lib/rook
skipUpgradeChecks: false
continueUpgradeAfterChecksEvenIfNotHealthy: false
mon:
count: 3
allowMultiplePerNode: false
mgr:
count: 2
allowMultiplePerNode: false
modules:
- name: pg_autoscaler
enabled: true
- name: rook
enabled: true
dashboard:
enabled: true
ssl: false
storage:
useAllNodes: true
useAllDevices: false
deviceFilter: "^sd[b-z]" # Only use secondary drives
CephBlockPool
Defines storage pools for block devices:
apiVersion: ceph.rook.io/v1
kind: CephBlockPool
metadata:
name: ceph-blockpool
namespace: storage
spec:
failureDomain: host
replicated:
size: 3
requireSafeReplicaSize: true
parameters:
compression_mode: aggressive
compression_algorithm: snappy
mirroring:
enabled: false
CephObjectStore
Configures S3-compatible object storage:
apiVersion: ceph.rook.io/v1
kind: CephObjectStore
metadata:
name: ceph-objectstore
namespace: storage
spec:
metadataPool:
failureDomain: host
replicated:
size: 3
dataPool:
failureDomain: host
replicated:
size: 3
preservePoolsOnDelete: true
gateway:
instances: 2
placement:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: role
operator: In
values: [storage-node]
resources:
requests:
cpu: 500m
memory: 1Gi
limits:
cpu: 2000m
memory: 4Gi
Component Details
MON (Monitor) Daemons
Monitors maintain the cluster map and provide consensus:
# Check monitor status
kubectl -n storage exec deploy/rook-ceph-tools -- ceph mon stat
# View monitor details
kubectl -n storage exec deploy/rook-ceph-tools -- ceph mon dump
# Check quorum status
kubectl -n storage exec deploy/rook-ceph-tools -- ceph quorum_status
Functions:
- Maintain cluster membership map
- Provide authentication services
- Store cluster configuration
- Ensure data consistency through consensus
MGR (Manager) Daemons
Managers handle cluster orchestration and provide APIs:
# Check manager status
kubectl -n storage exec deploy/rook-ceph-tools -- ceph mgr stat
# View available modules
kubectl -n storage exec deploy/rook-ceph-tools -- ceph mgr module ls
# Enable/disable modules
kubectl -n storage exec deploy/rook-ceph-tools -- ceph mgr module enable dashboard
Functions:
- Cluster health monitoring
- Performance metrics collection
- Dashboard and API services
- Plugin management (Prometheus, dashboard, etc.)
OSD (Object Storage Daemon)
OSDs manage physical storage devices:
# View OSD topology
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd tree
# Check OSD performance
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd perf
# View individual OSD details
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd metadata 0
# Check OSD utilization
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd df tree
Functions:
- Store and retrieve actual data objects
- Handle replication and recovery
- Perform data scrubbing and repair
- Report health status to monitors
CSI Integration
Volume Provisioning Flow
CSI Components
# Check CSI driver pods
kubectl get pods -n storage -l app=csi-rbdplugin
kubectl get pods -n storage -l app=csi-cephfsplugin
# View CSI node registration
kubectl get csinodes
# Check available storage classes
kubectl get storageclass | grep ceph
Performance Tuning
OSD Configuration
# OSD-specific configuration
spec:
storage:
nodes:
- name: k8s-1
devices:
- name: /dev/sdb
config:
osdsPerDevice: "1"
deviceClass: "ssd"
metadataDevice: "/dev/sdc1" # Optional metadata device
Pool Optimization
# Adjust placement group count
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd pool set ceph-blockpool pg_num 128
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd pool set ceph-blockpool pgp_num 128
# Enable pool compression
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd pool set ceph-blockpool compression_mode aggressive
# Tune replication settings
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd pool set ceph-blockpool size 3
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd pool set ceph-blockpool min_size 2
Client Performance
# RBD CSI configuration for performance
apiVersion: v1
kind: ConfigMap
metadata:
name: rook-ceph-csi-config
data:
csi-cluster-config-json: |
[
{
"clusterID": "storage",
"monitors": [
"rook-ceph-mon-a:6789",
"rook-ceph-mon-b:6789",
"rook-ceph-mon-c:6789"
],
"cephFS": {
"netNamespaceFilePath": "/var/run/netns/cni-xxx"
},
"rbd": {
"netNamespaceFilePath": "/var/run/netns/cni-xxx"
},
"readAffinity": {
"enabled": true,
"crushLocationLabels": ["region", "zone"]
}
}
]
Monitoring and Metrics
Prometheus Integration
Rook automatically configures Prometheus monitoring:
# Check Prometheus targets for Ceph
kubectl port-forward -n monitoring svc/prometheus-operated 9090:9090
# Navigate to http://localhost:9090/targets and search for ceph
# Key metrics to monitor
curl -s http://localhost:9090/api/v1/query?query=ceph_cluster_total_bytes
curl -s http://localhost:9090/api/v1/query?query=ceph_cluster_total_used_bytes
curl -s http://localhost:9090/api/v1/query?query=ceph_osd_up
Common Monitoring Queries
# Cluster utilization percentage
100 * (ceph_cluster_total_used_bytes / ceph_cluster_total_bytes)
# OSD availability
sum(ceph_osd_up) / count(ceph_osd_up) * 100
# PG health status
ceph_pg_active_clean / ceph_pg_total * 100
# Average OSD utilization
avg by (ceph_daemon) (ceph_osd_utilization)
# Storage growth rate
rate(ceph_cluster_total_used_bytes[1h]) * 3600
Dashboard Access
# Get dashboard password
kubectl -n storage get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode
# Port forward to dashboard
kubectl -n storage port-forward svc/rook-ceph-mgr-dashboard 7000:7000
# Access at http://localhost:7000 (admin / <password>)
Troubleshooting
Common Issues
# Check cluster health
kubectl -n storage exec deploy/rook-ceph-tools -- ceph health detail
# View recent cluster events
kubectl get events -n storage --sort-by=.metadata.creationTimestamp
# Check OSD startup issues
kubectl logs -n storage -l app=rook-ceph-osd
# Verify device discovery
kubectl logs -n storage -l app=rook-discover
# Check operator logs
kubectl logs -n storage -l app=rook-ceph-operator
Recovery Procedures
# Restart failed OSD
kubectl -n storage delete pod -l ceph_daemon_id=0
# Force remove stuck OSD
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd out 0
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd purge 0 --yes-i-really-mean-it
# Repair inconsistent PGs
kubectl -n storage exec deploy/rook-ceph-tools -- ceph pg repair <pg-id>
# Emergency toolbox access
kubectl -n storage exec -it deploy/rook-ceph-tools -- bash
Maintenance Operations
Cluster Upgrades
# Update Ceph version in CephCluster
spec:
cephVersion:
image: quay.io/ceph/ceph:v19.2.3 # New version
allowUnsupported: false
Adding Storage
# Add new device to existing node
kubectl -n storage edit cephcluster rook-ceph
# Add device to devices list
# Scale OSD deployment
kubectl -n storage get deployment -l app=rook-ceph-osd
# Verify new OSD integration
kubectl -n storage exec deploy/rook-ceph-tools -- ceph osd tree
Rook-Ceph provides a robust, self-healing storage foundation that integrates seamlessly with Kubernetes, delivering enterprise-grade distributed storage capabilities with operational simplicity.