About Ceph

Ceph is a distributed network storage and file system designed to provide excellent performance, reliability, and scalability.  Ceph is based on a reliable and scalable distributed object store, with a distributed metadata management cluster layered on top to provide a distributed file system with POSIX semantics.  There are a variety of ways to interact with the system:

  • Distributed file system.  An dynamic cluster of metadata servers create and manage and file system hierarchy, providing POSIX file system access via the Ceph file system client in recent Linux kernels or via a FUSE driver.
  • Object storage.  A librados library provides applications direct access to the underlying distributed object store.  Clients talk directly with storage nodes to store named blobs of data and attributes, while the cluster transparently handles replication and recovery internally.
  • S3-compatible storage.  A radosgw proxy server provides an S3-compatible REST interface to the distributed object storage system, allowing applications designed to work with Amazon’s S3 service to use a private installation of Ceph instead.
  • Rados block device (RBD).  The RBD driver provides a shared network block device via a Linux kernel block device driver (2.6.37+) or a Qemu/KVM storage driver based on librados.  In contrast to alternatives like iSCSI or AoE, RBD images are striped and replicated across the Ceph object storage cluster, providing reliable, scalable, and thinly provisioned access to block storage.  RBD supports read-only snapshots with rollback.

Ceph fills two significant gaps in the array of currently available file systems:

  1. Robust, open-source distributed storage — Ceph is released under the terms of the LGPL, which means it is free software (as in speech and beer). Ceph will provide a variety of key features that are generally lacking from existing open-source file systems, including seamless scalability (the ability to simply add disks to expand volumes), intelligent load balancing, and efficient, easy to use snapshot functionality.
  2. Scalability — Ceph is built from the ground up to seamlessly and gracefully scale from gigabytes to petabytes and beyond. Scalability is considered in terms of workload as well as total storage. Ceph is designed to handle workloads in which tens thousands of clients or more simultaneously access the same file, or write to the same directory–usage scenarios that bring typical enterprise storage systems to their knees.

Here are some of the key features that make Ceph different from existing file systems that you may have worked with:

  1. Seamless scaling — A Ceph filesystem can be seamlessly expanded by simply adding storage nodes (OSDs). However, unlike most existing file systems, Ceph proactively migrates data onto new devices in order to maintain a balanced distribution of data. This effectively utilizes all available resources (disk bandwidth and spindles) and avoids data hot spots (e.g., active data residing primarly on old disks while newer disks sit empty and idle).
  2. Strong reliability and fast recovery — All data in Ceph is replicated across multiple OSDs. If any OSD fails, data is automatically re-replicated to other devices. However, unlike typical RAID systems, the replicas for data on each disk are spread out among a large number of other disks, and when a disk fails, the replacement replicas are also distributed across many disks. This allows recovery to proceed in parallel (with dozens of disks copying to dozens of other disks), removing the need for explicit “spare” disks (which are effectively wasted until they are needed) and preventing a single disk from becoming a “RAID rebuild” bottleneck.
  3. Adaptive MDS — The Ceph metadata server (MDS) is designed to dynamically adapt its behavior to the current workload. As the size and popularity of the file system hierarchy changes over time, that hierarchy is dynamically redistributed among available metadata servers in order to balance load and most effectively use server resources. (In contrast, current file systems force system administrators to carve their data set into static “volumes” and assign volumes to servers. Volume sizes and workloads inevitably shift over time, forcing administrators to constantly shuffle data between servers or manually allocate new resources where they are currently needed.) Similarly, if thousands of clients suddenly access a single file or directory, that metadata is dynamically replicated across multiple servers to distribute the workload.

For more information about the underlying architecture of Ceph, please see the wiki.