We’ve updated the configuration framework (again) so that only a single configuration file is needed for the entire cluster.

The ceph.conf file consists of a global section, a section for each daemon type (e.g., mon, mds, osd), and a section for each daemon instance (e.g., mon0, mds.foo, osd12).  This allows you to specify options in a generic fashion where possible, using a few simple variable substitions, or in the section specific to the daemon type or daemon.  For example,

[global]

        pid file = /var/run/ceph/$name.pid

[osd]

        osd data = /data/osd$id

[osd0]

        host = node0

        debug osd = 10   ; just for this osd

[osd1]

        host = node1

and so forth. You can then distribute the file unmodified to all nodes, and on each machine the startup script will only pay attention to the daemons assignd to that host.

See the wiki for details.

Yehuda and I did some performance tuning with dbench a couple weeks back and made some significant improvements.  Here are the rough numbers, before I forget.  We were testing on a simple client/server setup to make a reasonable comparison with NFS: single server on a single SATA disk, and a single client. Since we were mainly interested in metadata latency, we were using just a single thread (’dbench 1′).

• sync NFS  ~2.5 MB/sec
• Ceph ~7 MB/sec
• local disk on server ~11 MB/sec
• async NFS ~13 MB/sec

The async NFS was presumably faster than the local disk because the fsync() (or close()) wasn’t really waiting for anything to be flushed to disk on the server.  Considering Ceph started out around 2 MB/sec two days earlier, we were pretty happy, and there’s still room for improvement.

UPDATE 7/7/09: I’m not sure what machine I was using at the time, but in retrospect the local disk numbers look way too low.  Don’t read too much into these numbers!

I’ve tagged a v0.7 release.  Probably the biggest change in this release (aside from the usual bug fixes and performance improvements) is the new start/stop and configuration framework.  Notably, the entire cluster configuration can be described by a single cluster.conf file that is shared by all nodes (distributed via scp or NFS or whatever) and used for mkfs, startup, and shutdown.

New in v0.7:

• smart osd ’sync’ behavior
• osd bug fixes
• fast truncate strategy
• improved start/stop scripts
• new cluster configuration framework

Source tarballs are at http://ceph.newdream.net/download, debian packages are at http://ceph.newdream.net/debian, and the source repository can be cloned via git.

Yehuda and I spent last week polishing his configuration framework and reworking the way everything is configured and started up.  I think the end result is pretty slick:

There are now two configuration files.  The first, cluster.conf, defines which hosts participate in the cluster, which daemons run on which hosts, and what paths are used to store data.  It is used by the /etc/init.d/ceph init script (src/init-ceph in the git tree) and mkcephfs.  The trick is that the cluster.conf defines daemon startup parameters for the entire cluster, but by default the init script only pays attention to those assigned to the local host, allowing you do distribute the same file across the cluster without adjusting it for each host.  Alternatively, the -a switch (e.g. /etc/init.d/ceph -a start) will start (or stop) daemons on all hosts via ssh.

The ceph.conf file defines runtime parameters, like debugging levels, log locations, and thread pool sizes, and so forth.  By default everything looks at /etc/ceph/ceph.conf, or you can specify a separate configuration file on a per-daemon basis via the cluster.conf.

The new framework is described in detail in the wiki.

UPDATE: Okay, we’ve since revised this scheme to use a single ceph.conf file.  Even better.