v0.17
- kclient: fix multiple mds mdsmap decoding
- kclient: fix mon subscription renewal
- crush: fix crush map creation with empty buckets (occurs on larger clusters)
- osdmap: fix encoding bug (crashes kclient); make kclient not crash
- msgr: simplified policy, failure model
- mon: less push, more pull
- mon: request routing
- mon cluster expansion
- osd: fix pg parsing, restarts on larger clusters

v0.18
- osd: basic ENOSPC handling
- big endian fixes (required protocol/disk format change)
- osd: improved object -> pg hash function; selectable
- crush: selectable hash function(s)
- mds restart bug fixes
- kclient: mds reconnect bug fixes
- fixed mds log trimming bug
- fixed mds cap vs snap deadlock
- filestore: faster flushing
- uclient,kclient: snapshot fixes
- mds: fix recursive accounting bug
- uclient: fixes for 32bit clients
- auth: 'none' security framework
- mon: "safely" bail on write errors (e.g. ENOSPC)
- mds: fix replay/reconnect race (caused (fast) client reconnect to fail)
- mds: misc journal replay, session fixes

v0.19
- ms_dispatch fairness
- kclient: bad fsid deadlock fix
- tids in fixed msg header (protocol change)
- feature bits during connection handshake
- remove erank from ceph_entity_addr
- disk format, compat/incompat bits
- journal format improvements
- kclient: cephx
- improved truncation
- cephx: lots of fixes
- mkcephfs: cephx support
- debian: packaging fixes

v0.20
- osd: new filestore, journaling infrastructure
- msgr: wire protocol improvements (lower per-message overhead)
- mds: reduced memory utilization (still more to do!)
- mds: many single mds fixes
- mds: many clustered mds fixes
- auth: many auth_x cleanups, fixes, improvements
- kclient: many bug fixes
- librados: some cleanup, c++ api now usable

v0.21

- 

- qa: snap test.  maybe walk through 2.6.* kernel trees?
- osd: rebuild pg log
- osd: handle storage errors
- rebuild mds hierarchy
- kclient: retry alloc on ENOMEM when reading from connection?


filestore performance notes
- write ordering options
  - fs only (no journal)
  - fs, journal
  - fs + journal in parallel
  - journal sync, then fs
- and the issues
  - latency
  - effect of a btrfs hang
  - unexpected error handling (EIO, ENOSPC)
  - impact on ack, sync ordering semantics.
  - how to throttle request stream to disk io rate
  - rmw vs delayed mode

- if journal is on fs, then
  - throttling isn't an issue, but
  - fs stalls are also journal stalls

- fs only
  - latency: commits are bad.
  - hang: bad.
  - errors: could be handled, aren't
  - acks: supported
  - throttle: fs does it
  - rmw: pg toggles mode
- fs, journal
  - latency: good, unless fs hangs
  - hang: bad.  latency spikes.  overall throughput drops.
  - errors: could probably be handled, isn't.
  - acks: supported
  - throttle: btrfs does it (by hanging), which leads to a (necessary) latency spike
  - rmw: pg toggles mode
- fs | journal
  - latency: good
  - hang: no latency spike.  fs throughput may drop, to the extent btrfs throughput necessarily will.
  - errors: not detected until later.  could journal addendum record.  or die (like we do now)
  - acks: could be flexible.. maybe supported, maybe not.  will need some extra locking smarts?
  - throttle: ??
  - rmw: rmw must block on prior fs writes.
- journal, fs (writeahead)
  - latency: good (commit only, no acks)
  - hang: same as |
  - errors: same as |
  - acks: never.
  - throttle: ??
  - rmw: rmw must block on prior fs writes.
  * JourningObjectStore interface needs work?



greg
- csync data import/export tool?
- uclient: readdir from cache
- mds: basic auth checks

later
- document on-wire protocol
- client reconnect after long eviction; and slow delayed reconnect
- repair
- mds security enforcement
- client, user authentication
- cas
- osd failure declarations
- rename over old files should flush data, or revert back to old contents
- clean up SimpleMessenger interface and usage a little. Can probably unify
	some/all of shutdown, wait, destroy. Possibly move destroy into put()
	and make get/put usage more consistent/stringently mandated.

rados
- snapc interface
  - mon: allocate snapid, adjust pool_t seq
  - librados: snapc manipulation, set pool handle snapc
- make rest interface superset of s3?
  - create/delete snapshots
  - list, access snapped version
- perl swig wrapper
- 'rados call foo.bar'?
- merge pgs
- autosize pg_pools?

repair
- namespace reconstruction tool
- repair pg (rebuild log)  (online or offline?  ./cosd --repair_pg 1.ef?)
- repair file ioctl?
- are we concerned about
  - scrubbing
  - reconstruction after loss of subset of cdirs
  - reconstruction after loss of md log
- data object 
  - path backpointers?
  - parent dir pointer?
- mds scrubbing

kclient
- mdsc: preallocate reply(ies?)
- mdsc: mempool for cap writeback?
- osdc: combine request, request+reply messages into single pool.
- ENOMEM
  - message pools
  - sockets?  (this can actual generates a lockdep warning :/)
- fs-portable file layout virtual xattr (see Andreas' -fsdevel thread)
- statlite
- add cap to release if we get fouled up in fill_inode et al?
- fix up ESTALE handling
- don't retry on ENOMEM on non-nofail requests in kick_requests
- make cap import/export more efficient?
- flock, fnctl locks
- ACLs
  - init security xattrs
- should we try to ref CAP_PIN on special inodes that are open?  
- fix readdir vs fragment race by keeping a separate frag pos, and ignoring dentries below it
- inotify for updates from other clients?

vfs issues
- a getattr mask would be really nice

filestore
- make min sync interval self-tuning (ala xfs, ext3?)
- get file csum?
- clonerange writeahead journal data into objects?

btrfs
- clone compressed inline extents
- ioctl to pull out data csum?

osd
- separate reads/writes into separate op queues?
- gracefully handle ENOSPC
- gracefully handle EIO?
- what to do with lost objects.. continue peering?
- segregate backlog from log ondisk?
- preserve pg logs on disk for longer period
- make scrub interruptible
- optionally separate osd interfaces (ips) for clients and osds (replication, peering, etc.)
- pg repair
- pg split should be a work queue
- optimize remove wrt recovery pushes?

uclient
- fix client_lock vs other mutex with C_SafeCond
- clean up check_caps to more closely mirror kclient logic
- readdir from cache
- fix readdir vs fragment race by keeping a separate frag pos, and ignoring dentries below it
- hadoop: clean up assert usage

mds
- special case commit of stray dir to avoid having to commit, re-commit strays?
  - once we commit one stray, we have to re-commit later to remove it, which means we commit other new ones.
- put inode dirty fields into dirty_bits_t to reduce per-inode memory footprint
- don't sync log on every clientreplay request?
- pass issued, wanted into eval(lock) when eval() already has it?  (and otherwise optimize eval paths..)
- add an up:shadow mode?
  - tail the mds log as it is written
  - periodically check head so that we trim, too
- handle slow client reconnect (i.e. after mds has gone active)
- anchor_destroy needs to xlock linklock.. which means it needs a Mutation wrapper?
  - ... when it gets a caller.. someday..
- add FILE_CAP_EXTEND capability bit
- dir fragment
  - maybe just take dftlock for now, to keep it simple.
- dir merge
- snap
  - hard link backpointers
    - anchor source dir
    - build snaprealm for any hardlinked file
    - include snaps for all (primary+remote) parents
  - how do we properly clean up inodes when doing a snap purge?
    - when they are mid-recover?  see 136470cf7ca876febf68a2b0610fa3bb77ad3532
  - what if a recovery is queued, or in progress, and the inode is then cowed?  can that happen?  
  - proper handling of cache expire messages during rejoin phase?
    -> i think cache expires are fine; the rejoin_ack handler just has to behave if rejoining items go missing

- clustered
  - on replay, but dirty scatter replicas on lists so that they get flushed?  or does rejoin handle that?
  - linkage vs cdentry replicas and remote rename....
  - rename: importing inode... also journal imported client map?

mon
- don't allow lpg_num expansion and osd addition at the same time?
- how to shrink cluster?
- how to tell osd to cleanly shut down
- paxos need to clean up old states.
  - default: simple max of (state count, min age), so that we have at least N hours of history, say?
  - osd map: trim only old maps < oldest "in" osd up_from

osdmon
- monitor needs to monitor some osds...

pgmon
- check for orphan pgs
- monitor pg states, notify on out?
- watch osd utilization; adjust overload in cluster map

crush
- allow forcefeed for more complicated rule structures.  (e.g. make force_stack a list< set<int> >)

simplemessenger
- close idle connections?

objectcacher
- read locks?
- maintain more explicit inode grouping instead of wonky hashes

cas
- chunking.  see TTTD in
   ESHGHI, K.
   A framework for analyzing and improving content-based chunking algorithms.
   Tech. Rep. HPL-2005-30(R.1), Hewlett Packard Laboratories, Palo Alto, 2005. 

radosgw
 - handle gracefully location related requests
 - logging control (?)
 - parse date/time better
 - upload using post
 - torrent
 - handle gracefully PUT/GET requestPayment



-- for nicer kclient debug output (everything but messenger, but including msg in/out)
echo 'module ceph +p' > /sys/kernel/debug/dynamic_debug/control ; echo 'file fs/ceph/messenger.c -p' > /sys/kernel/debug/dynamic_debug/control ; echo 'file ' `grep -- --- /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` +p  > /sys/kernel/debug/dynamic_debug/control ; echo 'file ' `grep -- === /sys/kernel/debug/dynamic_debug/control | grep ceph | awk '{print $1}' | sed 's/:/ line /'` +p  > /sys/kernel/debug/dynamic_debug/control
