- Feature Name: Data/network freeze
- Status: completed
- Start Date: 2016-02-18
- Authors: Ben Darnell
- RFC PR: [#4499](https://github.com/cockroachdb/cockroach/pull/4499)
- Cockroach Issue:

# Summary

This RFC outlines the plan for freezing our data formats and network
protocols.

# Motivation

We currently make backwards-incompatible changes to data formats
without providing any means to upgrade without data loss. This will
need to stop before beta for obvious reasons.

# Detailed design

## Freeze plan

The freeze will proceed in several steps.

### Stage 0 (pre-beta)

Anything goes; changes to on-disk formats do not require any kind of
migration path.

### Stage 1: Guaranteed upgrade path (Mar 30, 2016)

In stage 1, we require backwards-compatibility with data written by
any previous stage 1 build. It should always be possible to upgrade by
stopping all of the old nodes and then bringing up the new version.
It's OK at this stage if old and new versions cannot be run
concurrently, or if the migration process takes some time (e.g.
rewriting all data before the node can start up).

It is acceptable at this stage if the process is somewhat manual (e.g.
running some sort of yet-to-be-written backup/restore process and
stopping/restarting all nodes at once). However, it is preferable if
any migrations are done automatically when a node starts up with an
old data directory.

### Stage 2: Online upgrades (Date TBD)

Beginning in stage 2, we require that any upgrade be able to be
performed without taking the cluster offline: old and new nodes must
be able to coexist. The exact date for this stage is yet to be
determined, but will be during the beta period and before 1.0.

## Affected code

Any code could potentially be affected by the freeze, but areas that
will deserve special scrutiny include:

* All `.proto` definitions
* The packages `keys` and `util/encoding`
* All system tables (defined in `sql/system.go`)

## Migration strategies

It is difficult to come up with a universal migration strategy, since
different changes will require different approaches (for example,
`.proto` changes could perhaps be made by rewriting data on disk at
startup, while changes to key construction may require the change to
be coordinated in a distributed fashion). Therefore we leave the
specifics of a migration process until the need arises.

To facilitate future changes, we will introduce version numbers at
several levels. Initially the behavior around these version numbers
will be conservative and cross-version communication will be limited.
That makes these version numbers a blunt instrument to be reserved for
major changes.

* The on-disk format (via a file that lives outside RocksDB). Servers
  will refuse to load a database with a higher version number than
  they understand.
* The network protocol (via GRPC header). Servers and clients will
  treat a higher version number than they understand as an error.
* Gossip (perhaps via a new node attribute, or field in the
  `NodeDescriptor` proto). The rebalance/allocation system will not
  choose to place a replica on a node with a different version number.
* The SQL `TableDescriptor`

# Drawbacks

After the freeze, some changes will be much harder to make.

# Alternatives

None.

# Unresolved questions

* When exactly do we begin the stage 2 freeze?
* Is there anything else worth doing at this point to facilitate
  future migrations?
* What about downgrades? It's scary to upgrade when there is no going
  back. However, supporting downgrades adds even more complexity and I
  don't think it's worth making this commitment during beta (maybe for
  1.0, though).
