Resolution: Won't Fix
Design Document for TRecordStream (this is basically the design doc circulated on the public thrift lists under the name TRobustOfflineStream in May 08 with the addition of the requirement of handling small synchronous writes)
TRecordStream is a Thrift transport that encodes data in a format
suitable for storage in a file (not synchronous communication).
TRecordStream achieves following design goals:
- Be self-describing and extensible. A file containing a TRecordStream
must contain enough metadata for an application to read it with no other
context. It should be possible to add new features without breaking
backwards and forwards compatibility. It should be possible to completely
change the format without confusing old or programs.
- Be robust against disk corruption. All data and metadata must (optionally)
be checksummed. It must be possible to recover and continue reading
uncorrupted data after corruption is encountered.
- Be (optionally) human-readable. TRecordStream will also be used for
plan-text, line-oriented, human-readable data. Allowing a plain-text,
line-oriented, human-readable header format will be advantageous for this
- Support asynchronous file I/O. This feature will not be implemented in the
first version of TRecordStream, but the implementation must support
the eventual inclusion of this feature.
- Be performant. No significant sacrifice of speed should be made in order to
achieve any of the other design goals.
- Support small synchronous writes
TRecordStream will not do any I/O itself, but will instead focus on
preparing the data format and depend on an underlying transport (TFDTransport,
for example) to write the data to a file.
TRecordStream will have two distinct formats: binary and plain text.
Binary-format streams shall begin with a format version number, encoded as a
32-bit big-endian integer. The version number must not exceed 2^24-1, so the
first byte of a TRecordStream will always be 0. The version number
shall be repeated once to guard against corruption. If the two copies of the
version number do not match, the stream must be considered corrupt, and
recovery should proceed as described below (TODO).
Plain-text streams shall begin with the string ASCII "TROS: " (that is a space
after the colon), followed by the decimal form of the version number
(ASCII-encoded), followed by a linefeed (ASCII 0x0a) character. The full
version line shall be repeated.
This document describes version 1 of the format. Version 1 streams are
composed of series of chunks. Variable-length chunks are supported, but their
use is discoraged because they make recovering from corrupt chunk headers
difficult. Each chunk begins with the redundant version identifiers described
Following the version numbers, a binary-format stream shall contain the
following fields, in order and with no padding:
- The (32-bit) CRC-32 of the header length + header data.
- The 32-bit big endian header length.
- A variable-length header, which is a TBinaryProtocol-serialized Thrift
structure (whose exact structure is defined in
A plain-text stream should follow the versions with:
- The string "Header-Checksum: "
- The eight-character (leading-zero-padded) hexadecimal encoding of the
unsigned CRC-32 of the header (which does not include the CRC-32).
- A linefeed (0x0a).
- A header consisting of zero or more entries, where each entry consists of
- An entry name, which is an ASCII string consisting of alphanumeric
characters, dashes ("-"), underscores, and periods (full-stops).
- A colon followed by a space.
- An entry value, which is a printable ASCII string not including any
- A linefeed.
- A linefeed.
Header entry names may be repeated. The handling of repeated names is
dependent on the particular name. Unless otherwise specified, all entries
with a given name other than the last are ignored.
The actual data will be stored in sub-chunks, which may optionally be
compressed. (The chunk header will define the compression format used.) The
chunk header will specify the following fields for each sub-chunk:
- (optional) Offset within the chunk. If ommitted, it should be assumed to
immediately follow the previous sub-chunk.
- (required) Length of the (optionally) compressed sub-chunk. This is the
physical number of bytes in the stream taken up by the sub-cunk.
- (optional) Uncompressed length of the sub-chunk. Used as an optimization
- (optional) CRC-32 of the (optionally compressed) sub-chunk.
- (optional) CRC-32 of the uncompressed sub-chunk.
If no compression format is specified, the sub-chunks should be assumed to be
in "raw" format.