Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10314

Update index file format

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 3.0.0 rc1
    • Component/s: None
    • Labels:
      None

      Description

      As CASSANDRA-9738 may not make it into 3.0rc1, but having an off-heap key-cache is still a goal, we should change the index file format to meet off-heap requirements (so I've set fixver to 3.0rc1).

      Off-heap (and mmap'd index files) need the offsets of the individual IndexInfo objects and the at least the offset field of IndexInfo structures.

      The format I propose is as follows:

       (long) position (vint since 3.0, 64bit before)
        (int) serialized size of data that follows (vint since 3.0, 32bit before)
       -- following for indexed entries only (so serialized size > 0)
       (long) header-length (vint since 3.0)
        (int) DeletionTime.localDeletionTime (32 bit int)
       (long) DeletionTime.markedForDeletionAt (64 bit long)
        (int) number of IndexInfo objects (vint since 3.0, 32bit before)
          (*) serialized IndexInfo objects, see below
          (*) offsets of serialized IndexInfo objects, since version "ma" (3.0)
              Each IndexInfo object's offset is relative to the first IndexInfo object.
      
          (*) IndexInfo.firstName (ClusteringPrefix serializer, either Clustering.serializer.serialize or Slice.Bound.serializer.serialize)
          (*) IndexInfo.lastName (ClusteringPrefix serializer, either Clustering.serializer.serialize or Slice.Bound.serializer.serialize)
       (long) IndexInfo.offset (vint encoded since 3.0, 64bit int before)
       (long) IndexInfo.width (vint encoded since 3.0, 64bit int before)
       (bool) IndexInfo.endOpenMarker != null              (if 3.0)
        (int) IndexInfo.endOpenMarker.localDeletionTime    (if 3.0 && IndexInfo.endOpenMarker != null)
       (long) IndexInfo.endOpenMarker.markedForDeletionAt  (if 3.0 && IndexInfo.endOpenMarker != null)
      

      Regarding the IndexInfo.offset and .width fields there are two options.

      • Serialize both of them or
      • Serialize only the offset field plus a last byte offset to be able to recalculate the width of the last IndexInfo

      The first option is probably the simpler one, the second saves a few bytes (those of the vint encoded width).

      EDIT: update vint fields (as per CASSANDRA-10232)
      EDIT2: add header-length fields (as per CASSANDRA-10232)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                snazy Robert Stupp
                Reporter:
                snazy Robert Stupp
                Authors:
                Robert Stupp
                Reviewers:
                Ariel Weisberg
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: