Uploaded image for project: 'HBase'
  1. HBase
  2. HBASE-7987

Snapshot Manifest file instead of multiple empty files

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.99.0
    • Component/s: snapshots
    • Labels:
      None

      Description

      Currently taking a snapshot means creating one empty file for each file in the source table directory, plus copying the .regioninfo file for each region, the table descriptor file and a snapshotInfo file.

      during the restore or snapshot verification we traverse the filesystem (fs.listStatus()) to find the snapshot files, and we open the .regioninfo files to get the information.

      to avoid hammering the NameNode and having lots of empty files, we can use a manifest file that contains the list of files and information that we need.
      To keep the RS parallelism that we have, each RS can write its own manifest.

      message SnapshotDescriptor {
        required string name;
        optional string table;
        optional int64 creationTime;
        optional Type type;
        optional int32 version;
      }
      
      message SnapshotRegionManifest {
        optional int32 version;
        required RegionInfo regionInfo;
        repeated FamilyFiles familyFiles;
      
        message StoreFile {
          required string name;
          optional Reference reference;
        }
      
        message FamilyFiles {
          required bytes familyName;
          repeated StoreFile storeFiles;
        }
      }
      
      /hbase/.snapshot/<snapshotName>
      /hbase/.snapshot/<snapshotName>/snapshotInfo
      /hbase/.snapshot/<snapshotName>/<tableName>
      /hbase/.snapshot/<snapshotName>/<tableName>/tableInfo
      /hbase/.snapshot/<snapshotName>/<tableName>/regionManifest(.n)
      

        Attachments

        1. HBASE-7987-v6.patch
          533 kB
          Matteo Bertozzi
        2. HBASE-7987-v5.patch
          527 kB
          Matteo Bertozzi
        3. HBASE-7987-v4.patch
          469 kB
          Matteo Bertozzi
        4. HBASE-7987-v3.patch
          457 kB
          Matteo Bertozzi
        5. HBASE-7987-v2.patch
          449 kB
          Matteo Bertozzi
        6. HBASE-7987-v1.patch
          446 kB
          Matteo Bertozzi
        7. HBASE-7987-v0.patch
          444 kB
          Matteo Bertozzi
        8. HBASE-7987-v2.sketch
          80 kB
          Matteo Bertozzi
        9. HBASE-7987.sketch
          58 kB
          Matteo Bertozzi

          Activity

            People

            • Assignee:
              mbertozzi Matteo Bertozzi
              Reporter:
              mbertozzi Matteo Bertozzi
            • Votes:
              0 Vote for this issue
              Watchers:
              14 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: