Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1435

Provide an option to store fsimage compressed

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.

      Description

      Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.

      If we could store fsimage compressed, the problem could be greatly alleviated.

      I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.

      The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.

      1. trunkImageCompress4.patch
        17 kB
        Hairong Kuang
      2. trunkImageCompress3.patch
        17 kB
        Hairong Kuang
      3. trunkImageCompress2.patch
        17 kB
        Hairong Kuang
      4. trunkImageCompress1.patch
        16 kB
        Hairong Kuang
      5. trunkImageCompress.patch
        14 kB
        Hairong Kuang
      6. checkpoint-limitandcompress.patch
        13 kB
        Yilei Lu

        Issue Links

          Activity

          Harsh J made changes -
          Link This issue incorporates HDFS-1614 [ HDFS-1614 ]
          Konstantin Shvachko made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Jeff Hammerbacher made changes -
          Link This issue relates to HDFS-1465 [ HDFS-1465 ]
          Hairong Kuang made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags [Incompatible change, Reviewed]
          Release Note This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.
          Resolution Fixed [ 1 ]
          Hairong Kuang made changes -
          Attachment trunkImageCompress4.patch [ 12457865 ]
          Hairong Kuang made changes -
          Attachment trunkImageCompress3.patch [ 12457584 ]
          Hairong Kuang made changes -
          Attachment trunkImageCompress2.patch [ 12457279 ]
          Hairong Kuang made changes -
          Link This issue is related to HDFS-1457 [ HDFS-1457 ]
          Hairong Kuang made changes -
          Link This issue is related to HDFS-1458 [ HDFS-1458 ]
          Yilei Lu made changes -
          Attachment checkpoint-limitandcompress.patch [ 12457154 ]
          Hairong Kuang made changes -
          Attachment trunkImageCompress1.patch [ 12457112 ]
          Hairong Kuang made changes -
          Attachment trunkImageCompress.patch [ 12456902 ]
          Hairong Kuang made changes -
          Field Original Value New Value
          Link This issue is blocked by HADOOP-6996 [ HADOOP-6996 ]
          Hairong Kuang created issue -

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development