Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1435

Provide an option to store fsimage compressed


    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      This provides an option to store fsimage compressed. The layout version is bumped to -25. The user could configure if s/he wants the fsimage to be compressed or not and which codec to use. By default the fsimage is not compressed.


      Our HDFS has fsimage as big as 20G bytes. It consumes a lot of network bandwidth when secondary NN uploads a new fsimage to primary NN.

      If we could store fsimage compressed, the problem could be greatly alleviated.

      I plan to provide a new configuration hdfs.image.compressed with a default value of false. If it is set to be true, fsimage is stored as compressed.

      The fsimage will have a new layout with a new field "compressed" in its header, indicating if the namespace is stored compressed or not.


        1. trunkImageCompress4.patch
          17 kB
          Hairong Kuang
        2. trunkImageCompress3.patch
          17 kB
          Hairong Kuang
        3. trunkImageCompress2.patch
          17 kB
          Hairong Kuang
        4. checkpoint-limitandcompress.patch
          13 kB
          Yilei Lu
        5. trunkImageCompress1.patch
          16 kB
          Hairong Kuang
        6. trunkImageCompress.patch
          14 kB
          Hairong Kuang

          Issue Links



              • Assignee:
                hairong Hairong Kuang
                hairong Hairong Kuang
              • Votes:
                0 Vote for this issue
                13 Start watching this issue


                • Created: