Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-1114

Reducing NameNode memory usage by an alternate hash table

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.22.0
    • Component/s: namenode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      NameNode uses a java.util.HashMap to store BlockInfo objects. When there are many blocks in HDFS, this map uses a lot of memory in the NameNode. We may optimize the memory usage by a light weight hash table implementation.

      1. benchmark20100618.patch
        4 kB
        Tsz Wo Nicholas Sze
      2. GSet20100525.pdf
        105 kB
        Tsz Wo Nicholas Sze
      3. gset20100608.pdf
        105 kB
        Tsz Wo Nicholas Sze
      4. gset20100702.pdf
        107 kB
        Tsz Wo Nicholas Sze
      5. gset20100702.tex
        14 kB
        Tsz Wo Nicholas Sze
      6. h1114_20100607.patch
        14 kB
        Tsz Wo Nicholas Sze
      7. h1114_20100614b.patch
        24 kB
        Tsz Wo Nicholas Sze
      8. h1114_20100615.patch
        25 kB
        Tsz Wo Nicholas Sze
      9. h1114_20100616b.patch
        27 kB
        Tsz Wo Nicholas Sze
      10. h1114_20100617.patch
        27 kB
        Tsz Wo Nicholas Sze
      11. h1114_20100617b_y0.20.1xx.patch
        27 kB
        Tsz Wo Nicholas Sze
      12. h1114_20100617b.patch
        27 kB
        Tsz Wo Nicholas Sze
      13. h1114_20100617b2_y0.20.1xx.patch
        27 kB
        Tsz Wo Nicholas Sze

        Activity

        Tsz Wo Nicholas Sze created issue -
        Tsz Wo Nicholas Sze made changes -
        Field Original Value New Value
        Assignee Tsz Wo (Nicholas), SZE [ szetszwo ]
        Tsz Wo Nicholas Sze made changes -
        Attachment GSet20100525.pdf [ 12445502 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100607.patch [ 12446559 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment gset20100608.pdf [ 12446606 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100614b.patch [ 12447075 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100615.patch [ 12447157 ]
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100616b.patch [ 12447296 ]
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100617.patch [ 12447361 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Open [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100617b.patch [ 12447372 ]
        Tsz Wo Nicholas Sze made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Tsz Wo Nicholas Sze made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314241 ]
        Resolution Fixed [ 1 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment benchmark20100618.patch [ 12447496 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100617b_y0.20.1xx.patch [ 12447858 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment h1114_20100617b2_y0.20.1xx.patch [ 12447970 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment gset20100702.pdf [ 12448615 ]
        Tsz Wo Nicholas Sze made changes -
        Attachment gset20100702.tex [ 12448618 ]
        Konstantin Shvachko made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Tsz Wo Nicholas Sze
            Reporter:
            Tsz Wo Nicholas Sze
          • Votes:
            0 Vote for this issue
            Watchers:
            14 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development