Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2362

More Improvements on NameNode Scalability

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      This jira acts as an umbrella jira to track all the improvements we've done recently to improve Namenode's performance, responsiveness, and hence scalability. Those improvements include:
      1. Incremental block reports (HDFS-395)
      2. BlockManager.reportDiff optimization for processing block reports (HDFS-2477)
      3. Upgradable lock to allow simutaleous read operation while reportDiff is in progress in processing block reports (HDFS-2490)
      4. More CPU efficient data structure for under-replicated/over-replicated/invalidate blocks (HDFS-2476)
      5. Increase granularity of write operations in ReplicationMonitor thus reducing contention for write lock (HDFS-2495)
      6. Support variable block sizes
      7. Release RPC handlers while waiting for edit log is synced to disk
      8. Reduce network traffic pressure to the master rack where NN is located by lowering read priority of the replicas on the rack
      9. A standalone KeepAlive heartbeat thread
      10. Reduce Multiple traversals of path directory to one for most namespace manipulations
      11. Move logging out of write lock section.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hairong Hairong Kuang
            • Votes:
              0 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated: