Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-14366

Improve HDFS append performance

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.2
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.2.1, 2.9.3, 3.1.3
    • Component/s: hdfs
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      In our HDFS cluster we observed that append operation can take as much as 10X write lock time than other write operations. By collecting flamegraph on the namenode (see attachment: append-flamegraph.png), we found that most of the append call is spent on getNumLiveDataNodes():

        /** @return the number of live datanodes. */
        public int getNumLiveDataNodes() {
          int numLive = 0;
          synchronized (this) {
            for(DatanodeDescriptor dn : datanodeMap.values()) {
              if (!isDatanodeDead(dn) ) {
                numLive++;
              }
            }
          }
          return numLive;
        }
      

      this method synchronizes on the DatanodeManager which is particularly expensive in large clusters since datanodeMap is being modified in many places such as processing DN heartbeats.

      For append operation, getNumLiveDataNodes() is invoked in isSufficientlyReplicated:

        /**
         * Check if a block is replicated to at least the minimum replication.
         */
        public boolean isSufficientlyReplicated(BlockInfo b) {
          // Compare against the lesser of the minReplication and number of live DNs.
          final int replication =
              Math.min(minReplication, getDatanodeManager().getNumLiveDataNodes());
          return countNodes(b).liveReplicas() >= replication;
        }
      

      The way that the replication is calculated is not very optimal, as it will call getNumLiveDataNodes() every time even though usually minReplication is much smaller than the latter.

       

        Attachments

        1. HDFS-14366.001.patch
          1 kB
          Chao Sun
        2. HDFS-14366.000.patch
          1 kB
          Chao Sun
        3. append-flamegraph.png
          397 kB
          Chao Sun

          Issue Links

            Activity

              People

              • Assignee:
                csun Chao Sun
                Reporter:
                csun Chao Sun
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: