Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-7285 Erasure Coding Support inside HDFS
  3. HDFS-8920

Erasure Coding: when recovering lost blocks, logs can be too verbose and hurt performance

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: HDFS-7285
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When we test reading data with datanodes killed, DFSInputStream::getBestNodeDNAddrPair becomes a hot spot method and effectively blocks the client JVM. This log seems too verbose:

          if (chosenNode == null) {
            DFSClient.LOG.warn("No live nodes contain block " + block.getBlock() +
                " after checking nodes = " + Arrays.toString(nodes) +
                ", ignoredNodes = " + ignoredNodes);
            return null;
          }
      

        Attachments

        1. HDFS-8920-HDFS-7285.1.patch
          4 kB
          Rui Li
        2. HDFS-8920-HDFS-7285.2.patch
          4 kB
          Rui Li

          Activity

            People

            • Assignee:
              lirui Rui Li
              Reporter:
              lirui Rui Li
            • Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: