Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-10677 Über-jira: Enhancements to NNThroughputBenchmark tool
  3. HDFS-9379

Make NNThroughputBenchmark$BlockReportStats support more than 10 datanodes

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently, the NNThroughputBenchmark test BlockReportStats relies on sorted datanodes array in the lexicographical order of datanode's xferAddr.

      • There is an assertion of datanode's xferAddr lexicographical order when filling the datanodes, see the code.
      • When searching the datanode by DatanodeInfo, it uses binary search against the datanodes array, see the code

      In DatanodeID, the xferAddr is defined as host:port. In NNThroughputBenchmark, the port is simply the index of the tiny datanode plus one.

      The problem here is that, when there are more than 9 tiny datanodes (numThreads), the lexicographical order of datanode's xferAddr will be invalid as the string value of datanode index is not in lexicographical order any more. For example,

      ...
      192.168.54.40:8
      192.168.54.40:9
      192.168.54.40:10
      192.168.54.40:11
      ...
      

      192.168.54.40:9 is greater than 192.168.54.40:10. The assertion will fail and the binary search won't work.

      The simple fix is to calculate the datanode index by port directly, instead of using binary search.

        Attachments

        1. HDFS-9379.000.patch
          2 kB
          Mingliang Liu

          Issue Links

            Activity

              People

              • Assignee:
                liuml07 Mingliang Liu
                Reporter:
                liuml07 Mingliang Liu
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: