Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5068

Convert NNThroughputBenchmark to a Tool to allow generic options.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: benchmarks
    • Labels:
      None
    • Target Version/s:

      Description

      Currently NNThroughputBenchmark does not recognize generic options like -conf, etc. A simple way to enable such functionality is to make it implement Tool interface.

      1. NNThBenchTool.patch
        5 kB
        Konstantin Shvachko
      2. NNThBenchTool.patch
        6 kB
        Konstantin Shvachko

        Activity

        Hide
        hadoopqa Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12596165/NNThBenchTool.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 1 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4767//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4767//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12596165/NNThBenchTool.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . The javadoc tool did not generate any warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/4767//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4767//console This message is automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Hi Konstantin! Could you please point me to documentation / tell me how to actually run this tool in order to test the changes?

        Show
        raviprak Ravi Prakash added a comment - Hi Konstantin! Could you please point me to documentation / tell me how to actually run this tool in order to test the changes?
        Hide
        shv Konstantin Shvachko added a comment -

        I usually run it using hadoop command:

        hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark
        

        Then you will see available operations, options.
        You can about background and motivation in the introductory Jira HADOOP-2149. The list of operations was subsequently extended, most of the documentation is just the JavaDoc for the class and related operations.
        To test this change you can add some generic options to the hadoop command. E.g. if I want to point it to a specific config file I'd run:

        hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -conf /home/shv/conf/my-hdfs-site.xml -op create -threads 200 -files 500000 -filesPerDir 100
        

        LMK if it works for you.

        Show
        shv Konstantin Shvachko added a comment - I usually run it using hadoop command: hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark Then you will see available operations, options. You can about background and motivation in the introductory Jira HADOOP-2149 . The list of operations was subsequently extended, most of the documentation is just the JavaDoc for the class and related operations. To test this change you can add some generic options to the hadoop command. E.g. if I want to point it to a specific config file I'd run: hadoop org.apache.hadoop.hdfs.server.namenode.NNThroughputBenchmark -conf /home/shv/conf/my-hdfs-site.xml -op create -threads 200 -files 500000 -filesPerDir 100 LMK if it works for you.
        Hide
        raviprak Ravi Prakash added a comment -

        +1. LGTM. Thanks Konstantin!

        Just fyi, for other people trying this test, I had to set dfs.namenode.fs-limits.min-block-size to 16 in my hdfs-site.xml because BLOCK_SIZE = 16; in the code, otherwise the test would hang indefinitely.

        Show
        raviprak Ravi Prakash added a comment - +1. LGTM. Thanks Konstantin! Just fyi, for other people trying this test, I had to set dfs.namenode.fs-limits.min-block-size to 16 in my hdfs-site.xml because BLOCK_SIZE = 16; in the code, otherwise the test would hang indefinitely.
        Hide
        shv Konstantin Shvachko added a comment -

        Ravi you got a good point. I was running in the test environment, which turns off minimum block size verification automatically. The new patch sets DFS_NAMENODE_MIN_BLOCK_SIZE_KEY = 0 for the benchmark.

        Show
        shv Konstantin Shvachko added a comment - Ravi you got a good point. I was running in the test environment, which turns off minimum block size verification automatically. The new patch sets DFS_NAMENODE_MIN_BLOCK_SIZE_KEY = 0 for the benchmark.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12597531/NNThBenchTool.patch
        against trunk revision .

        -1 patch. Trunk compilation may be broken.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4804//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597531/NNThBenchTool.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4804//console This message is automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12597531/NNThBenchTool.patch
        against trunk revision .

        -1 patch. Trunk compilation may be broken.

        Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4806//console

        This message is automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12597531/NNThBenchTool.patch against trunk revision . -1 patch . Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/4806//console This message is automatically generated.
        Hide
        raviprak Ravi Prakash added a comment -

        Thanks Konstantin! My +1 stands!

        Show
        raviprak Ravi Prakash added a comment - Thanks Konstantin! My +1 stands!
        Hide
        shv Konstantin Shvachko added a comment -

        Committed this to trunk and branch-2.3. Will move it further down if requested.

        Show
        shv Konstantin Shvachko added a comment - Committed this to trunk and branch-2.3. Will move it further down if requested.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-trunk-Commit #4262 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4262/)
        HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #4262 (See https://builds.apache.org/job/Hadoop-trunk-Commit/4262/ ) HDFS-5068 . Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/302/)
        HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #302 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/302/ ) HDFS-5068 . Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/)
        HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1492 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1492/ ) HDFS-5068 . Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/)
        HDFS-5068. Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114)

        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1519 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1519/ ) HDFS-5068 . Convert NNThroughputBenchmark to a Tool to allow generic options. Contributed by Konstantin Shvachko. (shv: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1514114 ) /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/NNThroughputBenchmark.java
        Hide
        shv Konstantin Shvachko added a comment -

        Merged into branch 2.3

        Show
        shv Konstantin Shvachko added a comment - Merged into branch 2.3

          People

          • Assignee:
            shv Konstantin Shvachko
            Reporter:
            shv Konstantin Shvachko
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development