Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6729

Accurately compute the test execute time in DFSIO

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput as the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future.

      public static void testWrite() throws Exception {
          FileSystem fs = cluster.getFileSystem();
          long tStart = System.currentTimeMillis();
          bench.writeTest(fs); // this line of code will cause extra time consumption as fs.delete(*,*) by the writeTest method
          long execTime = System.currentTimeMillis() - tStart;
          bench.analyzeResult(fs, TestType.TEST_TYPE_WRITE, execTime);
        }
      
      private void writeTest(FileSystem fs) throws IOException {
        Path writeDir = getWriteDir(config);
        fs.delete(getDataDir(config), true);
        fs.delete(writeDir, true);    
        runIOTest(WriteMapper.class, writeDir);
        }
      

       

      https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java

      1. MAPREDUCE-6729.001.patch
        6 kB
        mingleizhang
      2. MAPREDUCE-6729.002.patch
        6 kB
        Kai Zheng

        Issue Links

          Activity

          Hide
          mingleizhang mingleizhang added a comment -

          Is there anyone wanna get this jira ? If not, I will work on this soon.

          Show
          mingleizhang mingleizhang added a comment - Is there anyone wanna get this jira ? If not, I will work on this soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhangminglei opened a pull request:

          https://github.com/apache/hadoop/pull/111

          MAPREDUCE-6729. Hitting performance and error when lots of files to w…

          …rite or read.

          When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput while the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhangminglei/hadoop trunk

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/111.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #111


          commit 380cea926aa7219c4e8444692957f6d1b04c9849
          Author: zhangminglei <zml13856086071@163.com>
          Date: 2016-07-07T09:00:16Z

          MAPREDUCE-6729. Hitting performance and error when lots of files to write or read.

          When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput while the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhangminglei opened a pull request: https://github.com/apache/hadoop/pull/111 MAPREDUCE-6729 . Hitting performance and error when lots of files to w… …rite or read. When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput while the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhangminglei/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/111.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #111 commit 380cea926aa7219c4e8444692957f6d1b04c9849 Author: zhangminglei <zml13856086071@163.com> Date: 2016-07-07T09:00:16Z MAPREDUCE-6729 . Hitting performance and error when lots of files to write or read. When doing DFSIO test as a distributed i/o benchmark tool. Then especially writes plenty of files to disk or read from, both can cause performance issue and imprecise value in a way. The question is that existing practices needs to delete files when before running a job and that will cause extra time consumption and furthermore cause performance issue, statistical time error and imprecise throughput while the files are lots of. So we need to replace or improve this hack to prevent this from happening in the future.
          Hide
          drankye Kai Zheng added a comment -

          Thanks mingleizhang for the reporting and contribution! It sounds good.

          Tsuyoshi Ozawa, could you help take a look? This was found in a benchmark test we performed some time ago. Thanks!

          Show
          drankye Kai Zheng added a comment - Thanks mingleizhang for the reporting and contribution! It sounds good. Tsuyoshi Ozawa , could you help take a look? This was found in a benchmark test we performed some time ago. Thanks!
          Hide
          mingleizhang mingleizhang added a comment -

          Thanks Kai Zheng for review this code.

          Show
          mingleizhang mingleizhang added a comment - Thanks Kai Zheng for review this code.
          Hide
          drankye Kai Zheng added a comment -

          mingleizhang,

          Would you please rename your patch like MAPREDUCE-6729-v1.patch and then submit it to trigger the Jenkins test?

          Show
          drankye Kai Zheng added a comment - mingleizhang , Would you please rename your patch like MAPREDUCE-6729 -v1.patch and then submit it to trigger the Jenkins test?
          Hide
          mingleizhang mingleizhang added a comment -

          Thanks Kai Zheng for the useful comment. And I have renamed the patch name just now.

          Show
          mingleizhang mingleizhang added a comment - Thanks Kai Zheng for the useful comment. And I have renamed the patch name just now.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 26s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 6m 12s trunk passed
          +1 compile 0m 22s trunk passed
          +1 checkstyle 0m 15s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 23s trunk passed
          +1 javadoc 0m 12s trunk passed
          -1 mvninstall 0m 12s hadoop-mapreduce-client-jobclient in the patch failed.
          -1 compile 0m 12s hadoop-mapreduce-client-jobclient in the patch failed.
          -1 javac 0m 12s hadoop-mapreduce-client-jobclient in the patch failed.
          -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 0 unchanged - 50 fixed = 1 total (was 50)
          -1 mvnsite 0m 12s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 mvneclipse 0m 11s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          -1 findbugs 0m 10s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 javadoc 0m 10s the patch passed
          -1 unit 0m 12s hadoop-mapreduce-client-jobclient in the patch failed.
          +1 asflicense 0m 14s The patch does not generate ASF License warnings.
          10m 51s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux c5a318ee3b80 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a3f93be
          Default Java 1.8.0_91
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 6m 12s trunk passed +1 compile 0m 22s trunk passed +1 checkstyle 0m 15s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 23s trunk passed +1 javadoc 0m 12s trunk passed -1 mvninstall 0m 12s hadoop-mapreduce-client-jobclient in the patch failed. -1 compile 0m 12s hadoop-mapreduce-client-jobclient in the patch failed. -1 javac 0m 12s hadoop-mapreduce-client-jobclient in the patch failed. -1 checkstyle 0m 12s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient: The patch generated 1 new + 0 unchanged - 50 fixed = 1 total (was 50) -1 mvnsite 0m 12s hadoop-mapreduce-client-jobclient in the patch failed. +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 findbugs 0m 10s hadoop-mapreduce-client-jobclient in the patch failed. +1 javadoc 0m 10s the patch passed -1 unit 0m 12s hadoop-mapreduce-client-jobclient in the patch failed. +1 asflicense 0m 14s The patch does not generate ASF License warnings. 10m 51s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux c5a318ee3b80 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a3f93be Default Java 1.8.0_91 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt compile https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt javac https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-compile-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt checkstyle https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt mvnsite https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6596/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6598/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6598/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6599/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6599/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 5s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6600/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 5s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6600/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Kai Zheng Sure, I will take a look.

          mingleizhang I think the patch cannot be applied because the patch is generated at not root dir of Hadoop source code tree.

          .../test/java/org/apache/hadoop/fs/TestDFSIO.java | 54 +++++++++++-----------

          Please generate the patch on the root directory of hadoop source code tree by using git diff command:

          ~workplace/hadoop$ git diff --no-prefix (latest commit) > MAPREDUCE-6729.001.patch
          
          Show
          ozawa Tsuyoshi Ozawa added a comment - Kai Zheng Sure, I will take a look. mingleizhang I think the patch cannot be applied because the patch is generated at not root dir of Hadoop source code tree. .../test/java/org/apache/hadoop/fs/TestDFSIO.java | 54 +++++++++++----------- Please generate the patch on the root directory of hadoop source code tree by using git diff command: ~workplace/hadoop$ git diff --no-prefix (latest commit) > MAPREDUCE-6729.001.patch
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          FYI, how to contribute page on Hadoop wiki is also useful. http://wiki.apache.org/hadoop/HowToContribute

          Show
          ozawa Tsuyoshi Ozawa added a comment - FYI, how to contribute page on Hadoop wiki is also useful. http://wiki.apache.org/hadoop/HowToContribute
          Hide
          mingleizhang mingleizhang added a comment -

          Thanks Tsuyoshi Ozawa for review and I will try it soon.

          Show
          mingleizhang mingleizhang added a comment - Thanks Tsuyoshi Ozawa for review and I will try it soon.
          Hide
          mingleizhang mingleizhang added a comment -

          I am always happy to help.

          Show
          mingleizhang mingleizhang added a comment - I am always happy to help.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 7s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6601/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 7s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6601/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/111
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6602/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s MAPREDUCE-6729 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/111 Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6602/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          ozawa Tsuyoshi Ozawa added a comment -

          Ah, I got why BOT fails to fetch your change: the BOT tries to fetch from your PR, instead of patch file itself.

          I found that you need to rebase your change on trunk. Please push it to your github branch.
          https://github.com/apache/hadoop/pull/111/commits

          Show
          ozawa Tsuyoshi Ozawa added a comment - Ah, I got why BOT fails to fetch your change: the BOT tries to fetch from your PR, instead of patch file itself. I found that you need to rebase your change on trunk. Please push it to your github branch. https://github.com/apache/hadoop/pull/111/commits
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zhangminglei opened a pull request:

          https://github.com/apache/hadoop/pull/112

          MAPREDUCE-6729. Accurately compute the test execute time in DFSIO

          Update github-side PR to works well.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zhangminglei/hadoop trunk

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/hadoop/pull/112.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #112


          commit 2a295d0a1e80df0f9153b7600ff3f38b7c3faee5
          Author: zhangminglei <zml13856086071@163.com>
          Date: 2016-07-08T03:29:04Z

          MAPREDUCE-6729. Accurately compute the test execute time in DFSIO


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zhangminglei opened a pull request: https://github.com/apache/hadoop/pull/112 MAPREDUCE-6729 . Accurately compute the test execute time in DFSIO Update github-side PR to works well. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zhangminglei/hadoop trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/hadoop/pull/112.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #112 commit 2a295d0a1e80df0f9153b7600ff3f38b7c3faee5 Author: zhangminglei <zml13856086071@163.com> Date: 2016-07-08T03:29:04Z MAPREDUCE-6729 . Accurately compute the test execute time in DFSIO
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user zhangminglei closed the pull request at:

          https://github.com/apache/hadoop/pull/111

          Show
          githubbot ASF GitHub Bot added a comment - Github user zhangminglei closed the pull request at: https://github.com/apache/hadoop/pull/111
          Hide
          drankye Kai Zheng added a comment -

          Re-uploaded the same patch to trigger the building.

          Show
          drankye Kai Zheng added a comment - Re-uploaded the same patch to trigger the building.
          Hide
          hadoopqa Hadoop QA added a comment -
          +1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 32s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 33s trunk passed
          +1 compile 0m 54s trunk passed
          +1 checkstyle 0m 17s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 mvneclipse 0m 16s trunk passed
          +1 findbugs 0m 25s trunk passed
          +1 javadoc 0m 12s trunk passed
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 22s the patch passed
          +1 javac 0m 22s the patch passed
          +1 checkstyle 0m 15s the patch passed
          +1 mvnsite 0m 28s the patch passed
          +1 mvneclipse 0m 14s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 0m 29s the patch passed
          +1 javadoc 0m 11s the patch passed
          +1 unit 126m 0s hadoop-mapreduce-client-jobclient in the patch passed.
          +1 asflicense 0m 24s The patch does not generate ASF License warnings.
          141m 3s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Issue MAPREDUCE-6729
          GITHUB PR https://github.com/apache/hadoop/pull/112
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4027af10b870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d5f5a64
          Default Java 1.8.0_91
          findbugs v3.0.0
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/testReport/
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 32s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 33s trunk passed +1 compile 0m 54s trunk passed +1 checkstyle 0m 17s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 16s trunk passed +1 findbugs 0m 25s trunk passed +1 javadoc 0m 12s trunk passed +1 mvninstall 0m 22s the patch passed +1 compile 0m 22s the patch passed +1 javac 0m 22s the patch passed +1 checkstyle 0m 15s the patch passed +1 mvnsite 0m 28s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 0m 29s the patch passed +1 javadoc 0m 11s the patch passed +1 unit 126m 0s hadoop-mapreduce-client-jobclient in the patch passed. +1 asflicense 0m 24s The patch does not generate ASF License warnings. 141m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Issue MAPREDUCE-6729 GITHUB PR https://github.com/apache/hadoop/pull/112 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4027af10b870 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d5f5a64 Default Java 1.8.0_91 findbugs v3.0.0 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6612/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          mingleizhang mingleizhang added a comment -

          Thanks to Kai's help. And I've done this.

          Show
          mingleizhang mingleizhang added a comment - Thanks to Kai's help. And I've done this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          LGTM, +1. I'll commit this tomorrow JST if there is no objection.

          Show
          ajisakaa Akira Ajisaka added a comment - LGTM, +1. I'll commit this tomorrow JST if there is no objection.
          Hide
          mingleizhang mingleizhang added a comment -

          Akira Ajisaka Thanks for your review and I am looking forward this commit coming soon.

          Show
          mingleizhang mingleizhang added a comment - Akira Ajisaka Thanks for your review and I am looking forward this commit coming soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/hadoop/pull/112

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/hadoop/pull/112
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, and branch-2.8. Thanks mingleizhang for the contribution and thanks Tsuyoshi Ozawa and Kai Zheng for the reviews.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, and branch-2.8. Thanks mingleizhang for the contribution and thanks Tsuyoshi Ozawa and Kai Zheng for the reviews.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-trunk-Commit #10181 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10181/)
          MAPREDUCE-6729. Accurately compute the test execute time in DFSIO. (aajisaka: rev ce93595d7a18bd0561ae2c1983b51f6aba887179)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-trunk-Commit #10181 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10181/ ) MAPREDUCE-6729 . Accurately compute the test execute time in DFSIO. (aajisaka: rev ce93595d7a18bd0561ae2c1983b51f6aba887179) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/fs/TestDFSIO.java

            People

            • Assignee:
              mingleizhang mingleizhang
              Reporter:
              mingleizhang mingleizhang
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development