Details

    • Type: Sub-task Sub-task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Create a benchmark for append within the framework of TestDFSIO.

      1. appendDFSIO.patch
        7 kB
        Konstantin Shvachko
      2. appendDFSIO.patch
        3 kB
        Konstantin Shvachko
      3. appendDFSIO-MR.patch
        7 kB
        Konstantin Shvachko

        Issue Links

          Activity

          Hide
          Konstantin Shvachko added a comment -

          This adds append benchmark to TestDFSIO.

          Show
          Konstantin Shvachko added a comment - This adds append benchmark to TestDFSIO.
          Hide
          Konstantin Boudnik added a comment -

          +1

          Show
          Konstantin Boudnik added a comment - +1
          Hide
          Konstantin Shvachko added a comment -

          I ran test-core and test-patch against branch 0.21. Both successful.

          .    [exec] There appear to be 108 release audit warnings before the patch and 108 release audit warnings after applying the patch.
               [exec] +1 overall.  
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          BUILD SUCCESSFUL
          Total time: 13 minutes 33 seconds
          
          Show
          Konstantin Shvachko added a comment - I ran test-core and test-patch against branch 0.21. Both successful. . [exec] There appear to be 108 release audit warnings before the patch and 108 release audit warnings after applying the patch. [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== BUILD SUCCESSFUL Total time: 13 minutes 33 seconds
          Hide
          Konstantin Shvachko added a comment -

          Not sure how to commit it to the current (trunk) code base.

          Show
          Konstantin Shvachko added a comment - Not sure how to commit it to the current (trunk) code base.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12420873/appendDFSIO.patch
          against trunk revision 822153.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          -1 patch. The patch command could not apply the patch.

          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/55/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12420873/appendDFSIO.patch against trunk revision 822153. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/55/console This message is automatically generated.
          Hide
          Konstantin Shvachko added a comment -

          Hudson is trying to apply this to hdfs trunk, which does not have this code after HDFS-641. So since there were no objections from folks in charge of HDFS-641 I am commiting this to 0.21 and HDFS-641 branches.

          Show
          Konstantin Shvachko added a comment - Hudson is trying to apply this to hdfs trunk, which does not have this code after HDFS-641 . So since there were no objections from folks in charge of HDFS-641 I am commiting this to 0.21 and HDFS-641 branches.
          Hide
          Konstantin Shvachko added a comment -

          Two separate patches for 0.21 and HDFS-641 branches.
          Some changes:

          • I made testIO() run on MiniDFSCluster. So now it is really tested on hdfs instead of local file system.
          • Changed usage.
          • Turned on append in the configuration for the test.
          • testIO() prints stats after each of the three jobs.
          Show
          Konstantin Shvachko added a comment - Two separate patches for 0.21 and HDFS-641 branches. Some changes: I made testIO() run on MiniDFSCluster. So now it is really tested on hdfs instead of local file system. Changed usage. Turned on append in the configuration for the test. testIO() prints stats after each of the three jobs.
          Hide
          Konstantin Shvachko added a comment -

          Re-ran test-patch for both branches. No new warnings. Attaching results for the MR branch:

               [exec] There appear to be 169 release audit warnings before the patch and 169 release audit warnings after applying the patch.
               [exec] +1 overall.  
               [exec]     +1 @author.  The patch does not contain any @author tags.
               [exec]     +1 tests included.  The patch appears to include 3 new or modified tests.
               [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
               [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
               [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
               [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
               [exec] ======================================================================
               [exec] ======================================================================
               [exec]     Finished build.
               [exec] ======================================================================
               [exec] ======================================================================
          BUILD SUCCESSFUL
          Total time: 23 minutes 5 seconds
          
          Show
          Konstantin Shvachko added a comment - Re-ran test-patch for both branches. No new warnings. Attaching results for the MR branch: [exec] There appear to be 169 release audit warnings before the patch and 169 release audit warnings after applying the patch. [exec] +1 overall. [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== BUILD SUCCESSFUL Total time: 23 minutes 5 seconds
          Hide
          Konstantin Shvachko added a comment -

          I just committed this.

          Show
          Konstantin Shvachko added a comment - I just committed this.

            People

            • Assignee:
              Konstantin Shvachko
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development