Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6363

[NNBench] Lease mismatch error when running with multiple mappers

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Command :

      ./yarn jar ../share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.0.1-tests.jar nnbench -operation create_write -numberOfFiles 1000 -blockSize 268435456 -bytesToWrite 1024000000 -baseDir /benchmarks/NNBench`hostname -s` -replicationFactorPerFile 3 -maps 100 -reduces 10

      Trace :

      013-06-21 10:44:53,763 INFO org.apache.hadoop.ipc.Server: IPC Server handler 7 on 9005, call org.apache.hadoop.hdfs.protocol.ClientProtocol.addBlock from 192.168.105.214:36320: error: org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
      org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException: Lease mismatch on /benchmarks/NNBenchlinux-185/data/file_linux-214__0 owned by DFSClient_attempt_1371782327901_0001_m_000048_0_1383437860_1 but is accessed by DFSClient_attempt_1371782327901_0001_m_000084_0_1880545303_1
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2351)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.analyzeFileState(FSNamesystem.java:2098)
      at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2019)
      at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:501)
      at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:213)
      at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:52012)
      at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:435)
      at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:925)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1710)
      at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1706)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)

      1. HDFS4929.patch
        2 kB
        Brahma Reddy Battula
      2. MAPREDUCE-6363-001.patch
        2 kB
        Brahma Reddy Battula
      3. MAPREDUCE-6363-002.patch
        3 kB
        Brahma Reddy Battula
      4. MAPREDUCE-6363-003.patch
        8 kB
        Vlad Sharanhovich
      5. MAPREDUCE-6363-004.patch
        8 kB
        Brahma Reddy Battula
      6. MAPREDUCE-6363-005.patch
        8 kB
        Vlad Sharanhovich
      7. MAPREDUCE-6363-006.patch
        9 kB
        Bibin A Chundatt
      8. MAPREDUCE-6363-007.patch
        10 kB
        Bibin A Chundatt
      9. MAPREDUCE-6363-008.patch
        10 kB
        Bibin A Chundatt
      10. nnbench.log
        18 kB
        Akira Ajisaka

        Issue Links

          Activity

          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          As file name is not unique for each mapper(same file name for each mapper),,Hence we are getting the lease mismatch error...

          Can we pass mapper ID or some unique name for following...Please can anyone suggest's on this...

          if (barrier()) {
          if (op.equals(OP_CREATE_WRITE))

          { startTimeTPmS = System.currentTimeMillis(); doCreateWriteOp("file_" + hostName + "_", reporter); }

          else if (op.equals(OP_OPEN_READ))

          { startTimeTPmS = System.currentTimeMillis(); doOpenReadOp("file_" + hostName + "_", reporter); }

          else if (op.equals(OP_RENAME))

          { startTimeTPmS = System.currentTimeMillis(); doRenameOp("file_" + hostName + "_", reporter); }

          else if (op.equals(OP_DELETE))

          { startTimeTPmS = System.currentTimeMillis(); doDeleteOp("file_" + hostName + "_", reporter); }
          Show
          brahmareddy Brahma Reddy Battula added a comment - As file name is not unique for each mapper(same file name for each mapper),,Hence we are getting the lease mismatch error... Can we pass mapper ID or some unique name for following...Please can anyone suggest's on this... if (barrier()) { if (op.equals(OP_CREATE_WRITE)) { startTimeTPmS = System.currentTimeMillis(); doCreateWriteOp("file_" + hostName + "_", reporter); } else if (op.equals(OP_OPEN_READ)) { startTimeTPmS = System.currentTimeMillis(); doOpenReadOp("file_" + hostName + "_", reporter); } else if (op.equals(OP_RENAME)) { startTimeTPmS = System.currentTimeMillis(); doRenameOp("file_" + hostName + "_", reporter); } else if (op.equals(OP_DELETE)) { startTimeTPmS = System.currentTimeMillis(); doDeleteOp("file_" + hostName + "_", reporter); }
          Hide
          haridsv Hari Krishna Dara added a comment -

          I faced this exact same issue and my workaround is to modify NNBench to encode the counter as a sequence and use that counter value as part of the filename (instead of the hostname). This gives uniqueness to the filenames and avoids multiple maps running on the same node from stepping on each other. However, this requires that the subsequent operations (read, rename and delete) are run with the exact same number of maps.

          In fact, the way NNBench works is very flawed. You are expected to first run "create_write" and then follow up with the other operations. Since the hostname is in the filename, this only works when the maps for follow up operations are guaranteed to be run on exactly the same nodes, but there is no such guarantee. There is also an issue with the maps getting swarmed on a small number of nodes, instead of getting distributed across all nodes nicely. In my testing, when I ran with 13 maps on a cluster with 13 slaves, I had all the maps allocated on two nodes only. When I want to generate a lot of load, I want to increase the number of maps, but the way it currently works, these maps are very likely to swarm up on a few nodes and when they exceed the node capacity, just queue up and fail the barrier condition. I don't know the right solution, but I worked around for my use case by setting the replication count of the control files to the same as the number of nodes so that they have an equal chance getting assigned to any node. However, this only makes a sense when the number of maps is high compared to the capacity of the cluster.

          Show
          haridsv Hari Krishna Dara added a comment - I faced this exact same issue and my workaround is to modify NNBench to encode the counter as a sequence and use that counter value as part of the filename (instead of the hostname). This gives uniqueness to the filenames and avoids multiple maps running on the same node from stepping on each other. However, this requires that the subsequent operations (read, rename and delete) are run with the exact same number of maps. In fact, the way NNBench works is very flawed. You are expected to first run "create_write" and then follow up with the other operations. Since the hostname is in the filename, this only works when the maps for follow up operations are guaranteed to be run on exactly the same nodes, but there is no such guarantee. There is also an issue with the maps getting swarmed on a small number of nodes, instead of getting distributed across all nodes nicely. In my testing, when I ran with 13 maps on a cluster with 13 slaves, I had all the maps allocated on two nodes only. When I want to generate a lot of load, I want to increase the number of maps, but the way it currently works, these maps are very likely to swarm up on a few nodes and when they exceed the node capacity, just queue up and fail the barrier condition. I don't know the right solution, but I worked around for my use case by setting the replication count of the control files to the same as the number of nodes so that they have an equal chance getting assigned to any node. However, this only makes a sense when the number of maps is high compared to the capacity of the cluster.
          Hide
          beeflyme caixiaofeng added a comment -

          I have tried as @Hari Krishna Dara said.

          Show
          beeflyme caixiaofeng added a comment - I have tried as @Hari Krishna Dara said.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Attached patch with my proposal(having unique name for each file)

          Show
          brahmareddy Brahma Reddy Battula added a comment - Attached patch with my proposal(having unique name for each file)
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12693611/HDFS4929.patch
          against trunk revision 6b17eb9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient.

          Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9292//testReport/
          Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9292//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12693611/HDFS4929.patch against trunk revision 6b17eb9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/9292//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/9292//console This message is automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Any thoughts on this issue..? Thanks..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Any thoughts on this issue..? Thanks..
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Kindly review the patch !!!

          Show
          brahmareddy Brahma Reddy Battula added a comment - Kindly review the patch !!!
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 21s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 31s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 35s There were no new checkstyle issues.
          +1 whitespace 0m 0s The patch has no lines that end in whitespace.
          +1 install 1m 38s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 2.0.3) warnings.
          -1 mapreduce tests 102m 35s Tests failed in hadoop-mapreduce-client-jobclient.
              119m 22s  



          Reason Tests
          Timed out tests org.apache.hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12693611/HDFS4929.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 3d28611
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 31s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 35s There were no new checkstyle issues. +1 whitespace 0m 0s The patch has no lines that end in whitespace. +1 install 1m 38s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 2.0.3) warnings. -1 mapreduce tests 102m 35s Tests failed in hadoop-mapreduce-client-jobclient.     119m 22s   Reason Tests Timed out tests org.apache.hadoop.mapreduce.v2.TestSpeculativeExecutionWithMRApp Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12693611/HDFS4929.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 3d28611 hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5705/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Brahma Reddy Battula for reporting and taking this issue.

          doCreateWriteOp("file_" + hostName + "_" + Math.random(), reporter);
          

          hostname + "_" + Math.random() is not sufficient because there's no guarantee to open_create files and open_read/rename/delete the same files, as Hari Krishna Dara mentioned. Would you use taskid instead of random number? We can get the unique taskid in configure() method as follows:

                taskId = TaskAttemptID.forName(conf.get(MRJobConfig.TASK_ATTEMPT_ID))
                    .getTaskID().getId();
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Brahma Reddy Battula for reporting and taking this issue. doCreateWriteOp( "file_" + hostName + "_" + Math .random(), reporter); hostname + "_" + Math.random() is not sufficient because there's no guarantee to open_create files and open_read/rename/delete the same files, as Hari Krishna Dara mentioned. Would you use taskid instead of random number? We can get the unique taskid in configure() method as follows: taskId = TaskAttemptID.forName(conf.get(MRJobConfig.TASK_ATTEMPT_ID)) .getTaskID().getId();
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Akira Ajisaka thanks a lot for taking a look into this issue..

          Attached the patch to address your comment.. Kindly review..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Akira Ajisaka thanks a lot for taking a look into this issue.. Attached the patch to address your comment.. Kindly review..
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Brahma Reddy Battula for updating the patch.

          ...
          -          doCreateWriteOp("file_" + hostName + "_", reporter);
          +          doCreateWriteOp("file_" + taskId + "_", reporter);
                   } else if (op.equals(OP_OPEN_READ)) {
                     startTimeTPmS = System.currentTimeMillis();
          -          doOpenReadOp("file_" + hostName + "_", reporter);
          +          doOpenReadOp("file_" + taskId + "_", reporter);
                   } else if (op.equals(OP_RENAME)) {
                     startTimeTPmS = System.currentTimeMillis();
          -          doRenameOp("file_" + hostName + "_", reporter);
          +          doRenameOp("file_" + taskId + "_", reporter);
                   } else if (op.equals(OP_DELETE)) {
          ...
          

          Would you create a variable such as filePrefix and reuse it as follows? That way we can avoid inconsistent file names by typo.

                if (barrier()) {
                  String filePrefix = "file_" + taskId + "_";
                  startTimeTPmS = System.currentTimeMillis();
                  if (op.equals(OP_CREATE_WRITE)) {
                    doCreateWriteOp(filePrefix, reporter);
                  } else if (op.equals(OP_OPEN_READ)) {
                    doOpenReadOp(filePrefix, reporter);
          ...
          
          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Brahma Reddy Battula for updating the patch. ... - doCreateWriteOp( "file_" + hostName + "_" , reporter); + doCreateWriteOp( "file_" + taskId + "_" , reporter); } else if (op.equals(OP_OPEN_READ)) { startTimeTPmS = System .currentTimeMillis(); - doOpenReadOp( "file_" + hostName + "_" , reporter); + doOpenReadOp( "file_" + taskId + "_" , reporter); } else if (op.equals(OP_RENAME)) { startTimeTPmS = System .currentTimeMillis(); - doRenameOp( "file_" + hostName + "_" , reporter); + doRenameOp( "file_" + taskId + "_" , reporter); } else if (op.equals(OP_DELETE)) { ... Would you create a variable such as filePrefix and reuse it as follows? That way we can avoid inconsistent file names by typo. if (barrier()) { String filePrefix = "file_" + taskId + "_" ; startTimeTPmS = System .currentTimeMillis(); if (op.equals(OP_CREATE_WRITE)) { doCreateWriteOp(filePrefix, reporter); } else if (op.equals(OP_OPEN_READ)) { doOpenReadOp(filePrefix, reporter); ...
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks again.. Attached the patch to address your comment..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks again.. Attached the patch to address your comment..
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1 pending Jenkins. Thanks Brahma Reddy Battula.

          Show
          ajisakaa Akira Ajisaka added a comment - +1 pending Jenkins. Thanks Brahma Reddy Battula .
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          -1 pre-patch 5m 41s Findbugs (version ) appears to be broken on trunk.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 34s There were no new javac warning messages.
          +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 13s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 36s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 102m 57s Tests failed in hadoop-mapreduce-client-jobclient.
              119m 46s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestReduceFetch
            hadoop.mapred.TestMRIntermediateDataEncryption
            hadoop.mapred.TestReduceFetchFromPartialMem
            hadoop.mapred.TestMerge
            hadoop.mapred.pipes.TestPipeApplication



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739420/MAPREDUCE-6363-001.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / eef7b50
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/testReport/
          Java 1.7.0_55
          uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment -1 pre-patch 5m 41s Findbugs (version ) appears to be broken on trunk. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 34s There were no new javac warning messages. +1 release audit 0m 21s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 13s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 36s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 48s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 102m 57s Tests failed in hadoop-mapreduce-client-jobclient.     119m 46s   Reason Tests Failed unit tests hadoop.mapred.TestReduceFetch   hadoop.mapred.TestMRIntermediateDataEncryption   hadoop.mapred.TestReduceFetchFromPartialMem   hadoop.mapred.TestMerge   hadoop.mapred.pipes.TestPipeApplication Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739420/MAPREDUCE-6363-001.patch Optional Tests javac unit findbugs checkstyle git revision trunk / eef7b50 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/testReport/ Java 1.7.0_55 uname Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5801/console This message was automatically generated.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Test failures are unrelated to this patch ( will raise another jira to track testfailures)...Hope whitespace can be ignore while comitting.

          Show
          brahmareddy Brahma Reddy Battula added a comment - Test failures are unrelated to this patch ( will raise another jira to track testfailures)...Hope whitespace can be ignore while comitting.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 5m 52s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 30s There were no new javac warning messages.
          +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 35s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 32s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 107m 0s Tests failed in hadoop-mapreduce-client-jobclient.
              124m 11s  



          Reason Tests
          Failed unit tests hadoop.mapred.TestReduceFetch
            hadoop.mapred.TestMRIntermediateDataEncryption
            hadoop.mapred.pipes.TestPipeApplication
            hadoop.mapred.TestReduceFetchFromPartialMem



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12739423/MAPREDUCE-6363-002.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / eef7b50
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 52s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 30s There were no new javac warning messages. +1 release audit 0m 20s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 35s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 32s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 47s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 107m 0s Tests failed in hadoop-mapreduce-client-jobclient.     124m 11s   Reason Tests Failed unit tests hadoop.mapred.TestReduceFetch   hadoop.mapred.TestMRIntermediateDataEncryption   hadoop.mapred.pipes.TestPipeApplication   hadoop.mapred.TestReduceFetchFromPartialMem Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12739423/MAPREDUCE-6363-002.patch Optional Tests javac unit findbugs checkstyle git revision trunk / eef7b50 whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5802/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - Committing this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk and branch-2. Thanks Brahma Reddy Battula for the contribution and Hari Krishna Dara for the comment.

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk and branch-2. Thanks Brahma Reddy Battula for the contribution and Hari Krishna Dara for the comment.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8018/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8018 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8018/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Thanks a lot Akira Ajisaka for review and commiting!!! And thanks to others for looking into this issue..

          Show
          brahmareddy Brahma Reddy Battula added a comment - Thanks a lot Akira Ajisaka for review and commiting!!! And thanks to others for looking into this issue..
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #960 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/960/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #230 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/230/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2158 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2158/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #219 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/219/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #228 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/228/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2176 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2176/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula. (aajisaka: rev 75a2560e51387ea31ef4609ef434475bbbc628f7) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          uladz Vlad Sharanhovich added a comment -

          This fix is not resolving the problem but rather alleviating it using unique output file names for each mapper to avoid write conflicts when multiple mappers are running on the name node. Now, nor usage of random number (OMG!), not TaskID would generate a stable output file names that can be (re-)used for following nnbench runs of READ and DELETE operations. This means that only CREATE will work, everything else is still broken.

          Show
          uladz Vlad Sharanhovich added a comment - This fix is not resolving the problem but rather alleviating it using unique output file names for each mapper to avoid write conflicts when multiple mappers are running on the name node. Now, nor usage of random number (OMG!), not TaskID would generate a stable output file names that can be (re-)used for following nnbench runs of READ and DELETE operations. This means that only CREATE will work, everything else is still broken.
          Hide
          uladz Vlad Sharanhovich added a comment -

          I have a different way to fix this that actually fixes all CREATE/READ/RENAME/DELETE ops. Will upload a new patch for review shortly.

          Show
          uladz Vlad Sharanhovich added a comment - I have a different way to fix this that actually fixes all CREATE/READ/RENAME/DELETE ops. Will upload a new patch for review shortly.
          Hide
          uladz Vlad Sharanhovich added a comment -

          This patch fixes all CREATE/READ/RENAME/DELETE ops on NNBench test. The only limitation is that you have to run the same # mappers in READ/RENAME/DELETE ops and you have used in CREATE, which is kind of natural.

          Show
          uladz Vlad Sharanhovich added a comment - This patch fixes all CREATE/READ/RENAME/DELETE ops on NNBench test. The only limitation is that you have to run the same # mappers in READ/RENAME/DELETE ops and you have used in CREATE, which is kind of natural.
          Hide
          uladz Vlad Sharanhovich added a comment -

          MAPREDUCE-6363-003.patch is uploaded.

          Show
          uladz Vlad Sharanhovich added a comment - MAPREDUCE-6363 -003.patch is uploaded.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Vlad Sharanhovich for the comment and the patch.

          Now, nor usage of random number (OMG!), not TaskID would generate a stable output file names that can be (re-)used for following nnbench runs of READ and DELETE operations.

          Would you please tell me why the taskid does not generate a stable output? The id is unique and sequential for each map task.

          -        String filePrefix = "file_" + taskId + "_";
          +        String fileName = "file_" + value;
          

          Nice catch. We should remove the _.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Vlad Sharanhovich for the comment and the patch. Now, nor usage of random number (OMG!), not TaskID would generate a stable output file names that can be (re-)used for following nnbench runs of READ and DELETE operations. Would you please tell me why the taskid does not generate a stable output? The id is unique and sequential for each map task. - String filePrefix = "file_" + taskId + "_" ; + String fileName = "file_" + value; Nice catch. We should remove the _ .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'm thinking the task id is unique and sequential for each map task, so it can be reused by the following operations. Hi Vlad Sharanhovich, may I close this issue as fixed?

          Nice catch. We should remove the _.

          By the way, let's remove _ in a separate jira.

          Show
          ajisakaa Akira Ajisaka added a comment - I'm thinking the task id is unique and sequential for each map task, so it can be reused by the following operations. Hi Vlad Sharanhovich , may I close this issue as fixed? Nice catch. We should remove the _ . By the way, let's remove _ in a separate jira.
          Hide
          ajithshetty Ajith S added a comment -

          Hi Akira Ajisaka

          I think what Vlad Sharanhovich is pointing, that when we run CREATE test, we will create files withe unique names, thanks to taskid, so CREATE is fine. But when we run rename or delete, the taskid will be new and it will not actually rename or delete the files(created by CREATE benchmark) because it will not find the file name based on "file_"+taskId as taskId will be new. right.?

          Show
          ajithshetty Ajith S added a comment - Hi Akira Ajisaka I think what Vlad Sharanhovich is pointing, that when we run CREATE test, we will create files withe unique names, thanks to taskid, so CREATE is fine. But when we run rename or delete, the taskid will be new and it will not actually rename or delete the files(created by CREATE benchmark) because it will not find the file name based on "file_"+taskId as taskId will be new. right.?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          the taskid will be new

          Not new. NNBench uses a sequential id in the task id (1, 2, ...) instead of the entire task id.

                int taskId =
                    TaskAttemptID.forName(conf.get(MRJobConfig.TASK_ATTEMPT_ID))
                        .getTaskID().getId();
          

          Therefore the number is the same in the following operation. I verified rename operation ran successfully. Attaching the console log.

          Show
          ajisakaa Akira Ajisaka added a comment - the taskid will be new Not new. NNBench uses a sequential id in the task id (1, 2, ...) instead of the entire task id. int taskId = TaskAttemptID.forName(conf.get(MRJobConfig.TASK_ATTEMPT_ID)) .getTaskID().getId(); Therefore the number is the same in the following operation. I verified rename operation ran successfully. Attaching the console log.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Sorry for coming late.. Akira Ajisaka thanks for clarification and pasting the testresult here..Seems taskId might have confused .. can we change this variable like attemptID and remove _ in a separate jira.

          And can we close this jira since Vlad Sharanhovich not responding ..? what do you say..?

          Show
          brahmareddy Brahma Reddy Battula added a comment - Sorry for coming late.. Akira Ajisaka thanks for clarification and pasting the testresult here..Seems taskId might have confused .. can we change this variable like attemptID and remove _ in a separate jira. And can we close this jira since Vlad Sharanhovich not responding ..? what do you say..?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          By the way, I'm thinking Vlad's patch looks better than parsing TaskAttemptID. I'll create a separate jira for the follow-up. Could you post the patch there?

          Show
          ajisakaa Akira Ajisaka added a comment - By the way, I'm thinking Vlad's patch looks better than parsing TaskAttemptID. I'll create a separate jira for the follow-up. Could you post the patch there?
          Hide
          uladz Vlad Sharanhovich added a comment -

          Guys, one sec, let me read though comments and respond, don't close it just yet.

          Show
          uladz Vlad Sharanhovich added a comment - Guys, one sec, let me read though comments and respond, don't close it just yet.
          Hide
          uladz Vlad Sharanhovich added a comment -

          So, theoretically task ID on a map task can be used as it will be stable due to the fact that control files can not be merged by Hadoop and it is guaranteed that the number of map tasks will be at least the same (or equal at this case) as the number of control input files. Said that I'm strongly advise against it due to the fact that that you rely on inability of Hadoop framework to merge inputs, which is a big issue and as I remember there is a Jira that tracks implementation of rack-aware inputs merge. My point is that while the task ID will work right now (and maybe even for any feature version), there is no guarantee that at some point Hadoop would not change its behaviour.

          On another hand generating and passing a unique ID without control files as I have proposed in my implementation is a "natural" way as we just use value, which is already there but for some reason was set to 0 and not used before. We have one central functions that is responsible for unique number generation and each downstream task is guaranteed to use one unique IDs no matter what. I truly don't see any reason why to parse a task name and convert it into a task ID if each mapper get the value as input anyway.The IDs don't even need to ge sequential, jut unique!

          One more remark - the new reduce code also collects statistics from all mappers/reducers, before it was broken.
          And another remark - I had lots of trouble figuring out why NNBench does not work, so I've expanded error reporting too.

          Said all that, the version of the code that I have uploaded is successfully running of daily basis in our lab cluster and produces stable results. Meaning that it is fully tested in production environment, and I highly recommend to use this approach.

          Show
          uladz Vlad Sharanhovich added a comment - So, theoretically task ID on a map task can be used as it will be stable due to the fact that control files can not be merged by Hadoop and it is guaranteed that the number of map tasks will be at least the same (or equal at this case) as the number of control input files. Said that I'm strongly advise against it due to the fact that that you rely on inability of Hadoop framework to merge inputs, which is a big issue and as I remember there is a Jira that tracks implementation of rack-aware inputs merge. My point is that while the task ID will work right now (and maybe even for any feature version), there is no guarantee that at some point Hadoop would not change its behaviour. On another hand generating and passing a unique ID without control files as I have proposed in my implementation is a "natural" way as we just use value, which is already there but for some reason was set to 0 and not used before. We have one central functions that is responsible for unique number generation and each downstream task is guaranteed to use one unique IDs no matter what. I truly don't see any reason why to parse a task name and convert it into a task ID if each mapper get the value as input anyway.The IDs don't even need to ge sequential, jut unique! One more remark - the new reduce code also collects statistics from all mappers/reducers, before it was broken. And another remark - I had lots of trouble figuring out why NNBench does not work, so I've expanded error reporting too. Said all that, the version of the code that I have uploaded is successfully running of daily basis in our lab cluster and produces stable results. Meaning that it is fully tested in production environment, and I highly recommend to use this approach.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I'll create a separate jira for the follow-up. Could you post the patch there?

          or I can revert the commit.

          Rethinking this, I'd like to revert the commit because the commit causes a bug (unnecessarily _). If we are to backport them to other branch, we need to backport two commits.

          Show
          ajisakaa Akira Ajisaka added a comment - I'll create a separate jira for the follow-up. Could you post the patch there? or I can revert the commit. Rethinking this, I'd like to revert the commit because the commit causes a bug (unnecessarily _ ). If we are to backport them to other branch, we need to backport two commits.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Vlad Sharanhovich for the detailed comment.

          you rely on inability of Hadoop framework to merge inputs, which is a big issue and as I remember there is a Jira that tracks implementation of rack-aware inputs merge.

          Now Hadoop can use CombineFileInputFormat to merge inputs. NNBench does not use the feature.

          On another hand generating and passing a unique ID without control files as I have proposed in my implementation is a "natural" way as we just use value, which is already there but for some reason was set to 0 and not used before.

          Agree.

          One more remark - the new reduce code also collects statistics from all mappers/reducers, before it was broken.

          This issue looks to be reported by MAPREDUCE-6465. When your patch will be committed, then we can close it.

          And another remark - I had lots of trouble figuring out why NNBench does not work, so I've expanded error reporting too.

          Thanks a lot.

          I'll revert the commit shortly. Would you rebase the patch after reverting the commit?

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Vlad Sharanhovich for the detailed comment. you rely on inability of Hadoop framework to merge inputs, which is a big issue and as I remember there is a Jira that tracks implementation of rack-aware inputs merge. Now Hadoop can use CombineFileInputFormat to merge inputs. NNBench does not use the feature. On another hand generating and passing a unique ID without control files as I have proposed in my implementation is a "natural" way as we just use value, which is already there but for some reason was set to 0 and not used before. Agree. One more remark - the new reduce code also collects statistics from all mappers/reducers, before it was broken. This issue looks to be reported by MAPREDUCE-6465 . When your patch will be committed, then we can close it. And another remark - I had lots of trouble figuring out why NNBench does not work, so I've expanded error reporting too. Thanks a lot. I'll revert the commit shortly. Would you rebase the patch after reverting the commit?
          Hide
          uladz Vlad Sharanhovich added a comment -

          I'll revert the commit shortly. Would you rebase the patch after reverting the commit?

          Sure!

          Show
          uladz Vlad Sharanhovich added a comment - I'll revert the commit shortly. Would you rebase the patch after reverting the commit? Sure!
          Hide
          ajisakaa Akira Ajisaka added a comment -

          I've reverted the commit.

          Show
          ajisakaa Akira Ajisaka added a comment - I've reverted the commit.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #8384 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8384/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8384 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8384/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #2281 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2281/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #2281 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2281/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/332/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #332 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/332/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #340 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/340/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #340 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/340/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #1067 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1067/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1067 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1067/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          uladz Vlad Sharanhovich added a comment -

          I'm having some troubles with rebase and commit...

          Show
          uladz Vlad Sharanhovich added a comment - I'm having some troubles with rebase and commit...
          Hide
          uladz Vlad Sharanhovich added a comment -

          Actually I've just grabbed the latest HEAD and ported the patch, I'll upload the new rebased patch in 10 mins.

          Show
          uladz Vlad Sharanhovich added a comment - Actually I've just grabbed the latest HEAD and ported the patch, I'll upload the new rebased patch in 10 mins.
          Hide
          brahmareddy Brahma Reddy Battula added a comment -

          Vlad Sharanhovich rebased patch for you.. Kindly check...

          Show
          brahmareddy Brahma Reddy Battula added a comment - Vlad Sharanhovich rebased patch for you.. Kindly check...
          Hide
          uladz Vlad Sharanhovich added a comment -

          Thanks! But unfortunately there are several issue with this version of rebased patch. The file format is broken due to the first git line, this is not a patch format . And all my changes of info message to errors were lost. Uploading my rebased version.

          Show
          uladz Vlad Sharanhovich added a comment - Thanks! But unfortunately there are several issue with this version of rebased patch. The file format is broken due to the first git line, this is not a patch format . And all my changes of info message to errors were lost. Uploading my rebased version.
          Hide
          uladz Vlad Sharanhovich added a comment -

          Rebased version of MAPREDUCE-6363-003.patch.

          Show
          uladz Vlad Sharanhovich added a comment - Rebased version of MAPREDUCE-6363 -003.patch.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #2262 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2262/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/CHANGES.txt
          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #2262 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2262/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/CHANGES.txt hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #323 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/323/)
          Revert "MAPREDUCE-6363. [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #323 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/323/ ) Revert " MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with multiple mappers. Contributed by Brahma Reddy Battula." (aajisaka: rev 00804e24579083d03a67b86c2c57e70b910dda8d) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Thanks Vlad Sharanhovich for rebaseing the patch. Minor comments from me:

          1. Would you fix the formatting?

                    }
                    catch (IOException e) {
          

          should be

                    } catch (IOException e) {
          

          Hadoop code base is formatted according to Oracle's conventions except for indents. (http://www.oracle.com/technetwork/java/javase/documentation/codeconventions-142311.html#449)

          2. Would you raise the log level of the error message in doDeleteOp method as well?

          I'll try NNBench with your patch later.

          Show
          ajisakaa Akira Ajisaka added a comment - Thanks Vlad Sharanhovich for rebaseing the patch. Minor comments from me: 1. Would you fix the formatting? } catch (IOException e) { should be } catch (IOException e) { Hadoop code base is formatted according to Oracle's conventions except for indents. ( http://www.oracle.com/technetwork/java/javase/documentation/codeconventions-142311.html#449 ) 2. Would you raise the log level of the error message in doDeleteOp method as well? I'll try NNBench with your patch later.
          Hide
          hadoopqa Hadoop QA added a comment -



          -1 overall



          Vote Subsystem Runtime Comment
          0 pre-patch 6m 24s Pre-patch trunk compilation is healthy.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 tests included 0m 0s The patch appears to include 1 new or modified test files.
          +1 javac 7m 51s There were no new javac warning messages.
          +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
          +1 checkstyle 0m 35s There were no new checkstyle issues.
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 install 1m 28s mvn install still works.
          +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
          +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings.
          -1 mapreduce tests 96m 12s Tests failed in hadoop-mapreduce-client-jobclient.
              114m 13s  



          Reason Tests
          Failed unit tests hadoop.mapred.pipes.TestPipeApplication
            hadoop.mapreduce.lib.output.TestJobOutputCommitter
            hadoop.mapred.TestJobSysDirWithDFS
            hadoop.mapreduce.TestMRJobClient
            hadoop.mapreduce.security.TestMRCredentials
            hadoop.mapred.TestMRTimelineEventHandling
            hadoop.mapreduce.TestChild
            hadoop.mapreduce.security.TestBinaryTokenFile



          Subsystem Report/Notes
          Patch URL http://issues.apache.org/jira/secure/attachment/12753708/MAPREDUCE-6363-005.patch
          Optional Tests javac unit findbugs checkstyle
          git revision trunk / 65ccf2b
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/artifact/patchprocess/whitespace.txt
          hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
          Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/testReport/
          Java 1.7.0_55
          uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 6m 24s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. +1 tests included 0m 0s The patch appears to include 1 new or modified test files. +1 javac 7m 51s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. +1 checkstyle 0m 35s There were no new checkstyle issues. -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 28s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 findbugs 0m 49s The patch does not introduce any new Findbugs (version 3.0.0) warnings. -1 mapreduce tests 96m 12s Tests failed in hadoop-mapreduce-client-jobclient.     114m 13s   Reason Tests Failed unit tests hadoop.mapred.pipes.TestPipeApplication   hadoop.mapreduce.lib.output.TestJobOutputCommitter   hadoop.mapred.TestJobSysDirWithDFS   hadoop.mapreduce.TestMRJobClient   hadoop.mapreduce.security.TestMRCredentials   hadoop.mapred.TestMRTimelineEventHandling   hadoop.mapreduce.TestChild   hadoop.mapreduce.security.TestBinaryTokenFile Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12753708/MAPREDUCE-6363-005.patch Optional Tests javac unit findbugs checkstyle git revision trunk / 65ccf2b whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/artifact/patchprocess/whitespace.txt hadoop-mapreduce-client-jobclient test log https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/testReport/ Java 1.7.0_55 uname Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5964/console This message was automatically generated.
          Hide
          uladz Vlad Sharanhovich added a comment -

          Let me fix a couple of minor things and reupload the patch. Thanks Akira Ajisaka!

          Show
          uladz Vlad Sharanhovich added a comment - Let me fix a couple of minor things and reupload the patch. Thanks Akira Ajisaka !
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Any updates on this jira ?

          Show
          bibinchundatt Bibin A Chundatt added a comment - Any updates on this jira ?
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Hi Vlad Sharanhovich, how is this issue going on?

          Show
          ajisakaa Akira Ajisaka added a comment - Hi Vlad Sharanhovich , how is this issue going on?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 23s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_66
          +1 compile 0m 22s trunk passed with JDK v1.7.0_85
          +1 checkstyle 0m 8s trunk passed
          +1 mvnsite 0m 27s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 14s trunk passed with JDK v1.7.0_85
          +1 mvninstall 0m 25s the patch passed
          +1 compile 0m 18s the patch passed with JDK v1.8.0_66
          -1 javac 1m 37s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 189, now 189).
          +1 javac 0m 18s the patch passed
          +1 compile 0m 21s the patch passed with JDK v1.7.0_85
          -1 javac 2m 0s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85 with JDK v1.7.0_85 generated 1 new issues (was 189, now 189).
          +1 javac 0m 21s the patch passed
          +1 checkstyle 0m 7s the patch passed
          +1 mvnsite 0m 27s the patch passed
          +1 mvneclipse 0m 13s the patch passed
          -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 12s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 14s the patch passed with JDK v1.7.0_85
          -1 unit 107m 54s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          -1 unit 0m 17s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_85.
          -1 asflicense 0m 28s Patch generated 13 ASF License warnings.
          122m 25s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestSpecialCharactersInOutputPath
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
          JDK v1.7.0_85 Failed junit tests hadoop.mapred.TestSpecialCharactersInOutputPath
            hadoop.mapreduce.v2.TestMRJobsWithProfiler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12753708/MAPREDUCE-6363-005.patch
          JIRA Issue MAPREDUCE-6363
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 44185032385f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 28dfe72
          findbugs v3.0.0
          javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt
          whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/whitespace-eol.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt
          JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Max memory used 75MB
          Powered by Apache Yetus http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 23s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 22s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 8s trunk passed +1 mvnsite 0m 27s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 14s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 25s the patch passed +1 compile 0m 18s the patch passed with JDK v1.8.0_66 -1 javac 1m 37s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66 with JDK v1.8.0_66 generated 1 new issues (was 189, now 189). +1 javac 0m 18s the patch passed +1 compile 0m 21s the patch passed with JDK v1.7.0_85 -1 javac 2m 0s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85 with JDK v1.7.0_85 generated 1 new issues (was 189, now 189). +1 javac 0m 21s the patch passed +1 checkstyle 0m 7s the patch passed +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 13s the patch passed -1 whitespace 0m 0s The patch has 2 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 12s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 14s the patch passed with JDK v1.7.0_85 -1 unit 107m 54s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. -1 unit 0m 17s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_85. -1 asflicense 0m 28s Patch generated 13 ASF License warnings. 122m 25s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestSpecialCharactersInOutputPath   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.8.0_66 Timed out junit tests org.apache.hadoop.mapred.TestMRIntermediateDataEncryption JDK v1.7.0_85 Failed junit tests hadoop.mapred.TestSpecialCharactersInOutputPath   hadoop.mapreduce.v2.TestMRJobsWithProfiler Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12753708/MAPREDUCE-6363-005.patch JIRA Issue MAPREDUCE-6363 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 44185032385f 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 28dfe72 findbugs v3.0.0 javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/whitespace-eol.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_85.txt JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6184/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Attached rebased patch

          Show
          bibinchundatt Bibin A Chundatt added a comment - Attached rebased patch
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 34s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_66
          +1 compile 0m 22s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 9s trunk passed
          +1 mvnsite 0m 28s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 27s trunk passed
          +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 17s the patch passed with JDK v1.8.0_66
          +1 javac 0m 17s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.7.0_91
          +1 javac 0m 19s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 32s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          -1 unit 97m 19s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          -1 unit 103m 52s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 24s Patch generated 17 ASF License warnings.
          215m 11s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestNetworkedJob
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780986/MAPREDUCE-6363-006.patch
          JIRA Issue MAPREDUCE-6363
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux e44d61c7120d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / bac798a
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Max memory used 75MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 34s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 22s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 9s trunk passed +1 mvnsite 0m 28s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 27s trunk passed +1 javadoc 0m 12s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 22s the patch passed +1 compile 0m 17s the patch passed with JDK v1.8.0_66 +1 javac 0m 17s the patch passed +1 compile 0m 19s the patch passed with JDK v1.7.0_91 +1 javac 0m 19s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 32s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 -1 unit 97m 19s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. -1 unit 103m 52s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 24s Patch generated 17 ASF License warnings. 215m 11s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestNetworkedJob   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780986/MAPREDUCE-6363-006.patch JIRA Issue MAPREDUCE-6363 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux e44d61c7120d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / bac798a Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Max memory used 75MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6258/console This message was automatically generated.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 8m 1s trunk passed
          +1 compile 0m 21s trunk passed with JDK v1.8.0_66
          +1 compile 0m 24s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 8s trunk passed
          +1 mvnsite 0m 31s trunk passed
          +1 mvneclipse 0m 13s trunk passed
          +1 findbugs 0m 29s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 19s the patch passed with JDK v1.8.0_66
          +1 javac 0m 19s the patch passed
          +1 compile 0m 22s the patch passed with JDK v1.7.0_91
          +1 javac 0m 22s the patch passed
          +1 checkstyle 0m 9s the patch passed
          +1 mvnsite 0m 27s the patch passed
          +1 mvneclipse 0m 12s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          -1 unit 101m 33s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          -1 unit 108m 6s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 15 ASF License warnings.
          224m 40s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestNetworkedJob
            hadoop.mapreduce.v2.TestMRJobsWithProfiler
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780989/MAPREDUCE-6363-007.patch
          JIRA Issue MAPREDUCE-6363
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 6798e708d59b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 52b7757
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 8m 1s trunk passed +1 compile 0m 21s trunk passed with JDK v1.8.0_66 +1 compile 0m 24s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 8s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 13s trunk passed +1 findbugs 0m 29s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 24s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_66 +1 javac 0m 19s the patch passed +1 compile 0m 22s the patch passed with JDK v1.7.0_91 +1 javac 0m 22s the patch passed +1 checkstyle 0m 9s the patch passed +1 mvnsite 0m 27s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 -1 unit 101m 33s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. -1 unit 108m 6s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 15 ASF License warnings. 224m 40s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapred.TestNetworkedJob   hadoop.mapreduce.v2.TestMRJobsWithProfiler JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12780989/MAPREDUCE-6363-007.patch JIRA Issue MAPREDUCE-6363 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 6798e708d59b 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 52b7757 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6259/console This message was automatically generated.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Looks good to me. Thanks Bibin A Chundatt for updating the patch. One minor comment.
          1. Would you fix the formatting?

          +        }
          +        else {
          +          throw new IllegalArgumentException("unsupported operation ["+op+"]");
          

          should be

          +        } else {
          +          throw new IllegalArgumentException("unsupported operation ["+op+"]");
          

          I'm +1 if that is addressed.

          Show
          ajisakaa Akira Ajisaka added a comment - Looks good to me. Thanks Bibin A Chundatt for updating the patch. One minor comment. 1. Would you fix the formatting? + } + else { + throw new IllegalArgumentException( "unsupported operation [" +op+ "]" ); should be + } else { + throw new IllegalArgumentException( "unsupported operation [" +op+ "]" ); I'm +1 if that is addressed.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Akira Ajisaka
          Attaching patch after formatting. Thanks for the review

          Show
          bibinchundatt Bibin A Chundatt added a comment - Akira Ajisaka Attaching patch after formatting. Thanks for the review
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          +1 @author 0m 1s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
          +1 mvninstall 7m 37s trunk passed
          +1 compile 0m 20s trunk passed with JDK v1.8.0_66
          +1 compile 0m 23s trunk passed with JDK v1.7.0_91
          +1 checkstyle 0m 8s trunk passed
          +1 mvnsite 0m 29s trunk passed
          +1 mvneclipse 0m 14s trunk passed
          +1 findbugs 0m 28s trunk passed
          +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66
          +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91
          +1 mvninstall 0m 22s the patch passed
          +1 compile 0m 16s the patch passed with JDK v1.8.0_66
          +1 javac 0m 16s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.7.0_91
          +1 javac 0m 20s the patch passed
          +1 checkstyle 0m 8s the patch passed
          +1 mvnsite 0m 25s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s Patch has no whitespace issues.
          +1 findbugs 0m 34s the patch passed
          +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66
          +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91
          -1 unit 101m 15s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66.
          -1 unit 108m 39s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91.
          -1 asflicense 0m 27s Patch generated 15 ASF License warnings.
          224m 12s



          Reason Tests
          JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler
            hadoop.mapred.TestNetworkedJob
          JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:0ca8df7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781222/MAPREDUCE-6363-008.patch
          JIRA Issue MAPREDUCE-6363
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 72614300a565 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ed18527
          Default Java 1.7.0_91
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt
          unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt
          JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/testReport/
          asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
          Max memory used 76MB
          Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
          Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/console

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 1s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. +1 mvninstall 7m 37s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 23s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 8s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 14s trunk passed +1 findbugs 0m 28s trunk passed +1 javadoc 0m 13s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 15s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 22s the patch passed +1 compile 0m 16s the patch passed with JDK v1.8.0_66 +1 javac 0m 16s the patch passed +1 compile 0m 20s the patch passed with JDK v1.7.0_91 +1 javac 0m 20s the patch passed +1 checkstyle 0m 8s the patch passed +1 mvnsite 0m 25s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 0m 34s the patch passed +1 javadoc 0m 10s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 12s the patch passed with JDK v1.7.0_91 -1 unit 101m 15s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_66. -1 unit 108m 39s hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_91. -1 asflicense 0m 27s Patch generated 15 ASF License warnings. 224m 12s Reason Tests JDK v1.8.0_66 Failed junit tests hadoop.mapreduce.v2.TestMRJobsWithProfiler   hadoop.mapred.TestNetworkedJob JDK v1.7.0_91 Failed junit tests hadoop.mapred.TestNetworkedJob Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781222/MAPREDUCE-6363-008.patch JIRA Issue MAPREDUCE-6363 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 72614300a565 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ed18527 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt unit https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt unit test logs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_91.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6261/console This message was automatically generated.
          Hide
          bibinchundatt Bibin A Chundatt added a comment -

          Akira Ajisaka
          Testcase failures are not related to patch attached. Could you please help in reviewing patch.

          Show
          bibinchundatt Bibin A Chundatt added a comment - Akira Ajisaka Testcase failures are not related to patch attached. Could you please help in reviewing patch.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          +1, committing this.

          Show
          ajisakaa Akira Ajisaka added a comment - +1, committing this.
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6. Thanks Bibin A Chundatt for the contribution!

          Show
          ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6. Thanks Bibin A Chundatt for the contribution!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #9105 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9105/)
          MAPREDUCE-6363. [NNBench] Lease mismatch error when running with (aajisaka: rev 7b0964f354e90968c2dac2f7acc17214732aed64)

          • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java
          • hadoop-mapreduce-project/CHANGES.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9105 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9105/ ) MAPREDUCE-6363 . [NNBench] Lease mismatch error when running with (aajisaka: rev 7b0964f354e90968c2dac2f7acc17214732aed64) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/hdfs/NNBench.java hadoop-mapreduce-project/CHANGES.txt
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Reopen this because of missing doDeleteOp(fileName, reporter) in the v8 patch. Thanks J.Andreina for reporting this issue in HDFS-10191.

          Show
          ajisakaa Akira Ajisaka added a comment - Reopen this because of missing doDeleteOp(fileName, reporter) in the v8 patch. Thanks J.Andreina for reporting this issue in HDFS-10191 .
          Hide
          ajisakaa Akira Ajisaka added a comment -

          Rethinking this, it would be better to add doDeleteOp in a separate jira because 2.6.4, the v8 patch was committed to, has been released.

          Show
          ajisakaa Akira Ajisaka added a comment - Rethinking this, it would be better to add doDeleteOp in a separate jira because 2.6.4, the v8 patch was committed to, has been released.
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Closing the JIRA as part of 2.7.3 release.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.

            People

            • Assignee:
              bibinchundatt Bibin A Chundatt
              Reporter:
              brahmareddy Brahma Reddy Battula
            • Votes:
              2 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development