Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6554

MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 2.7.3, 2.6.4, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Create scenario so that MR app master gets preempted.
      On next MRAppMaster launch tried to recover previous job history file MRAppMaster#parsePreviousJobHistory

      2015-11-21 13:52:27,722 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED; cause: java.lang.NullPointerException
      java.lang.NullPointerException
              at java.io.StringReader.<init>(StringReader.java:50)
              at org.apache.avro.Schema$Parser.parse(Schema.java:917)
              at org.apache.avro.Schema.parse(Schema.java:966)
              at org.apache.hadoop.mapreduce.jobhistory.EventReader.<init>(EventReader.java:75)
              at org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
              at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
              at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
      2015-11-21 13:52:27,725 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping JobHistoryEventHandler. Size of the outstanding queue size is 0
      
      

      EventReader(EventReader stream)

       this.version = in.readLine();
      ...
          Schema myschema = new SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
          this.schema = Schema.parse(in.readLine());
      
      1. MAPREDUCE-6554-branch-2.7.001.patch
        3 kB
        Bibin A Chundatt
      2. MAPREDUCE-6554-branch-2.7.patch
        3 kB
        Bibin A Chundatt
      3. 0003-MAPREDUCE-6554.patch
        3 kB
        Bibin A Chundatt
      4. 0002-MAPREDUCE-6554.patch
        2 kB
        Bibin A Chundatt
      5. 0001-MAPREDUCE-6554.patch
        3 kB
        Bibin A Chundatt

        Activity

        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Closing the JIRA as part of 2.7.3 release.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Closing the JIRA as part of 2.7.3 release.
        Hide
        mauriciojost Mauricio Jost added a comment -

        Hello all. I am sorry to jump in, but I am facing this issue: is there a workaround?

        Show
        mauriciojost Mauricio Jost added a comment - Hello all. I am sorry to jump in, but I am facing this issue: is there a workaround?
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Bibin! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.

        Show
        jlowe Jason Lowe added a comment - Thanks, Bibin! I committed this to trunk, branch-2, branch-2.8, branch-2.7, and branch-2.6.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #9120 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9120/)
        MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in (jlowe: rev 9fbd579ab50f051c8ad63d316ca5a3e24822dcdd)

        • hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java
        • hadoop-mapreduce-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #9120 (See https://builds.apache.org/job/Hadoop-trunk-Commit/9120/ ) MAPREDUCE-6554 . MRAppMaster servicestart failing with NPE in (jlowe: rev 9fbd579ab50f051c8ad63d316ca5a3e24822dcdd) hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/EventReader.java hadoop-mapreduce-project/CHANGES.txt
        Hide
        jlowe Jason Lowe added a comment -

        +1 lgtm. The checkstyle issue was present before the patch as well, and the whitespace, tab, and license warnings are not due to this patch.

        Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 lgtm. The checkstyle issue was present before the patch as well, and the whitespace, tab, and license warnings are not due to this patch. Committing this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 39s branch-2.7 passed
        +1 compile 0m 22s branch-2.7 passed with JDK v1.8.0_66
        +1 compile 0m 23s branch-2.7 passed with JDK v1.7.0_91
        +1 checkstyle 0m 18s branch-2.7 passed
        +1 mvnsite 0m 33s branch-2.7 passed
        +1 mvneclipse 0m 16s branch-2.7 passed
        -1 findbugs 1m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-2.7 has 2 extant Findbugs warnings.
        +1 javadoc 0m 24s branch-2.7 passed with JDK v1.8.0_66
        +1 javadoc 0m 25s branch-2.7 passed with JDK v1.7.0_91
        +1 mvninstall 0m 23s the patch passed
        +1 compile 0m 19s the patch passed with JDK v1.8.0_66
        +1 javac 0m 19s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_91
        -1 javac 3m 43s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 403, now 402).
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 13s the patch passed
        +1 mvnsite 0m 29s the patch passed
        +1 mvneclipse 0m 11s the patch passed
        -1 whitespace 0m 3s The patch has 2189 line(s) that end in whitespace. Use git apply --whitespace=fix.
        -1 whitespace 0m 56s The patch has 99 line(s) with tabs.
        +1 findbugs 1m 18s the patch passed
        +1 javadoc 0m 20s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91
        +1 unit 1m 38s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 47s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        -1 asflicense 45m 30s Patch generated 81 ASF License warnings.
        84m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:date2016-01-15
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782418/MAPREDUCE-6554-branch-2.7.001.patch
        JIRA Issue MAPREDUCE-6554
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 3168486c0748 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.7 / dbed882
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/whitespace-eol.txt
        whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/whitespace-tabs.txt
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 67MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 39s branch-2.7 passed +1 compile 0m 22s branch-2.7 passed with JDK v1.8.0_66 +1 compile 0m 23s branch-2.7 passed with JDK v1.7.0_91 +1 checkstyle 0m 18s branch-2.7 passed +1 mvnsite 0m 33s branch-2.7 passed +1 mvneclipse 0m 16s branch-2.7 passed -1 findbugs 1m 16s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in branch-2.7 has 2 extant Findbugs warnings. +1 javadoc 0m 24s branch-2.7 passed with JDK v1.8.0_66 +1 javadoc 0m 25s branch-2.7 passed with JDK v1.7.0_91 +1 mvninstall 0m 23s the patch passed +1 compile 0m 19s the patch passed with JDK v1.8.0_66 +1 javac 0m 19s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_91 -1 javac 3m 43s hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91 with JDK v1.7.0_91 generated 1 new issues (was 403, now 402). +1 javac 0m 23s the patch passed +1 checkstyle 0m 13s the patch passed +1 mvnsite 0m 29s the patch passed +1 mvneclipse 0m 11s the patch passed -1 whitespace 0m 3s The patch has 2189 line(s) that end in whitespace. Use git apply --whitespace=fix. -1 whitespace 0m 56s The patch has 99 line(s) with tabs. +1 findbugs 1m 18s the patch passed +1 javadoc 0m 20s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 22s the patch passed with JDK v1.7.0_91 +1 unit 1m 38s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 1m 47s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. -1 asflicense 45m 30s Patch generated 81 ASF License warnings. 84m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:date2016-01-15 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782418/MAPREDUCE-6554-branch-2.7.001.patch JIRA Issue MAPREDUCE-6554 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 3168486c0748 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.7 / dbed882 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html javac hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/diff-compile-javac-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_91.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/whitespace-eol.txt whitespace https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/whitespace-tabs.txt JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 67MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6270/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 docker 0m 6s Dockerfile '/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/dev-support/docker/Dockerfile' not found, falling back to built-in.
        -1 docker 12m 37s Docker failed to build yetus/hadoop:date2016-01-14.



        Subsystem Report/Notes
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782294/MAPREDUCE-6554-branch-2.7.patch
        JIRA Issue MAPREDUCE-6554
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6269/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 docker 0m 6s Dockerfile '/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/dev-support/docker/Dockerfile' not found, falling back to built-in. -1 docker 12m 37s Docker failed to build yetus/hadoop:date2016-01-14. Subsystem Report/Notes JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12782294/MAPREDUCE-6554-branch-2.7.patch JIRA Issue MAPREDUCE-6554 Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6269/console This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Jason Lowe
        Thanks you for review . Attaching patch for branch-2.7

        Show
        bibinchundatt Bibin A Chundatt added a comment - Jason Lowe Thanks you for review . Attaching patch for branch-2.7
        Hide
        jlowe Jason Lowe added a comment -

        +1 latest patch lgtm. Could you also provide a patch for branch-2.7/branch-2.6?

        Show
        jlowe Jason Lowe added a comment - +1 latest patch lgtm. Could you also provide a patch for branch-2.7/branch-2.6?
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Jason Lowe
        Testcase/Findbug failures are not related to patch attached. Could you please review patch attached.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Jason Lowe Testcase/Findbug failures are not related to patch attached. Could you please review patch attached.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 9m 20s trunk passed
        +1 compile 0m 28s trunk passed with JDK v1.8.0_66
        +1 compile 0m 26s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 16s trunk passed
        +1 mvnsite 0m 38s trunk passed
        +1 mvneclipse 0m 16s trunk passed
        -1 findbugs 1m 24s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
        +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 28s the patch passed with JDK v1.8.0_66
        +1 javac 0m 28s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.7.0_91
        +1 javac 0m 23s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 33s the patch passed
        +1 mvneclipse 0m 13s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 35s the patch passed
        +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91
        +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        +1 asflicense 0m 21s Patch does not generate ASF License warnings.
        25m 0s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781837/0003-MAPREDUCE-6554.patch
        JIRA Issue MAPREDUCE-6554
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2fb346671a35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 1715864
        Default Java 1.7.0_91
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/testReport/
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 9m 20s trunk passed +1 compile 0m 28s trunk passed with JDK v1.8.0_66 +1 compile 0m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 16s trunk passed +1 mvnsite 0m 38s trunk passed +1 mvneclipse 0m 16s trunk passed -1 findbugs 1m 24s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 27s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 30s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 28s the patch passed +1 compile 0m 28s the patch passed with JDK v1.8.0_66 +1 javac 0m 28s the patch passed +1 compile 0m 23s the patch passed with JDK v1.7.0_91 +1 javac 0m 23s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 33s the patch passed +1 mvneclipse 0m 13s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 35s the patch passed +1 javadoc 0m 22s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91 +1 unit 2m 19s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 2m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. +1 asflicense 0m 21s Patch does not generate ASF License warnings. 25m 0s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12781837/0003-MAPREDUCE-6554.patch JIRA Issue MAPREDUCE-6554 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2fb346671a35 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 1715864 Default Java 1.7.0_91 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_66 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_91 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/testReport/ modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6268/console This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Jason Lowe
        Thank you for reviewing patch.Sorry to miss the second issue i had mentioned.
        Attached patch after handling both the cases

        Show
        bibinchundatt Bibin A Chundatt added a comment - Jason Lowe Thank you for reviewing patch.Sorry to miss the second issue i had mentioned. Attached patch after handling both the cases
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch, Bibin! The patch will fix the specific NPE reported in the summary, but it won't address the other case you reported with the Avro exception. How about if we check for a null schema and translate any AvroRuntimeException thrown by the parser into an IOException? I think that would be a bit safer than assuming every exception should be a parse error I/O exception.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch, Bibin! The patch will fix the specific NPE reported in the summary, but it won't address the other case you reported with the Avro exception. How about if we check for a null schema and translate any AvroRuntimeException thrown by the parser into an IOException? I think that would be a bit safer than assuming every exception should be a parse error I/O exception.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 8m 27s trunk passed
        +1 compile 0m 27s trunk passed with JDK v1.8.0_66
        +1 compile 0m 26s trunk passed with JDK v1.7.0_91
        +1 checkstyle 0m 14s trunk passed
        +1 mvnsite 0m 35s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
        +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91
        +1 mvninstall 0m 30s the patch passed
        +1 compile 0m 23s the patch passed with JDK v1.8.0_66
        +1 javac 0m 23s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_91
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 14s the patch passed
        +1 mvnsite 0m 34s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 24s the patch passed
        +1 javadoc 0m 24s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91
        +1 unit 2m 7s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91.
        -1 asflicense 0m 24s Patch generated 1 ASF License warnings.
        23m 6s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779077/0002-MAPREDUCE-6554.patch
        JIRA Issue MAPREDUCE-6554
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 2074024e973d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 70d6f20
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 76MB
        Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 8m 27s trunk passed +1 compile 0m 27s trunk passed with JDK v1.8.0_66 +1 compile 0m 26s trunk passed with JDK v1.7.0_91 +1 checkstyle 0m 14s trunk passed +1 mvnsite 0m 35s trunk passed +1 mvneclipse 0m 15s trunk passed -1 findbugs 1m 18s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 25s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 27s trunk passed with JDK v1.7.0_91 +1 mvninstall 0m 30s the patch passed +1 compile 0m 23s the patch passed with JDK v1.8.0_66 +1 javac 0m 23s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_91 +1 javac 0m 25s the patch passed +1 checkstyle 0m 14s the patch passed +1 mvnsite 0m 34s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 24s the patch passed +1 javadoc 0m 24s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_91 +1 unit 2m 7s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 2m 23s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_91. -1 asflicense 0m 24s Patch generated 1 ASF License warnings. 23m 6s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12779077/0002-MAPREDUCE-6554.patch JIRA Issue MAPREDUCE-6554 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 2074024e973d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 70d6f20 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html JDK v1.7.0_91 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 76MB Powered by Apache Yetus 0.2.0-SNAPSHOT http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6225/console This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Attaching updated patch.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Attaching updated patch.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Jason Lowe
        Sorry for late response.
        Instead of this.schema = parser.parse(in.readLine()); before passing to parser.parse() we can check for null and the return IOException.
        For IOException org.apache.hadoop.tools.rumen.CurrentJHParser#canParse returns false .

        Please do correct me if i am wrong.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Jason Lowe Sorry for late response. Instead of this.schema = parser.parse(in.readLine()); before passing to parser.parse() we can check for null and the return IOException. For IOException org.apache.hadoop.tools.rumen.CurrentJHParser#canParse returns false . Please do correct me if i am wrong.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the report and patch, Bibin! However I'm not sure this is the appropriate fix. EventReader is in mapreduce core and isn't necessarily restricted to the AM reading a prior attempt's history file. This class is also used by Rumen, for example.

        I'm wondering if the generalized exception handling should be restricted to just the use case in MRAppMaster where it is trying to read the prior history and AMInfos. Then we're targeting the use case we care about and won't affect the others.

        Show
        jlowe Jason Lowe added a comment - Thanks for the report and patch, Bibin! However I'm not sure this is the appropriate fix. EventReader is in mapreduce core and isn't necessarily restricted to the AM reading a prior attempt's history file. This class is also used by Rumen, for example. I'm wondering if the generalized exception handling should be restricted to just the use case in MRAppMaster where it is trying to read the prior history and AMInfos. Then we're targeting the use case we care about and won't affect the others.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 0s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 7m 43s trunk passed
        +1 compile 0m 20s trunk passed with JDK v1.8.0_66
        +1 compile 0m 24s trunk passed with JDK v1.7.0_85
        +1 checkstyle 0m 13s trunk passed
        +1 mvnsite 0m 31s trunk passed
        +1 mvneclipse 0m 14s trunk passed
        -1 findbugs 1m 8s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings.
        +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66
        +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85
        +1 mvninstall 0m 28s the patch passed
        +1 compile 0m 20s the patch passed with JDK v1.8.0_66
        +1 javac 0m 20s the patch passed
        +1 compile 0m 25s the patch passed with JDK v1.7.0_85
        +1 javac 0m 25s the patch passed
        +1 checkstyle 0m 12s the patch passed
        +1 mvnsite 0m 31s the patch passed
        +1 mvneclipse 0m 14s the patch passed
        +1 whitespace 0m 0s Patch has no whitespace issues.
        +1 findbugs 1m 16s the patch passed
        +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66
        +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85
        +1 unit 1m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66.
        +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_85.
        -1 asflicense 0m 22s Patch generated 1 ASF License warnings.
        30m 32s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:0ca8df7
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773665/0001-MAPREDUCE-6554.patch
        JIRA Issue MAPREDUCE-6554
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux a2a3ec253e6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / a30eccb
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html
        JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/testReport/
        asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/artifact/patchprocess/patch-asflicense-problems.txt
        modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
        Max memory used 75MB
        Powered by Apache Yetus http://yetus.apache.org
        Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 7m 43s trunk passed +1 compile 0m 20s trunk passed with JDK v1.8.0_66 +1 compile 0m 24s trunk passed with JDK v1.7.0_85 +1 checkstyle 0m 13s trunk passed +1 mvnsite 0m 31s trunk passed +1 mvneclipse 0m 14s trunk passed -1 findbugs 1m 8s hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core in trunk has 2 extant Findbugs warnings. +1 javadoc 0m 20s trunk passed with JDK v1.8.0_66 +1 javadoc 0m 25s trunk passed with JDK v1.7.0_85 +1 mvninstall 0m 28s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_66 +1 javac 0m 20s the patch passed +1 compile 0m 25s the patch passed with JDK v1.7.0_85 +1 javac 0m 25s the patch passed +1 checkstyle 0m 12s the patch passed +1 mvnsite 0m 31s the patch passed +1 mvneclipse 0m 14s the patch passed +1 whitespace 0m 0s Patch has no whitespace issues. +1 findbugs 1m 16s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_66 +1 javadoc 0m 25s the patch passed with JDK v1.7.0_85 +1 unit 1m 35s hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_66. +1 unit 1m 56s hadoop-mapreduce-client-core in the patch passed with JDK v1.7.0_85. -1 asflicense 0m 22s Patch generated 1 ASF License warnings. 30m 32s Subsystem Report/Notes Docker Image:yetus/hadoop:0ca8df7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12773665/0001-MAPREDUCE-6554.patch JIRA Issue MAPREDUCE-6554 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux a2a3ec253e6e 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a30eccb findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-warnings.html JDK v1.7.0_85 Test Results https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/testReport/ asflicense https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core U: hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core Max memory used 75MB Powered by Apache Yetus http://yetus.apache.org Console output https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6179/console This message was automatically generated.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        Two failures cases of i faced during test

        1. Null pointer in readline()
        2. During Schema.parse() AvroException was thrown.

        Both these cases caused failure of current application attempt.
        Handling both the cases added patch . please do review.

        Show
        bibinchundatt Bibin A Chundatt added a comment - Two failures cases of i faced during test Null pointer in readline() During Schema.parse() AvroException was thrown. Both these cases caused failure of current application attempt. Handling both the cases added patch . please do review.
        Hide
        bibinchundatt Bibin A Chundatt added a comment -

        i think fix should throw IOException if null==in.readLine() before passing the string to Schema.parse()

        Show
        bibinchundatt Bibin A Chundatt added a comment - i think fix should throw IOException if null==in.readLine() before passing the string to Schema.parse()

          People

          • Assignee:
            bibinchundatt Bibin A Chundatt
            Reporter:
            bibinchundatt Bibin A Chundatt
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development