Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2531

org.apache.hadoop.mapred.jobcontrol.getAssignedJobID throw class cast exception

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.22.0
    • Fix Version/s: 0.22.0, 0.23.0
    • Component/s: client
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      When using a combination of the mapred and mapreduce APIs (PIG) it is possible to have the following exception

      Caused by: java.lang.ClassCastException: org.apache.hadoop.mapreduce.JobID cannot be cast to
      org.apache.hadoop.mapred.JobID
      at org.apache.hadoop.mapred.jobcontrol.Job.getAssignedJobID(Job.java:71)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:239)
      at org.apache.pig.PigServer.launchPlan(PigServer.java:1325)
      ... 29 more

      This is because the JobID is just downcast. It should be calling JobID.downgrade

      1. MR-2531-V1-trunk.patch
        2 kB
        Robert Joseph Evans
      2. MR-2531-yarn-v1.patch
        2 kB
        Robert Joseph Evans

        Activity

        Hide
        Robert Joseph Evans added a comment -

        A quick fix to call JobID.downgrade instead of just casting. Included with it are unit tests to validate the results.

        Show
        Robert Joseph Evans added a comment - A quick fix to call JobID.downgrade instead of just casting. Included with it are unit tests to validate the results.
        Hide
        Hadoop QA added a comment -

        +1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480276/MR-2531-V1-trunk.patch
        against trunk revision 1126801.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed core unit tests.

        +1 contrib tests. The patch passed contrib unit tests.

        +1 system test framework. The patch passed system test framework compile.

        Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//testReport/
        Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480276/MR-2531-V1-trunk.patch against trunk revision 1126801. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//testReport/ Findbugs warnings: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/302//console This message is automatically generated.
        Hide
        Mahadev konar added a comment -

        +1 the change looks good. Ill commit this to trunk. Robert, can you also provide patch against MR-279 branch?

        Show
        Mahadev konar added a comment - +1 the change looks good. Ill commit this to trunk. Robert, can you also provide patch against MR-279 branch?
        Hide
        Robert Joseph Evans added a comment -

        I will work on that.

        Show
        Robert Joseph Evans added a comment - I will work on that.
        Hide
        Robert Joseph Evans added a comment -

        Attached is the yarn patch. The test is still under ant, even though the code has been moved to mvn. Not sure if this is an issue or not. Also not all of the ant contrib tests even compile. My test runs and passes though.

        Show
        Robert Joseph Evans added a comment - Attached is the yarn patch. The test is still under ant, even though the code has been moved to mvn. Not sure if this is an issue or not. Also not all of the ant contrib tests even compile. My test runs and passes though.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12480569/MR-2531-yarn-v1.patch
        against trunk revision 1127444.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified tests.

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/309//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12480569/MR-2531-yarn-v1.patch against trunk revision 1127444. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/309//console This message is automatically generated.
        Hide
        Arun C Murthy added a comment -

        I just committed this to both trunk and MR-279. Thanks Robert!

        Show
        Arun C Murthy added a comment - I just committed this to both trunk and MR-279. Thanks Robert!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #711 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/711/)
        MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. Contributed by Robert Evans.

        acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130992
        Files :

        • /hadoop/mapreduce/trunk/CHANGES.txt
        • /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
        • /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/jobcontrol/Job.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #711 (See https://builds.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/711/ ) MAPREDUCE-2531 . Fixed jobcontrol to downgrade JobID. Contributed by Robert Evans. acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1130992 Files : /hadoop/mapreduce/trunk/CHANGES.txt /hadoop/mapreduce/trunk/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java /hadoop/mapreduce/trunk/src/java/org/apache/hadoop/mapred/jobcontrol/Job.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #722 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/722/ )
        Hide
        Konstantin Shvachko added a comment -

        Reopening for committing to 0.22

        Show
        Konstantin Shvachko added a comment - Reopening for committing to 0.22
        Hide
        Po Cheung added a comment -

        Tested trunk patch against 0.22 and it passed. Please commit to 0.22.

        Show
        Po Cheung added a comment - Tested trunk patch against 0.22 and it passed. Please commit to 0.22.
        Hide
        Konstantin Shvachko added a comment -

        Committed to branch 0.22.
        Moved changes to 0.22 section in trunk and branch 0.23.

        Show
        Konstantin Shvachko added a comment - Committed to branch 0.22. Moved changes to 0.22 section in trunk and branch 0.23.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #1016 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1016/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #1016 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1016/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #1094 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1094/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #1094 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1094/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #1034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1034/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #1034 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1034/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #821 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/821/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179077
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #30 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/30/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179077 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-22-branch #80 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/80/)
        MAPREDUCE-2531. Fixed jobcontrol to downgrade JobID. Contributed by Robert Evans.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179063
        Files :

        • /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt
        • /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/jobcontrol/Job.java
        • /hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-22-branch #80 (See https://builds.apache.org/job/Hadoop-Mapreduce-22-branch/80/ ) MAPREDUCE-2531 . Fixed jobcontrol to downgrade JobID. Contributed by Robert Evans. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179063 Files : /hadoop/common/branches/branch-0.22/mapreduce/CHANGES.txt /hadoop/common/branches/branch-0.22/mapreduce/src/java/org/apache/hadoop/mapred/jobcontrol/Job.java /hadoop/common/branches/branch-0.22/mapreduce/src/test/mapred/org/apache/hadoop/mapred/jobcontrol/TestJobControl.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179077
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-0.23-Build #37 (See https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/37/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179077 Files : /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/)
        Move MAPREDUCE-2531 changes to Release 0.22 section.

        shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078
        Files :

        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #851 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/851/ ) Move MAPREDUCE-2531 changes to Release 0.22 section. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1179078 Files : /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt

          People

          • Assignee:
            Robert Joseph Evans
            Reporter:
            Robert Joseph Evans
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development