Uploaded image for project: 'Oozie'
  1. Oozie
  2. OOZIE-1551

Change hadoop-2 profile to use 2.2.0

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: trunk, 4.0.1
    • Fix Version/s: 4.1.0, 4.0.1
    • Component/s: build
    • Labels:
      None

      Description

      We should change the hadoop-2 profile to use 2.2.0 (GA) instead of 2.2.0-SNAPSHOT now that its been released.

      1. sharedlib_pig_pom_exclusion.patch
        0.6 kB
        Mona Chitnis
      2. OOZIE-1551.patch
        4 kB
        Robert Kanter
      3. OOZIE-1551.patch
        3 kB
        Robert Kanter
      4. OOZIE-1551.patch
        4 kB
        Robert Kanter
      5. OOZIE-1551.patch
        10 kB
        Robert Kanter
      6. OOZIE-1551_branch-4.patch
        4 kB
        Robert Kanter

        Issue Links

          Activity

          Hide
          rkanter Robert Kanter added a comment -

          The patch uses the 2.1.1-beta Hadoop when "hadoop-2" profile is activated.

          I don't have time to look into this at the moment, but these tests failed:

          org.apache.oozie.command.bundle.TestBundleStartXCommand.testBundleStartWithFailedCoordinator
          org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithMaxStatsSizeLimit
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testPigError
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig
          org.apache.oozie.action.hadoop.TestPigMain.testPigScript
          org.apache.oozie.action.hadoop.TestPigMain.testEmbeddedPigWithinPython
          org.apache.oozie.action.hadoop.TestPigMainWithOldAPI.testPigScript
          

          I'll try to get to this soon, but I figured I'd put the patch up in case anyone wanted to try to fix these tests in the meantime.

          Show
          rkanter Robert Kanter added a comment - The patch uses the 2.1.1-beta Hadoop when "hadoop-2" profile is activated. I don't have time to look into this at the moment, but these tests failed: org.apache.oozie.command.bundle.TestBundleStartXCommand.testBundleStartWithFailedCoordinator org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithMaxStatsSizeLimit org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig org.apache.oozie.action.hadoop.TestPigActionExecutor.testPigError org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig org.apache.oozie.action.hadoop.TestPigMain.testPigScript org.apache.oozie.action.hadoop.TestPigMain.testEmbeddedPigWithinPython org.apache.oozie.action.hadoop.TestPigMainWithOldAPI.testPigScript I'll try to get to this soon, but I figured I'd put the patch up in case anyone wanted to try to fix these tests in the meantime.
          Hide
          rkanter Robert Kanter added a comment -

          Also, I had to explicitly put protobufs 2.5.0 as a dependency in the hadoop-2 pom or it was otherwise picking up 2.4.1, which doesn't work with Hadoop 2.

          Show
          rkanter Robert Kanter added a comment - Also, I had to explicitly put protobufs 2.5.0 as a dependency in the hadoop-2 pom or it was otherwise picking up 2.4.1, which doesn't work with Hadoop 2.
          Hide
          rkanter Robert Kanter added a comment -

          Updating JIRA to use 2.2.0 (GA) instead of 2.1.1-beta.

          Show
          rkanter Robert Kanter added a comment - Updating JIRA to use 2.2.0 (GA) instead of 2.1.1-beta.
          Hide
          rkanter Robert Kanter added a comment -

          The new patch uses Hadoop 2.2.0. These are the tests that failed:

          org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testTimeOutWithUnresolvedMissingDependencies
          org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithMaxStatsSizeLimit
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testPigError
          org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig
          org.apache.oozie.action.hadoop.TestPigMain.testPigScript
          org.apache.oozie.action.hadoop.TestPigMain.testEmbeddedPigWithinPython
          org.apache.oozie.action.hadoop.TestPigMainWithOldAPI.testPigScript
          

          I haven't had a chance to really look into the causes yet.

          Show
          rkanter Robert Kanter added a comment - The new patch uses Hadoop 2.2.0. These are the tests that failed: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testTimeOutWithUnresolvedMissingDependencies org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithMaxStatsSizeLimit org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig org.apache.oozie.action.hadoop.TestPigActionExecutor.testPigError org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig org.apache.oozie.action.hadoop.TestPigMain.testPigScript org.apache.oozie.action.hadoop.TestPigMain.testEmbeddedPigWithinPython org.apache.oozie.action.hadoop.TestPigMainWithOldAPI.testPigScript I haven't had a chance to really look into the causes yet.
          Hide
          rkanter Robert Kanter added a comment -
          • I think TestCoordPushDependencyCheckXCommand was just being flakey
          • TestShareLibService#testAddShareLibDistributedCache fails because it checks the distributed cache for 2 jars that were added, but finds 4 jars. The extra 2 jars are:
            hdfs://localhost:64929/user/rkanter/target/MiniMRCluster_1118556015-tmpDir/MRAppJar.jar
            hdfs://localhost:64929/user/rkanter/target/MiniMRCluster_1118556015-tmpDir/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar
            

            which look to be something Hadoop is adding. I think we simply need to adjust the test to account for this.

          • I'm not really sure what's wrong with the pig tests, the error I see when running TestPigMain is this:
            ERROR 2244: Job failed, hadoop does not return any error message
            org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message
            	at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:141)
            	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
            	at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166)
            	at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
            	at org.apache.pig.Main.run(Main.java:430)
            	at org.apache.pig.PigRunner.run(PigRunner.java:49)
            	at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:284)
            	at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:224)
            	at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38)
            	at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
            	at org.apache.oozie.action.hadoop.TestPigMain.call(TestPigMain.java:117)
            	at org.apache.oozie.action.hadoop.TestPigMain.call(TestPigMain.java:37)
            	at org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:32)
            	at org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:30)
            	at java.security.AccessController.doPrivileged(Native Method)
            	at javax.security.auth.Subject.doAs(Subject.java:394)
            	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
            	at org.apache.oozie.action.hadoop.MainTestCase.execute(MainTestCase.java:30)
            	at org.apache.oozie.action.hadoop.PigTestCase.testPigScript(PigTestCase.java:50)
            	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            	at java.lang.reflect.Method.invoke(Method.java:597)
            	at junit.framework.TestCase.runTest(TestCase.java:168)
            	at junit.framework.TestCase.runBare(TestCase.java:134)
            	at junit.framework.TestResult$1.protect(TestResult.java:110)
            	at junit.framework.TestResult.runProtected(TestResult.java:128)
            	at junit.framework.TestResult.run(TestResult.java:113)
            	at junit.framework.TestCase.run(TestCase.java:124)
            	at junit.framework.TestSuite.runTest(TestSuite.java:243)
            	at junit.framework.TestSuite.run(TestSuite.java:238)
            	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
            	at org.apache.maven.surefire.junitcore.ClassDemarcatingRunner.run(ClassDemarcatingRunner.java:58)
            	at org.junit.runners.Suite.runChild(Suite.java:128)
            	at org.junit.runners.Suite.runChild(Suite.java:24)
            	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
            	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
            	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
            	at java.lang.Thread.run(Thread.java:695)
            

            Looking earlier, I see that the Yarn container failed to launch with the unhelpful exception:

            2013-10-18 17:30:18,316 [AsyncDispatcher event handler] WARN  org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger  - USER=test	OPERATION=Application Finished - Failed	TARGET=RMAppManager	RESULT=FAILURE	DESCRIPTION=App failed with state: FAILED	PERMISSIONS=Application application_1382142539659_0001 failed 2 times due to AM Container for appattempt_1382142539659_0001_000002 exited with  exitCode: 1 due to: Exception from container-launch: 
            org.apache.hadoop.util.Shell$ExitCodeException: 
            	at org.apache.hadoop.util.Shell.runCommand(Shell.java:464)
            	at org.apache.hadoop.util.Shell.run(Shell.java:379)
            	at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
            	at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
            	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283)
            	at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79)
            	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
            	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
            	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
            	at java.lang.Thread.run(Thread.java:695)
            

            It will require a deeper look to figure out the Pig issue.

          Show
          rkanter Robert Kanter added a comment - I think TestCoordPushDependencyCheckXCommand was just being flakey TestShareLibService#testAddShareLibDistributedCache fails because it checks the distributed cache for 2 jars that were added, but finds 4 jars. The extra 2 jars are: hdfs://localhost:64929/user/rkanter/target/MiniMRCluster_1118556015-tmpDir/MRAppJar.jar hdfs://localhost:64929/user/rkanter/target/MiniMRCluster_1118556015-tmpDir/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar which look to be something Hadoop is adding. I think we simply need to adjust the test to account for this. I'm not really sure what's wrong with the pig tests, the error I see when running TestPigMain is this: ERROR 2244: Job failed, hadoop does not return any error message org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job failed, hadoop does not return any error message at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:141) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:166) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84) at org.apache.pig.Main.run(Main.java:430) at org.apache.pig.PigRunner.run(PigRunner.java:49) at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:284) at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:224) at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:38) at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76) at org.apache.oozie.action.hadoop.TestPigMain.call(TestPigMain.java:117) at org.apache.oozie.action.hadoop.TestPigMain.call(TestPigMain.java:37) at org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:32) at org.apache.oozie.action.hadoop.MainTestCase$1.run(MainTestCase.java:30) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:394) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.oozie.action.hadoop.MainTestCase.execute(MainTestCase.java:30) at org.apache.oozie.action.hadoop.PigTestCase.testPigScript(PigTestCase.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:168) at junit.framework.TestCase.runBare(TestCase.java:134) at junit.framework.TestResult$1.protect(TestResult.java:110) at junit.framework.TestResult.runProtected(TestResult.java:128) at junit.framework.TestResult.run(TestResult.java:113) at junit.framework.TestCase.run(TestCase.java:124) at junit.framework.TestSuite.runTest(TestSuite.java:243) at junit.framework.TestSuite.run(TestSuite.java:238) at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83) at org.apache.maven.surefire.junitcore.ClassDemarcatingRunner.run(ClassDemarcatingRunner.java:58) at org.junit.runners.Suite.runChild(Suite.java:128) at org.junit.runners.Suite.runChild(Suite.java:24) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) Looking earlier, I see that the Yarn container failed to launch with the unhelpful exception: 2013-10-18 17:30:18,316 [AsyncDispatcher event handler] WARN org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger - USER=test OPERATION=Application Finished - Failed TARGET=RMAppManager RESULT=FAILURE DESCRIPTION=App failed with state: FAILED PERMISSIONS=Application application_1382142539659_0001 failed 2 times due to AM Container for appattempt_1382142539659_0001_000002 exited with exitCode: 1 due to: Exception from container-launch: org.apache.hadoop.util.Shell$ExitCodeException: at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) at org.apache.hadoop.util.Shell.run(Shell.java:379) at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589) at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:283) at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:79) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) at java.lang.Thread.run(Thread.java:695) It will require a deeper look to figure out the Pig issue.
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-1551

          Cleaning local svn workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          +1 TESTS
          . Tests run: 1347
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/850/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-1551 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1347 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/850/
          Hide
          rohini Rohini Palaniswamy added a comment -

          Mona Chitnis
          Can you take a look at the failing pig tests?

          Show
          rohini Rohini Palaniswamy added a comment - Mona Chitnis Can you take a look at the failing pig tests?
          Hide
          rkanter Robert Kanter added a comment -

          Would it be easier to fix the failing tests if we just push this change in now? Then people trying to look at the failing tests don't have to manually apply the patch.

          Show
          rkanter Robert Kanter added a comment - Would it be easier to fix the failing tests if we just push this change in now? Then people trying to look at the failing tests don't have to manually apply the patch.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Thought of getting everything fixed in one go instead of creating multiple small jiras. If it takes more time to resolve the issue then we can go ahead and commit this one.

          Show
          rohini Rohini Palaniswamy added a comment - Thought of getting everything fixed in one go instead of creating multiple small jiras. If it takes more time to resolve the issue then we can go ahead and commit this one.
          Hide
          rohini Rohini Palaniswamy added a comment -

          Issue is either avro jar is missing from classpath or correct version of avro jar is not in classpath.

          Found this in AM logs:
          2013-10-22 11:34:44,917 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
          java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1500()Ljava/lang/ThreadLocal;
          at org.apache.avro.Schema$Parser.parse(Schema.java:978)
          at org.apache.avro.Schema$Parser.parse(Schema.java:971)
          at org.apache.hadoop.mapreduce.jobhistory.AMStarted.<clinit>(AMStarted.java:10)
          at org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent.<init>(AMStartedEvent.java:36)
          at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1018)
          at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
          at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
          at java.security.AccessController.doPrivileged(Native Method)
          at javax.security.auth.Subject.doAs(Subject.java:396)
          at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
          at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
          at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)

          To debug hadoop 2 issues, set the following 2 properties in XTestCase in setUpEmbeddedHadoop() conf object.

          conf.set("yarn.nodemanager.delete.debug-delay-sec", "3600");
          conf.set("yarn.app.mapreduce.am.log.level", "DEBUG");

          Show
          rohini Rohini Palaniswamy added a comment - Issue is either avro jar is missing from classpath or correct version of avro jar is not in classpath. Found this in AM logs: 2013-10-22 11:34:44,917 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster java.lang.NoSuchMethodError: org.apache.avro.Schema.access$1500()Ljava/lang/ThreadLocal; at org.apache.avro.Schema$Parser.parse(Schema.java:978) at org.apache.avro.Schema$Parser.parse(Schema.java:971) at org.apache.hadoop.mapreduce.jobhistory.AMStarted.<clinit>(AMStarted.java:10) at org.apache.hadoop.mapreduce.jobhistory.AMStartedEvent.<init>(AMStartedEvent.java:36) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1018) at org.apache.hadoop.service.AbstractService.start(AbstractService.java:193) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441) at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374) To debug hadoop 2 issues, set the following 2 properties in XTestCase in setUpEmbeddedHadoop() conf object. conf.set("yarn.nodemanager.delete.debug-delay-sec", "3600"); conf.set("yarn.app.mapreduce.am.log.level", "DEBUG");
          Hide
          rohini Rohini Palaniswamy added a comment -

          Issue is wrong version of avro jar in classpath as it is NoSuchMethodError and not CNFE. Mona is going to follow up and fix it.

          Show
          rohini Rohini Palaniswamy added a comment - Issue is wrong version of avro jar in classpath as it is NoSuchMethodError and not CNFE. Mona is going to follow up and fix it.
          Hide
          chitnis Mona Chitnis added a comment -

          Thanks to Rohini's help in debugging the AM logs, the avro jar issue was isolated. I analyzed the dependency tree, and the pig jar was transitively pulling in avro-1.3.2, whereas the rest of the hadoop-2 jars were pulling in the right jar avro-1.7.4 and including in classpath. So we simply needed to add the 'avro' exclusion in pig's pom file.

          Attaching the patch. Robert, please include it in yours.

          Show
          chitnis Mona Chitnis added a comment - Thanks to Rohini's help in debugging the AM logs, the avro jar issue was isolated. I analyzed the dependency tree, and the pig jar was transitively pulling in avro-1.3.2, whereas the rest of the hadoop-2 jars were pulling in the right jar avro-1.7.4 and including in classpath. So we simply needed to add the 'avro' exclusion in pig's pom file. Attaching the patch. Robert, please include it in yours.
          Hide
          chitnis Mona Chitnis added a comment -

          The failing TestPigMain and TestPigActionExecutor unit tests pass now

          Show
          chitnis Mona Chitnis added a comment - The failing TestPigMain and TestPigActionExecutor unit tests pass now
          Hide
          rkanter Robert Kanter added a comment -

          Thanks for figuring that out Rohini and Mona. I've incorporated it into the newest patch. I'm running all of the unit tests against hadoop-2 again just to double check and I'll post the results when its done.

          Show
          rkanter Robert Kanter added a comment - Thanks for figuring that out Rohini and Mona. I've incorporated it into the newest patch. I'm running all of the unit tests against hadoop-2 again just to double check and I'll post the results when its done.
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-1551

          Cleaning local svn workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          +1 TESTS
          . Tests run: 1347
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/857/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-1551 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1347 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/857/
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-1551

          Cleaning local svn workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          -1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . -1 the patch does not add/modify any testcase
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          -1 COMPILE
          . -1 HEAD does not compile
          . -1 patch does not compile
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          -1 TESTS - patch does not compile, cannot run testcases
          -1 DISTRO
          . -1 distro tarball fails with the patch

          ----------------------------
          -1 Overall result, please check the reported -1(s)

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/858/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-1551 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN -1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . -1 the patch does not add/modify any testcase +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings -1 COMPILE . -1 HEAD does not compile . -1 patch does not compile . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files -1 TESTS - patch does not compile, cannot run testcases -1 DISTRO . -1 distro tarball fails with the patch ---------------------------- -1 Overall result, please check the reported -1(s) The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/858/
          Hide
          rkanter Robert Kanter added a comment -

          Yay, now there's only 2 failing tests:

          org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testTimeOutWithUnresolvedMissingDependencies 
          org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache 
          

          I'll try to take a look at them soon

          Show
          rkanter Robert Kanter added a comment - Yay, now there's only 2 failing tests: org.apache.oozie.command.coord.TestCoordPushDependencyCheckXCommand.testTimeOutWithUnresolvedMissingDependencies org.apache.oozie.service.TestShareLibService.testAddShareLibDistributedCache I'll try to take a look at them soon
          Hide
          rkanter Robert Kanter added a comment -

          The new patch fixes the TestShareLibService; I just had it allow either 2 or 4 jars in the cache so it will work against Hadoop 1 or 2. I also noticed that it was improperly dealing with the Services where if a test fails the Services will continue running (I fixed all of these before, but this was a new test class); so I fixed this too.

          TestCoordPushDependencyCheckXCommand doesn't seem to be a problem anymore.

          So now all tests are passing against Hadoop 2.2.0

          Show
          rkanter Robert Kanter added a comment - The new patch fixes the TestShareLibService ; I just had it allow either 2 or 4 jars in the cache so it will work against Hadoop 1 or 2. I also noticed that it was improperly dealing with the Services where if a test fails the Services will continue running (I fixed all of these before, but this was a new test class); so I fixed this too. TestCoordPushDependencyCheckXCommand doesn't seem to be a problem anymore. So now all tests are passing against Hadoop 2.2.0
          Hide
          rohini Rohini Palaniswamy added a comment -

          +1

          Show
          rohini Rohini Palaniswamy added a comment - +1
          Hide
          hadoopqa Hadoop QA added a comment -

          Testing JIRA OOZIE-1551

          Cleaning local svn workspace

          ----------------------------

          +1 PATCH_APPLIES
          +1 CLEAN
          +1 RAW_PATCH_ANALYSIS
          . +1 the patch does not introduce any @author tags
          . +1 the patch does not introduce any tabs
          . +1 the patch does not introduce any trailing spaces
          . +1 the patch does not introduce any line longer than 132
          . +1 the patch does adds/modifies 1 testcase(s)
          +1 RAT
          . +1 the patch does not seem to introduce new RAT warnings
          +1 JAVADOC
          . +1 the patch does not seem to introduce new Javadoc warnings
          +1 COMPILE
          . +1 HEAD compiles
          . +1 patch compiles
          . +1 the patch does not seem to introduce new javac warnings
          +1 BACKWARDS_COMPATIBILITY
          . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations
          . +1 the patch does not modify JPA files
          +1 TESTS
          . Tests run: 1347
          +1 DISTRO
          . +1 distro tarball builds with the patch

          ----------------------------
          +1 Overall result, good!, no -1s

          The full output of the test-patch run is available at

          . https://builds.apache.org/job/oozie-trunk-precommit-build/861/

          Show
          hadoopqa Hadoop QA added a comment - Testing JIRA OOZIE-1551 Cleaning local svn workspace ---------------------------- +1 PATCH_APPLIES +1 CLEAN +1 RAW_PATCH_ANALYSIS . +1 the patch does not introduce any @author tags . +1 the patch does not introduce any tabs . +1 the patch does not introduce any trailing spaces . +1 the patch does not introduce any line longer than 132 . +1 the patch does adds/modifies 1 testcase(s) +1 RAT . +1 the patch does not seem to introduce new RAT warnings +1 JAVADOC . +1 the patch does not seem to introduce new Javadoc warnings +1 COMPILE . +1 HEAD compiles . +1 patch compiles . +1 the patch does not seem to introduce new javac warnings +1 BACKWARDS_COMPATIBILITY . +1 the patch does not change any JPA Entity/Colum/Basic/Lob/Transient annotations . +1 the patch does not modify JPA files +1 TESTS . Tests run: 1347 +1 DISTRO . +1 distro tarball builds with the patch ---------------------------- +1 Overall result, good!, no -1s The full output of the test-patch run is available at . https://builds.apache.org/job/oozie-trunk-precommit-build/861/
          Hide
          rkanter Robert Kanter added a comment -

          I realized that for branch-4, the patch doesn't apply because it doesn't have TestShareLibService and the pom versions are different. I've rebased the patch on branch-4 and am currently running the tests just to make sure. It's similar enough to the trunk patch that I'll just commit both once its done running.

          Show
          rkanter Robert Kanter added a comment - I realized that for branch-4, the patch doesn't apply because it doesn't have TestShareLibService and the pom versions are different. I've rebased the patch on branch-4 and am currently running the tests just to make sure. It's similar enough to the trunk patch that I'll just commit both once its done running.
          Hide
          rkanter Robert Kanter added a comment -

          Thanks Rohini and Mona for your help. Committed to branch-4 and trunk!

          Show
          rkanter Robert Kanter added a comment - Thanks Rohini and Mona for your help. Committed to branch-4 and trunk!
          Hide
          rkanter Robert Kanter added a comment -

          Attaching branch-4 version of the patch for reference

          Show
          rkanter Robert Kanter added a comment - Attaching branch-4 version of the patch for reference
          Hide
          rkanter Robert Kanter added a comment -

          Closing issue; Oozie 4.0.1 is released

          Show
          rkanter Robert Kanter added a comment - Closing issue; Oozie 4.0.1 is released

            People

            • Assignee:
              rkanter Robert Kanter
              Reporter:
              rkanter Robert Kanter
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development