Oozie
  1. Oozie
  2. OOZIE-714

Fix Pig testcases to use Hadoop 0.23.X

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Oozie testcases TestPigActionExecutor and TestPigMain are failing due to
      1. the error "check value for mapreduce.framework.name", which is to be set to "yarn" in the conf.
      2. pig jar pulled from the online repositories is not the correct one required against hadoop 23

      Oozie version 3.1.4

        Activity

        Mona Chitnis created issue -
        Hide
        Mona Chitnis added a comment -

        For 3.1.4, the tests can pass by making the corresponding changes
        1. adding mapreduce.framework.name=yarn to the job conf
        2. running the tests against the pig-0.9.2 jar copied manually from the lib-hadoop23 subdirectory. Suggestions on how to mavenize this?

        The PigAE test is also failing on trunk with the following surefire-report

        Test set: org.apache.oozie.action.hadoop.TestPigActionExecutor
        3 -------------------------------------------------------------------------------
        4 Tests run: 9, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 152.759 sec <<< FAILURE!
        5 testExecutionStats(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 25.457 sec <<< FAILURE!
        6 junit.framework.AssertionFailedError
        7 at junit.framework.Assert.fail(Assert.java:47)
        8 at junit.framework.Assert.assertTrue(Assert.java:20)
        9 at junit.framework.Assert.assertTrue(Assert.java:27)
        10 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats(TestPigActionExecutor.java:230)
        11
        12 testExternalChildIds(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 19.045 sec <<< FAILURE!
        13 junit.framework.ComparisonFailure: expected:<SUCCEED...> but was:<FAILED/KILL...>
        14 at junit.framework.Assert.assertEquals(Assert.java:81)
        15 at junit.framework.Assert.assertEquals(Assert.java:87)
        16 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds(TestPigActionExecutor.java:271)
        17
        18 testExecutionStatsWithRetrieveStatsFalse(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 17.848 sec <<< FAILURE!
        19 junit.framework.ComparisonFailure: expected:<SUCCEED...> but was:<FAILED/KILL...>
        20 at junit.framework.Assert.assertEquals(Assert.java:81)
        21 at junit.framework.Assert.assertEquals(Assert.java:87)
        22 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse(TestPigActionExecutor.java:323)
        23
        24 testPig(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 21.063 sec <<< FAILURE!
        25 junit.framework.AssertionFailedError
        26 at junit.framework.Assert.fail(Assert.java:47)
        27 at junit.framework.Assert.assertTrue(Assert.java:20)
        28 at junit.framework.Assert.assertTrue(Assert.java:27)
        29 at org.apache.oozie.action.hadoop.TestPigActionExecutor._testSubmit(TestPigActionExecutor.java:196)
        30 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig(TestPigActionExecutor.java:347)
        31
        32 testUdfPig(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 18.146 sec <<< FAILURE!
        33 junit.framework.AssertionFailedError
        34 at junit.framework.Assert.fail(Assert.java:47)
        35 at junit.framework.Assert.assertTrue(Assert.java:20)
        36 at junit.framework.Assert.assertTrue(Assert.java:27)
        37 at org.apache.oozie.action.hadoop.TestPigActionExecutor._testSubmit(TestPigActionExecutor.java:196)
        38 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig(TestPigActionExecutor.java:415)

        Show
        Mona Chitnis added a comment - For 3.1.4, the tests can pass by making the corresponding changes 1. adding mapreduce.framework.name=yarn to the job conf 2. running the tests against the pig-0.9.2 jar copied manually from the lib-hadoop23 subdirectory. Suggestions on how to mavenize this? The PigAE test is also failing on trunk with the following surefire-report Test set: org.apache.oozie.action.hadoop.TestPigActionExecutor 3 ------------------------------------------------------------------------------- 4 Tests run: 9, Failures: 5, Errors: 0, Skipped: 0, Time elapsed: 152.759 sec <<< FAILURE! 5 testExecutionStats(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 25.457 sec <<< FAILURE! 6 junit.framework.AssertionFailedError 7 at junit.framework.Assert.fail(Assert.java:47) 8 at junit.framework.Assert.assertTrue(Assert.java:20) 9 at junit.framework.Assert.assertTrue(Assert.java:27) 10 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStats(TestPigActionExecutor.java:230) 11 12 testExternalChildIds(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 19.045 sec <<< FAILURE! 13 junit.framework.ComparisonFailure: expected:<SUCCEED...> but was:<FAILED/KILL...> 14 at junit.framework.Assert.assertEquals(Assert.java:81) 15 at junit.framework.Assert.assertEquals(Assert.java:87) 16 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExternalChildIds(TestPigActionExecutor.java:271) 17 18 testExecutionStatsWithRetrieveStatsFalse(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 17.848 sec <<< FAILURE! 19 junit.framework.ComparisonFailure: expected:<SUCCEED...> but was:<FAILED/KILL...> 20 at junit.framework.Assert.assertEquals(Assert.java:81) 21 at junit.framework.Assert.assertEquals(Assert.java:87) 22 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testExecutionStatsWithRetrieveStatsFalse(TestPigActionExecutor.java:323) 23 24 testPig(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 21.063 sec <<< FAILURE! 25 junit.framework.AssertionFailedError 26 at junit.framework.Assert.fail(Assert.java:47) 27 at junit.framework.Assert.assertTrue(Assert.java:20) 28 at junit.framework.Assert.assertTrue(Assert.java:27) 29 at org.apache.oozie.action.hadoop.TestPigActionExecutor._testSubmit(TestPigActionExecutor.java:196) 30 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testPig(TestPigActionExecutor.java:347) 31 32 testUdfPig(org.apache.oozie.action.hadoop.TestPigActionExecutor) Time elapsed: 18.146 sec <<< FAILURE! 33 junit.framework.AssertionFailedError 34 at junit.framework.Assert.fail(Assert.java:47) 35 at junit.framework.Assert.assertTrue(Assert.java:20) 36 at junit.framework.Assert.assertTrue(Assert.java:27) 37 at org.apache.oozie.action.hadoop.TestPigActionExecutor._testSubmit(TestPigActionExecutor.java:196) 38 at org.apache.oozie.action.hadoop.TestPigActionExecutor.testUdfPig(TestPigActionExecutor.java:415)
        Hide
        Mona Chitnis added a comment -

        Reviewboard patch (combo for OOZIE-714, 716 and 3.1 backport of OOZIE-691)

        https://reviews.apache.org/r/3944/

        Show
        Mona Chitnis added a comment - Reviewboard patch (combo for OOZIE-714 , 716 and 3.1 backport of OOZIE-691 ) https://reviews.apache.org/r/3944/
        Gavin made changes -
        Field Original Value New Value
        Workflow jira [ 12654633 ] no-reopen-closed, patch-avail [ 12655950 ]
        Hide
        Mona Chitnis added a comment -

        For the patch committed in trunk with this JIRA ID (also available under the same JIRA ID in Apache ReviewBoard system), I grant license to ASF for inclusion in ASF works (as per the Apache License §5) 

        Show
        Mona Chitnis added a comment - For the patch committed in trunk with this JIRA ID (also available under the same JIRA ID in Apache ReviewBoard system), I grant license to ASF for inclusion in ASF works (as per the Apache License §5) 
        Mona Chitnis made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Assignee Mona Chitnis [ chitnis ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Mona Chitnis
            Reporter:
            Mona Chitnis
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 2h
              2h
              Remaining:
              Remaining Estimate - 2h
              2h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development