HBase
  1. HBase
  2. HBASE-8681

TestDriver#testDriver fails on hadoop 2.0.x

    Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      TestDriver#testDriver relies on System.exit() call from ProgramDriver#driver() because LauncherSecurityManager latches onto checkExit().

      In hadoop 2.0, ProgramDriver#driver() has the following code:

        public void driver(String[] argv) throws Throwable {
          if (run(argv) == -1) {
            System.exit(-1);
          }
        }
      

      Meaning, if the execution is successful, System.exit() call would be bypassed.

      This leads to test failure such as the following:

      https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/552/testReport/org.apache.hadoop.hbase.mapreduce/TestDriver/testDriver/

        Issue Links

          Activity

          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12585979/8681-v1.txt
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified tests.

          +1 hadoop1.0. The patch compiles against the hadoop 1.0 profile.

          +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 lineLengths. The patch does not introduce lines longer than 100

          +1 site. The mvn site goal succeeds with this patch.

          +1 core tests. The patch passed unit tests in .

          Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
          Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12585979/8681-v1.txt against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified tests. +1 hadoop1.0 . The patch compiles against the hadoop 1.0 profile. +1 hadoop2.0 . The patch compiles against the hadoop 2.0 profile. +1 javadoc . The javadoc tool did not generate any warning messages. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 lineLengths . The patch does not introduce lines longer than 100 +1 site . The mvn site goal succeeds with this patch. +1 core tests . The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/5919//console This message is automatically generated.
          Hide
          Ted Yu added a comment -

          Aleksey Gorshkov:
          What do you think ?

          Show
          Ted Yu added a comment - Aleksey Gorshkov : What do you think ?
          Hide
          Elliott Clark added a comment -

          Why not delete dead code. If it's not going to be run then removing it seems the best.

          Show
          Elliott Clark added a comment - Why not delete dead code. If it's not going to be run then removing it seems the best.
          Hide
          Ted Yu added a comment -

          I can remove the dead code in patch v2.

          Will wait for comment(s) from Nick and Aleksey.

          Show
          Ted Yu added a comment - I can remove the dead code in patch v2. Will wait for comment(s) from Nick and Aleksey.
          Hide
          Nick Dimiduk added a comment -

          I'll take a closer look at this in a few hours.

          Show
          Nick Dimiduk added a comment - I'll take a closer look at this in a few hours.
          Hide
          Nick Dimiduk added a comment -

          ProgramDriver on branch hadoop-2 contains the following:

            /**
             * API compatible with Hadoop 1.x
             */
            public void driver(String[] argv) throws Throwable {
              if (run(argv) == -1) {
                System.exit(-1);
              }
            }
          
            public int run(String[] args)
              throws Throwable 
            {
              // Make sure they gave us a program name.
              if (args.length == 0) {
                System.out.println("An example program must be given as the" + 
                                   " first argument.");
                printUsage(programs);
                return -1;
              }
          ...
          

          It will work on a future hadoop-2 release, just not on 2.0.2-alpha. The compatibility fix was brought in via

          281f9a2 * MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run 1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of ProgramDriver. Contributed by Zhijie Shen. svn merge --ignore-ancestry -c 1485085 ../../trunk/
          
          Show
          Nick Dimiduk added a comment - ProgramDriver on branch hadoop-2 contains the following: /** * API compatible with Hadoop 1.x */ public void driver(String[] argv) throws Throwable { if (run(argv) == -1) { System.exit(-1); } } public int run(String[] args) throws Throwable { // Make sure they gave us a program name. if (args.length == 0) { System.out.println("An example program must be given as the" + " first argument."); printUsage(programs); return -1; } ... It will work on a future hadoop-2 release, just not on 2.0.2-alpha. The compatibility fix was brought in via 281f9a2 * MAPREDUCE-5156. Change hadoop examples ProgramDriver to be able to run 1.x examples jar on top of YARN. This change breaks 0.23.x direct usages of ProgramDriver. Contributed by Zhijie Shen. svn merge --ignore-ancestry -c 1485085 ../../trunk/
          Hide
          Ted Yu added a comment -

          2.1.0-beta is not released yet.

          Before we switch to 2.1.0-beta, build on hadoop-2.0.x would be broken.

          Show
          Ted Yu added a comment - 2.1.0-beta is not released yet. Before we switch to 2.1.0-beta, build on hadoop-2.0.x would be broken.
          Hide
          Nick Dimiduk added a comment -

          Indeed. That commit isn't on the 2.0.5-alpha branch either. I'm running locally all the tests modified by HBASE-8534 against -Dhadoop.profile=2.0 and will post an addendum for trunk with any necessary changes on that ticket.

          Show
          Nick Dimiduk added a comment - Indeed. That commit isn't on the 2.0.5-alpha branch either. I'm running locally all the tests modified by HBASE-8534 against -Dhadoop.profile=2.0 and will post an addendum for trunk with any necessary changes on that ticket.
          Hide
          Nick Dimiduk added a comment -

          Posted addendum to HBASE-8534 which addresses two issues I found vs the hadoop2 profile.

          Show
          Nick Dimiduk added a comment - Posted addendum to HBASE-8534 which addresses two issues I found vs the hadoop2 profile.
          Hide
          Ted Yu added a comment -

          Fixed by addendum to HBASE-8534

          Show
          Ted Yu added a comment - Fixed by addendum to HBASE-8534

            People

            • Assignee:
              Ted Yu
              Reporter:
              Ted Yu
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development