Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-4498

Remove hsqldb jar from Hadoop runtime classpath

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.2-alpha
    • Fix Version/s: 2.0.2-alpha
    • Component/s: build, examples
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The hsqldb jar is included in hadoop for the DBCountPageView example only. Currently the example is using hsqldb version 2.x; however, 2.x is incompatible with 1.8.x – having this jar in the hadoop class path conflicts with dependent projects like Oozie, Hive, and Pig which still use 1.8.x. As there are no features hsqldb 2.x that are used by the example, we should remove it from Hadoop's runtime classpath.

      1. MAPREDUCE-4498-v3.patch
        4 kB
        Robert Kanter
      2. MAPREDUCE-4498-v2.patch
        3 kB
        Robert Kanter
      3. MAPREDUCE-4498.patch
        3 kB
        Robert Kanter

        Activity

        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538569/MAPREDUCE-4498.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2684//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538569/MAPREDUCE-4498.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2684//console This message is automatically generated.
        Hide
        Thomas Graves added a comment -

        Actually I believe it was specifically upgraded to 2.X to fix a bug in the tests - see MAPREDUCE-3621. So if you do this I want to make sure thoses tests don't start failing.

        Show
        Thomas Graves added a comment - Actually I believe it was specifically upgraded to 2.X to fix a bug in the tests - see MAPREDUCE-3621 . So if you do this I want to make sure thoses tests don't start failing.
        Hide
        Alejandro Abdelnur added a comment - - edited

        It looks like we are including hsqldb in hadoop lib JARs only because of an example. This seems wrong, the right thing to do would be to have a lib-examples/ directory where hsqldb should live and the example should be run using the hadoop -libjars option. By doing this we don't pollute Hadoop classpath with unneeded runtime JARs and we don't create conflicts for projects that use/require other versions of hsqldb.

        Show
        Alejandro Abdelnur added a comment - - edited It looks like we are including hsqldb in hadoop lib JARs only because of an example. This seems wrong, the right thing to do would be to have a lib-examples/ directory where hsqldb should live and the example should be run using the hadoop -libjars option. By doing this we don't pollute Hadoop classpath with unneeded runtime JARs and we don't create conflicts for projects that use/require other versions of hsqldb.
        Hide
        Ashutosh Chauhan added a comment -

        +1 to Alejandro's suggestions. Keeping hadoop's classpath to absolute minimum will avoid headaches for all the downstream projects. I have been hit by this more then once.

        Show
        Ashutosh Chauhan added a comment - +1 to Alejandro's suggestions. Keeping hadoop's classpath to absolute minimum will avoid headaches for all the downstream projects. I have been hit by this more then once.
        Hide
        Alejandro Abdelnur added a comment -

        It seems it is not that simple. The DBCountPageView example starts/shutdowns HSQLDB server if the driver is for HSQLDB. This means that the HSQLDB JAR must be in the bootstrap classloader. A way to address this is to do all the start/shutdown of HSQL via reflection in the context of the Thread.currentThread().getContextClassLoader() classloader.

        Show
        Alejandro Abdelnur added a comment - It seems it is not that simple. The DBCountPageView example starts/shutdowns HSQLDB server if the driver is for HSQLDB. This means that the HSQLDB JAR must be in the bootstrap classloader. A way to address this is to do all the start/shutdown of HSQL via reflection in the context of the Thread.currentThread().getContextClassLoader() classloader.
        Hide
        Robert Joseph Evans added a comment -

        Why don't we update the example to show how to add HSQL to the classpath instead of forcing HSQL on everyone? You can add it in with an environment variable very easily.

        Show
        Robert Joseph Evans added a comment - Why don't we update the example to show how to add HSQL to the classpath instead of forcing HSQL on everyone? You can add it in with an environment variable very easily.
        Hide
        Robert Kanter added a comment -

        Alejandro and I tried to modify the example to use reflection to pick up the hsql jar from the -libjars argument, but that just caused it to fail in the DBInputFormat because DriverManager tries to instantiate the JDBC driver from the bootstrap classloader (-libjars is a different classloader). This approach also made the example's code unnecessarily complicated because of all of the reflection.

        Instead, we left the example's code alone, but moved the hsqldb 2.x jar out of the mapred/lib/ dir of the TAR and into a new mapred/lib-examples/ dir. Then to run the DBCountPageView example, you would do:

        $ export HADOOP_CLASSPATH=share/hadoop/mapreduce/lib-examples/hsqldb-2.0.0.jar
        $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar dbcount -libjars share/hadoop/mapreduce/lib-examples/hsqldb-2.0.0.jar
        
        Show
        Robert Kanter added a comment - Alejandro and I tried to modify the example to use reflection to pick up the hsql jar from the -libjars argument, but that just caused it to fail in the DBInputFormat because DriverManager tries to instantiate the JDBC driver from the bootstrap classloader (-libjars is a different classloader). This approach also made the example's code unnecessarily complicated because of all of the reflection. Instead, we left the example's code alone, but moved the hsqldb 2.x jar out of the mapred/lib/ dir of the TAR and into a new mapred/lib-examples/ dir. Then to run the DBCountPageView example, you would do: $ export HADOOP_CLASSPATH=share/hadoop/mapreduce/lib-examples/hsqldb-2.0.0.jar $ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.0.0-SNAPSHOT.jar dbcount -libjars share/hadoop/mapreduce/lib-examples/hsqldb-2.0.0.jar
        Hide
        Alejandro Abdelnur added a comment -

        +1. Is there any objection to this approach?

        Also, Robert would you look if there is any doc/wiki that explains how to run the dbcount example? We would need to update this as well.

        Show
        Alejandro Abdelnur added a comment - +1. Is there any objection to this approach? Also, Robert would you look if there is any doc/wiki that explains how to run the dbcount example? We would need to update this as well.
        Hide
        Robert Kanter added a comment -

        The only documentation I could find is in the javadoc comments for the DBCountPageView and the printout when you run the examples jar without specifying which example. I'll update the patch to add the instructions on how to run dbcount.

        Show
        Robert Kanter added a comment - The only documentation I could find is in the javadoc comments for the DBCountPageView and the printout when you run the examples jar without specifying which example. I'll update the patch to add the instructions on how to run dbcount.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538602/MAPREDUCE-4498-v2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples:

        org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2690//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2690//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538602/MAPREDUCE-4498-v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2690//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2690//console This message is automatically generated.
        Hide
        Robert Kanter added a comment -

        The TestCombineFileInputFormat test fails even without the patch; so it seems to be unrelated.

        Show
        Robert Kanter added a comment - The TestCombineFileInputFormat test fails even without the patch; so it seems to be unrelated.
        Hide
        Robert Kanter added a comment -

        v3 only adds some instructions in the javadoc for the example; no code or pom changes.

        Show
        Robert Kanter added a comment - v3 only adds some instructions in the javadoc for the example; no code or pom changes.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12538639/MAPREDUCE-4498-v3.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples:

        org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2691//testReport/
        Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2691//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12538639/MAPREDUCE-4498-v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-assemblies hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient hadoop-mapreduce-project/hadoop-mapreduce-examples: org.apache.hadoop.mapreduce.lib.input.TestCombineFileInputFormat +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2691//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2691//console This message is automatically generated.
        Hide
        Alejandro Abdelnur added a comment -

        +1

        Show
        Alejandro Abdelnur added a comment - +1
        Hide
        Alejandro Abdelnur added a comment -

        Thanks Robert. Committed to trunk and branch-2.

        Show
        Alejandro Abdelnur added a comment - Thanks Robert. Committed to trunk and branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2555 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2555/)
        MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2555 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2555/ ) MAPREDUCE-4498 . Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2620 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2620/)
        MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2620 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2620/ ) MAPREDUCE-4498 . Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2574/)
        MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2574/ ) MAPREDUCE-4498 . Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1128 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/)
        MAPREDUCE-4498. Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906
        Files :

        • /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml
        • /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java
        • /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml
        • /hadoop/common/trunk/hadoop-project/pom.xml
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1128 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1128/ ) MAPREDUCE-4498 . Remove hsqldb jar from Hadoop runtime classpath. (rkanter via tucu) (Revision 1369906) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1369906 Files : /hadoop/common/trunk/hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/pom.xml /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/DBCountPageView.java /hadoop/common/trunk/hadoop-mapreduce-project/pom.xml /hadoop/common/trunk/hadoop-project/pom.xml

          People

          • Assignee:
            Robert Kanter
            Reporter:
            Robert Kanter
          • Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development