Bigtop
  1. Bigtop
  2. BIGTOP-203

sqoop needs to have access to hadoop .jars in its lib

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.1.0
    • Fix Version/s: 0.2.0
    • Component/s: None
    • Labels:
      None

      Description

      Otherwise it fails like this:

      sqoop help
      Warning: /usr/lib/hbase does not exist! HBase imports will fail.
      Please set $HBASE_HOME to the root of your HBase installation.
      Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.getInstances(Ljava/lang/String;Ljava/lang/Class;)Ljava/util/List;
      at com.cloudera.sqoop.tool.SqoopTool.loadPlugins(SqoopTool.java:139)
      at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:209)
      at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:228)
      at com.cloudera.sqoop.Sqoop.main(Sqoop.java:237)

      The root cause of this issue is that sqoop has a hardcoded dependency on Cloudera's version of Hadoop

      1. BIGTOP-203.patch.txt
        0.9 kB
        Roman Shaposhnik

        Activity

        Roman Shaposhnik created issue -
        Hide
        Roman Shaposhnik added a comment -

        The following:
        org.apache.hadoop.mapreduce.lib.db.DBWritable
        org.apache.hadoop.mapreduce.lib.input.CombineFileSplit
        org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat
        org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader
        is also missing in pre .21 and I'm not sure it'll be trivial to workaround.

        Now, given that – I'm inclined to disable Sqoop in 0.2.0

        Thoughts?

        Show
        Roman Shaposhnik added a comment - The following: org.apache.hadoop.mapreduce.lib.db.DBWritable org.apache.hadoop.mapreduce.lib.input.CombineFileSplit org.apache.hadoop.mapreduce.lib.input.CombineFileInputFormat org.apache.hadoop.mapreduce.lib.input.CombineFileRecordReader is also missing in pre .21 and I'm not sure it'll be trivial to workaround. Now, given that – I'm inclined to disable Sqoop in 0.2.0 Thoughts?
        Hide
        Bruno Mahé added a comment -

        Since it does not meet basic criteria, we should disable it.

        Show
        Bruno Mahé added a comment - Since it does not meet basic criteria, we should disable it.
        Hide
        Bruno Mahé added a comment - - edited

        +1 on disabling it

        Show
        Bruno Mahé added a comment - - edited +1 on disabling it
        Hide
        Eli Collins added a comment -

        What's the upstream Sqoop issue for this?

        Show
        Eli Collins added a comment - What's the upstream Sqoop issue for this?
        Show
        Roman Shaposhnik added a comment - https://issues.apache.org/jira/browse/SQOOP-384
        Hide
        Roman Shaposhnik added a comment -

        Just to be extremely clear as to what's going on: we're not removing Sqoop from Bigtop, we're disabling it for 0.2.0 release and we plan to enable it back again when we transition to either Hadoop 0.22 or 0.23. Those Hadoop release have all the APIs that Sqoop expects.

        Show
        Roman Shaposhnik added a comment - Just to be extremely clear as to what's going on: we're not removing Sqoop from Bigtop, we're disabling it for 0.2.0 release and we plan to enable it back again when we transition to either Hadoop 0.22 or 0.23. Those Hadoop release have all the APIs that Sqoop expects.
        Roman Shaposhnik made changes -
        Field Original Value New Value
        Attachment BIGTOP-203.patch.txt [ 12501689 ]
        Roman Shaposhnik made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Bruno Mahé added a comment -

        +1. Can't wait to reactivate it!

        Show
        Bruno Mahé added a comment - +1. Can't wait to reactivate it!
        Hide
        Peter Linnell added a comment -

        +1 Will be nice to have this back in 0.3.0

        Show
        Peter Linnell added a comment - +1 Will be nice to have this back in 0.3.0
        Roman Shaposhnik made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Bhavesh Shah added a comment -

        I am also facing same kind of problems.
        I have also set the $HBASE_HOME and $ZOOKEEPER_HOME.
        how to solve it? pls suggest me.

        Thanks.

        Show
        Bhavesh Shah added a comment - I am also facing same kind of problems. I have also set the $HBASE_HOME and $ZOOKEEPER_HOME. how to solve it? pls suggest me. Thanks.
        Gavin made changes -
        Workflow no-reopen-closed, patch-avail [ 12639883 ] patch-available, re-open possible [ 12665872 ]
        Roman Shaposhnik made changes -
        Status Resolved [ 5 ] Closed [ 6 ]

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development