Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-2489

Sqoop2: Hive with Parquet in Kite Connector

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.99.6
    • Fix Version/s: 1.99.8
    • Component/s: sqoop2-kite-connector
    • Labels:
      None

      Description

      java.lang.NoClassDefFoundError: org/apache/hadoop/hive/ql/io/HiveOutputFormat
      	at java.lang.ClassLoader.defineClass1(Native Method)
      	at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
      	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
      	at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
      	at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	at java.lang.Class.forName0(Native Method)
      	at java.lang.Class.forName(Class.java:190)
      	at org.kitesdk.data.spi.hive.HiveUtils.getHiveParquetOutputFormat(HiveUtils.java:446)
      	at org.kitesdk.data.spi.hive.HiveUtils.<clinit>(HiveUtils.java:91)
      	at org.kitesdk.data.spi.hive.HiveManagedMetadataProvider.create(HiveManagedMetadataProvider.java:83)
      	at org.kitesdk.data.spi.hive.HiveManagedDatasetRepository.create(HiveManagedDatasetRepository.java:77)
      	at org.kitesdk.data.Datasets.create(Datasets.java:239)
      	at org.kitesdk.data.Datasets.create(Datasets.java:307)
      	at org.kitesdk.data.Datasets.create(Datasets.java:335)
      	at org.apache.sqoop.connector.kite.KiteDatasetExecutor.createDataset(KiteDatasetExecutor.java:70)
      	at org.apache.sqoop.connector.kite.KiteLoader.getExecutor(KiteLoader.java:52)
      	at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:62)
      	at org.apache.sqoop.connector.kite.KiteLoader.load(KiteLoader.java:36)
      	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:250)
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.ql.io.HiveOutputFormat
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	... 31 more
      

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user sleefd opened a pull request:

        https://github.com/apache/sqoop/pull/8

        fix the problem of NoClassDefFoundError of HiveOutputFormat when impo…

        for jira SQOOP-2489 @abec

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/sleefd/sqoop sqoop2

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/sqoop/pull/8.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #8


        commit 3baa7430c6fcac8d0847aaa0c49fead4fadce372
        Author: slee <sleefd@gmail.com>
        Date: 2015-08-28T07:44:50Z

        fix the problem of NoClassDefFoundError of HiveOutputFormat when import or export to hive through kite connector


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user sleefd opened a pull request: https://github.com/apache/sqoop/pull/8 fix the problem of NoClassDefFoundError of HiveOutputFormat when impo… for jira SQOOP-2489 @abec You can merge this pull request into a Git repository by running: $ git pull https://github.com/sleefd/sqoop sqoop2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/sqoop/pull/8.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #8 commit 3baa7430c6fcac8d0847aaa0c49fead4fadce372 Author: slee <sleefd@gmail.com> Date: 2015-08-28T07:44:50Z fix the problem of NoClassDefFoundError of HiveOutputFormat when import or export to hive through kite connector
        Hide
        sleefd slee added a comment -

        patch based on branch-1.99.6

        Show
        sleefd slee added a comment - patch based on branch-1.99.6
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Switching to "Patch available", so that it shows up in the review queue.

        Show
        jarcec Jarek Jarcec Cecho added a comment - Switching to "Patch available", so that it shows up in the review queue.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Can you add integration tests that will try Parquet import as part of the patch slee? I'm afraid that without it we can always regress here. I think that adding the test shouldn't be that difficult as we already have test importing to CSV here. Perhaps let's just make it run with data provider and try all various file formats?

        Show
        jarcec Jarek Jarcec Cecho added a comment - Can you add integration tests that will try Parquet import as part of the patch slee ? I'm afraid that without it we can always regress here. I think that adding the test shouldn't be that difficult as we already have test importing to CSV here . Perhaps let's just make it run with data provider and try all various file formats?
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Can you add integration tests that will try Parquet import as part of the patch slee? I'm afraid that without it we can always regress here. I think that adding the test shouldn't be that difficult as we already have test importing to CSV here. Perhaps let's just make it run with data provider and try all various file formats?

        Show
        jarcec Jarek Jarcec Cecho added a comment - Can you add integration tests that will try Parquet import as part of the patch slee ? I'm afraid that without it we can always regress here. I think that adding the test shouldn't be that difficult as we already have test importing to CSV here . Perhaps let's just make it run with data provider and try all various file formats?
        Hide
        sleefd slee added a comment -

        Jarek Jarcec Cecho SQOOP-2269 has already added the integration test for avro and parquet format.

        Show
        sleefd slee added a comment - Jarek Jarcec Cecho SQOOP-2269 has already added the integration test for avro and parquet format.
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        Thanks for digging into it slee. I'm glad to see that we have some coverage already available via SQOOP-2269. This particular bug did not surface there, so I feel that we should update them such that - without the fix the test fails whereas with the fix it will succeed. This way we will have test for this particular problem and we will ensure that we will never regress in the future. Does it make sense?

        Show
        jarcec Jarek Jarcec Cecho added a comment - Thanks for digging into it slee . I'm glad to see that we have some coverage already available via SQOOP-2269 . This particular bug did not surface there, so I feel that we should update them such that - without the fix the test fails whereas with the fix it will succeed. This way we will have test for this particular problem and we will ensure that we will never regress in the future. Does it make sense?
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        I've assigned this one to you slee.

        Show
        jarcec Jarek Jarcec Cecho added a comment - I've assigned this one to you slee .
        Hide
        jarcec Jarek Jarcec Cecho added a comment -

        I'll drop the "Patch available" state for now as we're waiting on integration test for the fix. Please do not hesitate and put the state back when ready!

        Show
        jarcec Jarek Jarcec Cecho added a comment - I'll drop the "Patch available" state for now as we're waiting on integration test for the fix. Please do not hesitate and put the state back when ready!

          People

          • Assignee:
            sleefd slee
            Reporter:
            abec Abraham Elmahrek
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development