Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-1050

Sqoop Hook does not package HDFS model jars which is required

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • None
    • None
    • None
    • None

    Description

      Sqoop hook should package hdfs model jar and when sqoop commands to import are run, see this error since hdfs model jar is missing resulting in errors while loading data.

      2016-07-22 17:06:36,181 DEBUG [main]: ql.Driver (DriverContext.java:shutdown(132)) - Shutting down query
      LOAD DATA INPATH 'hdfs://localhost.localdomain:8020/user/hive/x_service' INTO TABLE `x_service`
      2016-07-22 17:06:36,181 INFO  [main]: CliDriver (SessionState.java:printInfo(939)) - Time taken: 4.791 seconds
      2016-07-22 17:06:36,181 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogBegin(148)) - <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
      2016-07-22 17:06:36,181 INFO  [main]: log.PerfLogger (PerfLogger.java:PerfLogEnd(176)) - </PERFLOG method=releaseLocks start=1469232396181 end=1469232396181 duration=0 from=org.apache.hadoop.hive.ql.Driver>
      2016-07-22 17:06:36,182 DEBUG [main]: session.SessionState (SessionState.java:close(1452)) - Removing resource dir /tmp/f60f6488-691f-440d-8fa0-01e26873f8a5_resources
      2016-07-22 17:06:36,191 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(115)) - ==> AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path)
      2016-07-22 17:06:36,192 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(123)) - AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling pluginClassLoader.loadClass()
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(79)) - ==> AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path)
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(87)) - AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling pluginClassLoader.findClass()
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(97)) - AtlasPluginClassLoader.findClass(org.apache.hadoop.fs.Path): calling componentClassLoader.findClass()
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(133)) - AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): calling componentClassLoader.loadClass()
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(142)) - <== AtlasPluginClassLoader.loadClass(org.apache.hadoop.fs.Path): class org.apache.hadoop.fs.Path
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: hook.HiveHook (HiveHook.java:processHiveEntity(632)) - Registering DFS Path hdfs://localhost.localdomain:8020/user/hive/x_service
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(115)) - ==> AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes)
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(123)) - AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes): calling pluginClassLoader.loadClass()
      2016-07-22 17:06:36,193 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(79)) - ==> AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes)
      2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(87)) - AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes): calling pluginClassLoader.findClass()
      2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:findClass(97)) - AtlasPluginClassLoader.findClass(org.apache.atlas.fs.model.FSDataTypes): calling componentClassLoader.findClass()
      2016-07-22 17:06:36,194 DEBUG [Atlas Logger 0]: classloader.AtlasPluginClassLoader (AtlasPluginClassLoader.java:loadClass(133)) - AtlasPluginClassLoader.loadClass(org.apache.atlas.fs.model.FSDataTypes): calling componentClassLoader.loadClass()
      2016-07-22 17:06:36,197 ERROR [Atlas Logger 0]: hook.HiveHook (HiveHook.java:run(188)) - Atlas hook failed due to error
      java.lang.NoClassDefFoundError: org/apache/atlas/fs/model/FSDataTypes
              at org.apache.atlas.hive.bridge.HiveMetaStoreBridge.fillHDFSDataSet(HiveMetaStoreBridge.java:556)
              at org.apache.atlas.hive.hook.HiveHook.processHiveEntity(HiveHook.java:634)
              at org.apache.atlas.hive.hook.HiveHook.registerProcess(HiveHook.java:599)
              at org.apache.atlas.hive.hook.HiveHook.fireAndForget(HiveHook.java:227)
              at org.apache.atlas.hive.hook.HiveHook.access$200(HiveHook.java:82)
              at org.apache.atlas.hive.hook.HiveHook$2.run(HiveHook.java:186)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: java.lang.ClassNotFoundException: org.apache.atlas.fs.model.FSDataTypes
              at java.lang.ClassLoader.findClass(ClassLoader.java:530)
              at org.apache.atlas.plugin.classloader.AtlasPluginClassLoader$MyClassLoader.findClass(AtlasPluginClassLoader.java:333)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
              at org.apache.atlas.plugin.classloader.AtlasPluginClassLoader.loadClass(AtlasPluginClassLoader.java:137)
              ... 11 more
      

      Attachments

        Issue Links

          Activity

            People

              suma.shivaprasad Suma Shivaprasad
              suma.shivaprasad Suma Shivaprasad
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: