Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-3188

Sqoop1 (import + --target-dir) with empty directory (/usr/lib/hive) fails with error (java.lang.NoClassDefFoundError: org/json/JSONObject)

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Sqoop1 (import + --target-dir) with empty directory (/usr/lib/hive) fails with error (java.lang.NoClassDefFoundError: org/json/JSONObject), see test case below.

      Test Case

      #################
      # STEP 01 - Create Table and Data
      #################
      
      export MYCONN=jdbc:mysql://mysql.sqoop.com:3306/sqoop
      export MYUSER=sqoop
      export MYPSWD=sqoop
      
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "drop table t1"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "create table t1 (c1 int, c2 date, c3 varchar(10))"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "insert into t1 values (1, current_date, 'some data')"
      sqoop eval --connect $MYCONN --username $MYUSER --password $MYPSWD --query "select * from t1"
      
      Output:
      -----------------------------------------
      | c1          | c2         | c3         | 
      -----------------------------------------
      | 1           | 2017-05-10 | some data  | 
      -----------------------------------------
      
      #################
      # STEP 02 - Import Data into HDFS 
      #################
      
      sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
      hdfs dfs -cat /user/root/t1/part*
      
      Output:
      17/05/10 13:46:24 INFO mapreduce.ImportJobBase: Transferred 23 bytes in 22.65 seconds (1.0155 bytes/sec)
      17/05/10 13:46:24 INFO mapreduce.ImportJobBase: Retrieved 1 records.
      ~~~~~
      1,2017-05-10,some data
      
      #################
      # STEP 03 - Create Bogus Hive Directory and Attempt to Import into HDFS
      #################
      
      mkdir /usr/lib/hive
      chmod 777 /usr/lib/hive
      sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
      
      Output:
      17/05/10 13:47:44 INFO mapreduce.ImportJobBase: Beginning import of t1
      Exception in thread "main" java.lang.NoClassDefFoundError: org/json/JSONObject
      	at org.apache.sqoop.util.SqoopJsonUtil.getJsonStringforMap(SqoopJsonUtil.java:43)
      	at org.apache.sqoop.SqoopOptions.writeProperties(SqoopOptions.java:776)
      	at org.apache.sqoop.mapreduce.JobBase.putSqoopOptionsToConfiguration(JobBase.java:388)
      	at org.apache.sqoop.mapreduce.JobBase.createJob(JobBase.java:374)
      	at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:256)
      	at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:692)
      	at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:127)
      	at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:513)
      	at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621)
      	at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
      	at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      	at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
      	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
      	at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
      	at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
      Caused by: java.lang.ClassNotFoundException: org.json.JSONObject
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      	at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      	... 15 more
      
      #################
      # STEP 04 - Remove Bogus Hive Directory and Attempt to Import into HDFS 
      #################
      
      rm -rf /usr/lib/hive
      sqoop import --connect $MYCONN --username $MYUSER --password $MYPSWD --table t1 --target-dir /user/root/t1 --delete-target-dir --num-mappers 1
      hdfs dfs -cat /user/root/t1/part*
      
      Output:
      17/05/10 13:52:30 INFO mapreduce.ImportJobBase: Transferred 23 bytes in 22.6361 seconds (1.0161 bytes/sec)
      17/05/10 13:52:30 INFO mapreduce.ImportJobBase: Retrieved 1 records.
      ~~~~~
      1,2017-05-10,some data
      

        Attachments

          Activity

            People

            • Assignee:
              ericlin Eric Lin
              Reporter:
              markuskemper@me.com Markus Kemper
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: