Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-951

--export-dir to support subdirectories

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Debian GNU/Linux 6.0

      Description

      I am using sqoop-1.4.2 to export to Sql.

      --export-dir does not work when the dir being passed is the root of subdirectories. -export-dir is not doing any recursive lookup for files. It expects directory with files that you want export.
      It would be great if one could pass a directory with subdirectories.

      Example:

      The following command exports the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01 --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      Found 1 items
      rw-rr- 1 hadoop supergroup 15931406 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01/part-r-00001

      The following command does not export the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/
      Found 44 items
      rw-rr- 1 hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/_SUCCESS
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-02
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-03
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-04
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-05
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-06

        Attachments

        1. SQOOP-951.patch
          3 kB
          Vasanth kumar RJ

          Issue Links

            Activity

              People

              • Assignee:
                vasanthkumar Vasanth kumar RJ
                Reporter:
                mlabour Matthieu Labour
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: