Sqoop
  1. Sqoop
  2. SQOOP-951

--export-dir to support subdirectories

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Debian GNU/Linux 6.0

      Description

      I am using sqoop-1.4.2 to export to Sql.

      --export-dir does not work when the dir being passed is the root of subdirectories. -export-dir is not doing any recursive lookup for files. It expects directory with files that you want export.
      It would be great if one could pass a directory with subdirectories.

      Example:

      The following command exports the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01 --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      Found 1 items
      rw-rr- 1 hadoop supergroup 15931406 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01/part-r-00001

      The following command does not export the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/
      Found 44 items
      rw-rr- 1 hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/_SUCCESS
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-02
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-03
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-04
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-05
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-06

      1. SQOOP-951.patch
        3 kB
        Vasanth kumar RJ

        Issue Links

          Activity

          Hide
          Jarek Jarcec Cecho added a comment -

          Cancelling the patch as there is discussion on the review board about direction of the patch. Vasanth kumar RJ, please do not hesitate and set this back to "Patch available" once uploading refreshed patch!

          Show
          Jarek Jarcec Cecho added a comment - Cancelling the patch as there is discussion on the review board about direction of the patch. Vasanth kumar RJ , please do not hesitate and set this back to "Patch available" once uploading refreshed patch!

            People

            • Assignee:
              Vasanth kumar RJ
              Reporter:
              Matthieu Labour
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development