Sqoop
  1. Sqoop
  2. SQOOP-951

--export-dir to support subdirectories

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.4.3
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Environment:

      Debian GNU/Linux 6.0

      Description

      I am using sqoop-1.4.2 to export to Sql.

      --export-dir does not work when the dir being passed is the root of subdirectories. -export-dir is not doing any recursive lookup for files. It expects directory with files that you want export.
      It would be great if one could pass a directory with subdirectories.

      Example:

      The following command exports the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01 --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      Found 1 items
      rw-rr- 1 hadoop supergroup 15931406 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01/part-r-00001

      The following command does not export the data to Sql

      ~/sqoop-1.4.2.bin__hadoop-1.0.0/bin/sqoop export --connect jdbc:postgresql://ec2-XX-XXX-XXX-XXX.compute-1.amazonaws.com:XXXX/xxxxxxxxxxxxx --username xxxxxxxxxx --password xxxxxxxxxx --table ml_ys_log_gmt_daily_experiment_2 --export-dir =hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized --input-fields-terminated-by='\t' --lines-terminated-by='\n' --verbose --batch

      hadoop@domU-XX-XX-XX-XX-XX-XX:/mnt/var/lib/hadoop/steps/2$ hadoop fs -ls hdfs:///mnt/var/lib/hadoop/dfs/logs_daily_sanitized/
      Found 44 items
      rw-rr- 1 hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/_SUCCESS
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-01
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-02
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-03
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-04
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-05
      drwxr-xr-x - hadoop supergroup 0 2013-03-15 17:03 /mnt/var/lib/hadoop/dfs/logs_daily_sanitized/dt=2013-02-06

      1. SQOOP-951.patch
        3 kB
        Vasanth kumar RJ

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          34d 16h 59m 1 Vasanth kumar RJ 19/Apr/13 13:05
          Patch Available Patch Available Open Open
          180d 9h 28m 1 Jarek Jarcec Cecho 16/Oct/13 22:34
          spapin made changes -
          Comment [ I believe this is the same feature request as in SQOOP-213 ]
          Jarek Jarcec Cecho made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Fix Version/s 1.4.5 [ 12324765 ]
          Hide
          Jarek Jarcec Cecho added a comment -

          Cancelling the patch as there is discussion on the review board about direction of the patch. Vasanth kumar RJ, please do not hesitate and set this back to "Patch available" once uploading refreshed patch!

          Show
          Jarek Jarcec Cecho added a comment - Cancelling the patch as there is discussion on the review board about direction of the patch. Vasanth kumar RJ , please do not hesitate and set this back to "Patch available" once uploading refreshed patch!
          Vasanth kumar RJ made changes -
          Fix Version/s 1.4.5 [ 12324765 ]
          Fix Version/s 1.4.4 [ 12324082 ]
          Jarek Jarcec Cecho made changes -
          Remote Link This issue links to "Review board (Web Link)" [ 12159 ]
          Vasanth kumar RJ made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Vasanth kumar RJ made changes -
          Component/s sqoop2-client [ 12319216 ]
          Vasanth kumar RJ made changes -
          Affects Version/s 1.4.3 [ 12322644 ]
          Affects Version/s 1.4.2 [ 12320141 ]
          Vasanth kumar RJ made changes -
          Fix Version/s 1.4.4 [ 12324082 ]
          Vasanth kumar RJ made changes -
          Attachment SQOOP-951.patch [ 12579532 ]
          Vasanth kumar RJ made changes -
          Field Original Value New Value
          Assignee Vasanth kumar RJ [ vasanthkumar ]
          Matthieu Labour created issue -

            People

            • Assignee:
              Vasanth kumar RJ
              Reporter:
              Matthieu Labour
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development