Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1237

sqoop export of hdfs file with empty lines causes TextExportMapper.map to fail

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.3
    • Component/s: sqoop2-client
    • Labels:
      None

      Description

      When the hdfs file coming from different sources show empty lines, it causes break in sqoop.And the options -input-null-string do not work.
      This can be workaround by applying sed -i '/^$/d' <file> on the hdfs file.

      However it would be nice TextExportMapper can ignore blank lines., possibly by -ignore_blanks true option (or possibly default ignoring blank lines).

      Sqoop: 1.4.3 (cdh 4.3.1)
      command: sqoop export Dmapred.job.queue.name=<queue_name>-connect <connection> --username <username> --password <password> --table <table> --input-fields-terminated-by "|" --input-lines-terminated-by
      n --export-dir <export_dir> --input-null-string '
      N' --input-null-non-string '
      N'

      error:java.io.IOException: Can't export data, please check task tracker logs
      at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
      at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
      at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
      at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
      at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
      at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      at org.apache.hadoop.mapred.Child.main(Child.java:262)
      Caused by: java.util.NoSuchElementException
      at java.util.AbstractList$Itr.next(AbstractList.java:350)

        Attachments

        1. SQOOP-1237_1.patch
          5 kB
          Rekha Joshi

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rekhajoshm Rekha Joshi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: