Sqoop
  1. Sqoop
  2. SQOOP-1226

--password-file option triggers FileSystemClosed exception at end of Oozie action

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Centos 6.2 + jdk-1.6.0_31-fcs.x86_64

      Description

      When using the --password-file option, a Sqoop action running inside an Oozie workflow will ERROR out at the very end, like so:

      2013-10-31 13:38:45,095 INFO org.apache.sqoop.hive.HiveImport: Hive import complete.
      2013-10-31 13:38:45,098 INFO org.apache.sqoop.hive.HiveImport: Export directory is empty, removing it.
      2013-10-31 13:38:45,213 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
      2013-10-31 13:38:45,217 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:java.io.IOException: Filesystem closed
      2013-10-31 13:38:45,218 WARN org.apache.hadoop.mapred.Child: Error running child
      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565)
      	at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:589)
      	at java.io.FilterInputStream.close(FilterInputStream.java:155)
      	at org.apache.hadoop.util.LineReader.close(LineReader.java:149)
      	at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:243)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:222)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:421)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      2013-10-31 13:38:45,234 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
      

      With the --password option, the job completes with no error. I believe the --password-file option handling closes the FS which happens to be shared with the Oozie launcher, which can't write to it anymore on completion. The solution I found was adding:

        <property>
          <name>fs.hdfs.impl.disable.cache</name>
          <value>true</value>
        </property>
      

      in the sqoop action definition in the oozie workflow, and that works, but isn't really handy.
      Details are at https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pdsxiy5C_IY/OD8wR0rhHgMJ

      1. SQOOP-1226.patch
        0.8 kB
        Jarek Jarcec Cecho
      2. SQOOP-1226.patch
        0.7 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              David Morel
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development