Uploaded image for project: 'Sqoop'
  1. Sqoop
  2. SQOOP-1226

--password-file option triggers FileSystemClosed exception at end of Oozie action



    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
    • Environment:

      Centos 6.2 + jdk-1.6.0_31-fcs.x86_64


      When using the --password-file option, a Sqoop action running inside an Oozie workflow will ERROR out at the very end, like so:

      2013-10-31 13:38:45,095 INFO org.apache.sqoop.hive.HiveImport: Hive import complete.
      2013-10-31 13:38:45,098 INFO org.apache.sqoop.hive.HiveImport: Export directory is empty, removing it.
      2013-10-31 13:38:45,213 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
      2013-10-31 13:38:45,217 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:java.io.IOException: Filesystem closed
      2013-10-31 13:38:45,218 WARN org.apache.hadoop.mapred.Child: Error running child
      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565)
      	at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:589)
      	at java.io.FilterInputStream.close(FilterInputStream.java:155)
      	at org.apache.hadoop.util.LineReader.close(LineReader.java:149)
      	at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:243)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:222)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:421)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      2013-10-31 13:38:45,234 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task

      With the --password option, the job completes with no error. I believe the --password-file option handling closes the FS which happens to be shared with the Oozie launcher, which can't write to it anymore on completion. The solution I found was adding:


      in the sqoop action definition in the oozie workflow, and that works, but isn't really handy.
      Details are at https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pdsxiy5C_IY/OD8wR0rhHgMJ


        1. SQOOP-1226.patch
          0.7 kB
          Jarek Jarcec Cecho
        2. SQOOP-1226.patch
          0.8 kB
          Jarek Jarcec Cecho

          Issue Links



              • Assignee:
                jarcec Jarek Jarcec Cecho
                dmorel David Morel
              • Votes:
                0 Vote for this issue
                9 Start watching this issue


                • Created: