Sqoop
  1. Sqoop
  2. SQOOP-1226

--password-file option triggers FileSystemClosed exception at end of Oozie action

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.4.3
    • Fix Version/s: 1.4.5
    • Component/s: None
    • Labels:
      None
    • Environment:

      Centos 6.2 + jdk-1.6.0_31-fcs.x86_64

      Description

      When using the --password-file option, a Sqoop action running inside an Oozie workflow will ERROR out at the very end, like so:

      2013-10-31 13:38:45,095 INFO org.apache.sqoop.hive.HiveImport: Hive import complete.
      2013-10-31 13:38:45,098 INFO org.apache.sqoop.hive.HiveImport: Export directory is empty, removing it.
      2013-10-31 13:38:45,213 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
      2013-10-31 13:38:45,217 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:mapred (auth:SIMPLE) cause:java.io.IOException: Filesystem closed
      2013-10-31 13:38:45,218 WARN org.apache.hadoop.mapred.Child: Error running child
      java.io.IOException: Filesystem closed
      	at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565)
      	at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:589)
      	at java.io.FilterInputStream.close(FilterInputStream.java:155)
      	at org.apache.hadoop.util.LineReader.close(LineReader.java:149)
      	at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:243)
      	at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:222)
      	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:421)
      	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
      	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:396)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
      	at org.apache.hadoop.mapred.Child.main(Child.java:262)
      2013-10-31 13:38:45,234 INFO org.apache.hadoop.mapred.Task: Runnning cleanup for the task
      

      With the --password option, the job completes with no error. I believe the --password-file option handling closes the FS which happens to be shared with the Oozie launcher, which can't write to it anymore on completion. The solution I found was adding:

        <property>
          <name>fs.hdfs.impl.disable.cache</name>
          <value>true</value>
        </property>
      

      in the sqoop action definition in the oozie workflow, and that works, but isn't really handy.
      Details are at https://groups.google.com/a/cloudera.org/d/msg/cdh-user/pdsxiy5C_IY/OD8wR0rhHgMJ

      1. SQOOP-1226.patch
        0.8 kB
        Jarek Jarcec Cecho
      2. SQOOP-1226.patch
        0.7 kB
        Jarek Jarcec Cecho

        Issue Links

          Activity

          Hide
          Jarek Jarcec Cecho added a comment -

          I believe that Sqoop is doing the right thing here - closing resources that we've opened. This issue seems to me as a incorrect usage of the cache and the proposed solution seems as a valid approach.

          Show
          Jarek Jarcec Cecho added a comment - I believe that Sqoop is doing the right thing here - closing resources that we've opened. This issue seems to me as a incorrect usage of the cache and the proposed solution seems as a valid approach.
          Hide
          Jarek Jarcec Cecho added a comment -

          I've discussed the problem with Colin Patrick McCabe (Hadoop committer) and he recommended us to simply remove the FileSystem#close() call.

          Show
          Jarek Jarcec Cecho added a comment - I've discussed the problem with Colin Patrick McCabe (Hadoop committer) and he recommended us to simply remove the FileSystem#close() call.
          Hide
          Colin Patrick McCabe added a comment -

          Yeah. It's unfortunate that our FileSystem cache doesn't reference count the objects. However, we're kind of stuck with this behavior to avoid breaking existing applications that depend on it. Maybe in the future we can introduce a better API for the FileSystem instance cache.

          Show
          Colin Patrick McCabe added a comment - Yeah. It's unfortunate that our FileSystem cache doesn't reference count the objects. However, we're kind of stuck with this behavior to avoid breaking existing applications that depend on it. Maybe in the future we can introduce a better API for the FileSystem instance cache.
          Hide
          David Morel added a comment -

          Isn't there a potential for introducing leaks for standalone sqoop invocations then (completely uninformed question, bear with me)?

          Show
          David Morel added a comment - Isn't there a potential for introducing leaks for standalone sqoop invocations then (completely uninformed question, bear with me)?
          Hide
          Jarek Jarcec Cecho added a comment -

          That is very good question David Morel, thank you for asking! We've discussed this exact problem with Colin Patrick McCabe and since Sqoop is a command line utility (one process) the leak will be closed by OS when terminating the process. Even thought that this is not the best approach, it seems the best one considering current HDFS Cache implementation.

          Show
          Jarek Jarcec Cecho added a comment - That is very good question David Morel , thank you for asking! We've discussed this exact problem with Colin Patrick McCabe and since Sqoop is a command line utility (one process) the leak will be closed by OS when terminating the process. Even thought that this is not the best approach, it seems the best one considering current HDFS Cache implementation.
          Hide
          Hari Shreedharan added a comment -

          Unfortunately, this does introduce a limited a leak. Can you please add a comment that explains the reason that the close call is not made (and reference this jira) so someone does not introduce the close call again?

          Show
          Hari Shreedharan added a comment - Unfortunately, this does introduce a limited a leak. Can you please add a comment that explains the reason that the close call is not made (and reference this jira) so someone does not introduce the close call again?
          Hide
          Hari Shreedharan added a comment -

          +1. The new patch looks good. Committing after running tests

          Show
          Hari Shreedharan added a comment - +1. The new patch looks good. Committing after running tests
          Hide
          ASF subversion and git services added a comment -

          Commit a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 in branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a1e45de ]

          SQOOP-1226. --password-file option triggers FileSystemClosed exception at end of Oozie action

          (Jarek Jarcec Cecho via Hari Shreedharan)

          Show
          ASF subversion and git services added a comment - Commit a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 in branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=sqoop.git;h=a1e45de ] SQOOP-1226 . --password-file option triggers FileSystemClosed exception at end of Oozie action (Jarek Jarcec Cecho via Hari Shreedharan)
          Hide
          Hari Shreedharan added a comment -

          Committed! Thanks Jarcec!

          Show
          Hari Shreedharan added a comment - Committed! Thanks Jarcec!
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #835 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/835/)
          SQOOP-1226. --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0)

          • src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop100 #835 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop100/835/ ) SQOOP-1226 . --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 ) src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #876 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/876/)
          SQOOP-1226. --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0)

          • src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop200 #876 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop200/876/ ) SQOOP-1226 . --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 ) src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #870 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/870/)
          SQOOP-1226. --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0)

          • src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop20 #870 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop20/870/ ) SQOOP-1226 . --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 ) src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1072 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1072/)
          SQOOP-1226. --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0)

          • src/java/org/apache/sqoop/util/password/FilePasswordLoader.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Sqoop-ant-jdk-1.6-hadoop23 #1072 (See https://builds.apache.org/job/Sqoop-ant-jdk-1.6-hadoop23/1072/ ) SQOOP-1226 . --password-file option triggers FileSystemClosed exception at end of Oozie action (hshreedharan: https://git-wip-us.apache.org/repos/asf?p=sqoop.git&a=commit&h=a1e45de8e9c85f0911c5c6c7c3bb7acfa76190e0 ) src/java/org/apache/sqoop/util/password/FilePasswordLoader.java

            People

            • Assignee:
              Jarek Jarcec Cecho
              Reporter:
              David Morel
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development