Hadoop Common
  1. Hadoop Common
  2. HADOOP-8342

HDFS command fails with exception following merge of HADOOP-8325

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-alpha
    • Fix Version/s: 0.23.3, 2.0.2-alpha
    • Component/s: fs
    • Labels:
      None
    • Environment:

      QE tests on version 2.0.1205010603

      Description

      We are seeing most hdfs commands in our nightly acceptance tests fail with an exception as shown below. This started with a few hours of the merge of HADOOP-8325 on 4/30/2012

      hdfs --config conf/hadoop/ dfs -ls dirname
      ls: `dirname': No such file or directory
      12/05/01 16:57:52 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.IllegalStateException: Shutdown in progress, cannot remove a shutdownHook
      java.lang.IllegalStateException: Shutdown in progress, cannot remove a shutdownHook
      at org.apache.hadoop.util.ShutdownHookManager.removeShutdownHook(ShutdownHookManager.java:166)
      at org.apache.hadoop.fs.FileSystem$Cache.remove(FileSystem.java:2202)
      at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2231)
      at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2251)
      at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54)

      1. HDFS-3337.patch
        0.8 kB
        Alejandro Abdelnur

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/)
        svn merge -c 1333224 FIXES: HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1356970)

        Result = SUCCESS
        bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356970
        Files :

        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-0.23-Build #304 (See https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/304/ ) svn merge -c 1333224 FIXES: HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1356970) Result = SUCCESS bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1356970 Files : /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1068 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1068/)
        HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1068 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1068/ ) HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1033 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1033/)
        HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

        Result = FAILURE
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1033 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1033/ ) HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224) Result = FAILURE tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2191/)
        HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

        Result = ABORTED
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2191 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2191/ ) HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224) Result = ABORTED tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2174 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2174/)
        HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2174 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2174/ ) HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2248 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2248/)
        HADOOP-8342. HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224)

        Result = SUCCESS
        tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2248 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2248/ ) HADOOP-8342 . HDFS command fails with exception following merge of HADOOP-8325 (tucu) (Revision 1333224) Result = SUCCESS tucu : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1333224 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java
        Hide
        Alejandro Abdelnur added a comment -

        committed to trunk and branch-2

        Show
        Alejandro Abdelnur added a comment - committed to trunk and branch-2
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12525211/HDFS-3337.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javadoc. The javadoc tool appears to have generated 2 warning messages.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.fs.viewfs.TestViewFsTrash

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/923//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/923//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525211/HDFS-3337.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.fs.viewfs.TestViewFsTrash +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/923//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/923//console This message is automatically generated.
        Hide
        Aaron T. Myers added a comment -

        Marking PA for Tucu.

        Show
        Aaron T. Myers added a comment - Marking PA for Tucu.
        Hide
        Aaron T. Myers added a comment -

        +1, I tried the patch and observed that it fixes the issue.

        Show
        Aaron T. Myers added a comment - +1, I tried the patch and observed that it fixes the issue.
        Hide
        Alejandro Abdelnur added a comment -

        My bad, I've missed the removal of the shutdownhook from the closing iteration.

        Attached is a patch that removes the removal of the shutdownhook from the Cache when the cache is empty. In such situation the shutdownhook will remain and will just do a NOP.

        Randy, would you please give it a try? don't see how to test this properly from a testcase.

        thxs

        Show
        Alejandro Abdelnur added a comment - My bad, I've missed the removal of the shutdownhook from the closing iteration. Attached is a patch that removes the removal of the shutdownhook from the Cache when the cache is empty. In such situation the shutdownhook will remain and will just do a NOP. Randy, would you please give it a try? don't see how to test this properly from a testcase. thxs

          People

          • Assignee:
            Alejandro Abdelnur
            Reporter:
            Randy Clayton
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development