Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-462

Unit tests not working under Windows

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.21.0
    • Fix Version/s: 0.21.0
    • Component/s: namenode, test
    • Labels:
      None
    • Environment:

      Windows

    • Hadoop Flags:
      Reviewed

      Description

      Unit tests are failing on windows due to a problem with rename.
      The failing code is around line 520 in FSImage.java:

            assert curDir.exists() : "Current directory must exist.";
            assert !prevDir.exists() : "prvious directory must not exist.";
            assert !tmpDir.exists() : "prvious.tmp directory must not exist.";
            // rename current to tmp
            rename(curDir, tmpDir);
            // save new image
            if (!curDir.mkdir())
              throw new IOException("Cannot create directory " + curDir);
      

      and seems related to some open file or directory

      1. TEST-org.apache.hadoop.hdfs.TestHdfs.txt
        10 kB
        Tsz Wo Nicholas Sze
      2. HDFS-462.patch
        2 kB
        Jakob Homan

        Issue Links

          Activity

          Luca Telloli created issue -
          Hide
          Konstantin Shvachko added a comment -

          Yes, as we traced with Luca, the rename of current to previous.tmp fails because current/edits remains open after loadFSImaeg(). WinNT does not let rename directories if there are open files in it. This was introduced by HADOOP-5314. For regular name-node start up this makes sense, but fails in other scenarios.

          We should really-really resurrect windows build at least once a week as we used to have.

          Show
          Konstantin Shvachko added a comment - Yes, as we traced with Luca, the rename of current to previous.tmp fails because current/edits remains open after loadFSImaeg() . WinNT does not let rename directories if there are open files in it. This was introduced by HADOOP-5314 . For regular name-node start up this makes sense, but fails in other scenarios. We should really-really resurrect windows build at least once a week as we used to have.
          Konstantin Shvachko made changes -
          Field Original Value New Value
          Fix Version/s 0.21.0 [ 12314046 ]
          Affects Version/s 0.21.0 [ 12314046 ]
          Environment Windows
          Component/s name-node [ 12312926 ]
          Konstantin Shvachko made changes -
          Link This issue blocks HDFS-456 [ HDFS-456 ]
          Jakob Homan made changes -
          Assignee Jakob Homan [ jghoman ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          MiniDFSCluster also does not work in Windows.

          //The following test failed in Windows.  
          //See also the attached log TEST-org.apache.hadoop.hdfs.TestHdfs.txt
          public class TestHdfs extends junit.framework.TestCase {
            public void testHdfs() throws IOException {
              new MiniDFSCluster(new Configuration(), 1, true, null);
            }
          }
          
          Testcase: testHdfs took 1.219 sec
          	Caused an ERROR
          All specified directories are not accessible or do not exist.
          java.io.IOException: All specified directories are not accessible or do not exist.
          	at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:370)
          	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:99)
          	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:255)
          	at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:236)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:254)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:299)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:405)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:399)
          	at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1159)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:276)
          	at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:120)
          	at org.apache.hadoop.hdfs.TestHdfs.testHdfs(TestHdfs.java:25)
          
          Show
          Tsz Wo Nicholas Sze added a comment - MiniDFSCluster also does not work in Windows. //The following test failed in Windows. //See also the attached log TEST-org.apache.hadoop.hdfs.TestHdfs.txt public class TestHdfs extends junit.framework.TestCase { public void testHdfs() throws IOException { new MiniDFSCluster( new Configuration(), 1, true , null ); } } Testcase: testHdfs took 1.219 sec Caused an ERROR All specified directories are not accessible or do not exist. java.io.IOException: All specified directories are not accessible or do not exist. at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:370) at org.apache.hadoop.hdfs.server.namenode.FSDirectory.loadFSImage(FSDirectory.java:99) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.initialize(FSNamesystem.java:255) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.<init>(FSNamesystem.java:236) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:254) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:299) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:405) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:399) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1159) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:276) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:120) at org.apache.hadoop.hdfs.TestHdfs.testHdfs(TestHdfs.java:25)
          Tsz Wo Nicholas Sze made changes -
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HDFS-396 [ HDFS-396 ]
          Hide
          Jakob Homan added a comment -

          Attaching patch that fixes problem. Konstantin tested on his Windows box and it worked. Passes unit tests. Nicholas, I don't believe your issue is due to this problem, but rather from the URI patch that's related to this one.

          Show
          Jakob Homan added a comment - Attaching patch that fixes problem. Konstantin tested on his Windows box and it worked. Passes unit tests. Nicholas, I don't believe your issue is due to this problem, but rather from the URI patch that's related to this one.
          Jakob Homan made changes -
          Attachment HDFS-462.patch [ 12413051 ]
          Hide
          Jakob Homan added a comment -

          submitting patch

          Show
          Jakob Homan added a comment - submitting patch
          Jakob Homan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Konstantin Shvachko added a comment -

          Nicholas your case falls under HDFS-456. See error messages before that complaining the path is not a uri.

          Checked the patch on windows. TestDistributedUpgrade passes with, fails without it.
          +1

          Show
          Konstantin Shvachko added a comment - Nicholas your case falls under HDFS-456 . See error messages before that complaining the path is not a uri. Checked the patch on windows. TestDistributedUpgrade passes with, fails without it. +1
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12413051/HDFS-462.patch
          against trunk revision 792881.

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javadoc. The javadoc tool did not generate any warning messages.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 findbugs. The patch does not introduce any new Findbugs warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed core unit tests.

          -1 contrib tests. The patch failed contrib unit tests.

          Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/testReport/
          Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
          Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/artifact/trunk/build/test/checkstyle-errors.html
          Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12413051/HDFS-462.patch against trunk revision 792881. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed core unit tests. -1 contrib tests. The patch failed contrib unit tests. Test results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/testReport/ Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/artifact/trunk/build/test/checkstyle-errors.html Console output: http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-vesta.apache.org/15/console This message is automatically generated.
          Hide
          Jakob Homan added a comment -

          Again, no new tests as is fixing a defect that's being caught by the current unit tests (under Windows at least). Patch is ready to go.

          Show
          Jakob Homan added a comment - Again, no new tests as is fixing a defect that's being caught by the current unit tests (under Windows at least). Patch is ready to go.
          Hide
          Konstantin Shvachko added a comment -

          I just committed this. Thank you Jakob.

          Show
          Konstantin Shvachko added a comment - I just committed this. Thank you Jakob.
          Konstantin Shvachko made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/23/)

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #23 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/23/ )
          Tom White made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Patch Available Patch Available
          3d 1h 34m 1 Jakob Homan 09/Jul/09 20:25
          Patch Available Patch Available Resolved Resolved
          4d 4h 20m 1 Konstantin Shvachko 14/Jul/09 00:46
          Resolved Resolved Closed Closed
          406d 21h 1m 1 Tom White 24/Aug/10 21:48

            People

            • Assignee:
              Jakob Homan
              Reporter:
              Luca Telloli
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development