Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      Linux

    • Hadoop Flags:
      Reviewed

      Description

      Testsuite: org.apache.hadoop.mapred.TestFairScheduler
      Tests run: 13, Failures: 0, Errors: 13, Time elapsed: 0.042 sec
      
      Testcase: testAllocationFileParsing took 0.004 sec
              Caused an ERROR
      /home/tsz/hadoop/latest/build/contrib/fairscheduler/test/data/./test-pools (No such file or directory)
      java.io.FileNotFoundException: /home/tsz/hadoop/latest/build/contrib/fairscheduler/test/data/./test-pools (No such file or directory)
              at java.io.FileOutputStream.open(Native Method)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
              at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
              at java.io.FileWriter.<init>(FileWriter.java:46)
              at org.apache.hadoop.mapred.TestFairScheduler.setUp(TestFairScheduler.java:215)
      
      ...
      
      1. 4050_20080902debug.patch
        1 kB
        Tsz Wo Nicholas Sze
      2. HADOOP-4050.patch
        0.7 kB
        Matei Zaharia
      3. HADOOP-4050-v2.patch
        2 kB
        Matei Zaharia
      4. HADOOP-4050-v3.patch
        1.0 kB
        Matei Zaharia

        Issue Links

          Activity

          Owen O'Malley made changes -
          Component/s contrib/fair-share [ 12312456 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Hudson added a comment -
          Show
          Hudson added a comment - Integrated in Hadoop-trunk #595 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/595/ )
          Owen O'Malley made changes -
          Fix Version/s 0.19.0 [ 12313211 ]
          Owen O'Malley made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Owen O'Malley added a comment -

          I just committed this. Thanks, Matei!

          Show
          Owen O'Malley added a comment - I just committed this. Thanks, Matei!
          Tsz Wo Nicholas Sze made changes -
          Hadoop Flags [Reviewed]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          +1 patch looks good. I tested it in my Linux machine.

          Show
          Tsz Wo Nicholas Sze added a comment - +1 patch looks good. I tested it in my Linux machine.
          Matei Zaharia made changes -
          Attachment HADOOP-4050-v3.patch [ 12389438 ]
          Hide
          Matei Zaharia added a comment -

          Good point about running without ant. Here's a patch that uses the test.build.data environment variable.

          Show
          Matei Zaharia added a comment - Good point about running without ant. Here's a patch that uses the test.build.data environment variable.
          Hide
          Hemanth Yamijala added a comment -

          We noticed this issue also when we moved some classes into contrib for the scheduler in HADOOP-3445. Actually, I think the data dir comes from build-contrib.xml, as per this line:

          <sysproperty key="test.build.data" value="${build.test}/data"/>
          

          Also, there are a few test classes in streaming that are creating this directory. For e.g. UtilTest.redirectIfAntJunit calls mkdirs on the directory set as the system property as mentioned above, and it is called from TestStreamingExitStatus.

          Since a lot of contrib tests seem to need this, should we just create it in build-contrib.xml which is used by all contrib build files mostly ?

          Show
          Hemanth Yamijala added a comment - We noticed this issue also when we moved some classes into contrib for the scheduler in HADOOP-3445 . Actually, I think the data dir comes from build-contrib.xml, as per this line: <sysproperty key= "test.build.data" value= "${build.test}/data" /> Also, there are a few test classes in streaming that are creating this directory. For e.g. UtilTest.redirectIfAntJunit calls mkdirs on the directory set as the system property as mentioned above, and it is called from TestStreamingExitStatus. Since a lot of contrib tests seem to need this, should we just create it in build-contrib.xml which is used by all contrib build files mostly ?
          Hide
          steve_l added a comment -

          JUnit tests run under an IDE don't get their CWD set to the build directory; best to use the test.build.data trick as described, and call .mkdirs() on it, in case the entire tree is missing.

          Show
          steve_l added a comment - JUnit tests run under an IDE don't get their CWD set to the build directory; best to use the test.build.data trick as described, and call .mkdirs() on it, in case the entire tree is missing.
          Hide
          Matei Zaharia added a comment -

          That makes sense.. Do you want me to use that and make the data directory myself then, or is it better to leave it in the Ant file because the Ant file sets the process's working directory to that directory anyway? If you wanted to have the Java code create the directory, it would just require replacing the TEST_DIR = new File(".").getAbsolutePath() in my first patch with TEST_DIR = new File(System.getProperty("test.build.data","build/contrib/streaming/test/data")).getAbsolutePath().

          Show
          Matei Zaharia added a comment - That makes sense.. Do you want me to use that and make the data directory myself then, or is it better to leave it in the Ant file because the Ant file sets the process's working directory to that directory anyway? If you wanted to have the Java code create the directory, it would just require replacing the TEST_DIR = new File(".").getAbsolutePath() in my first patch with TEST_DIR = new File(System.getProperty("test.build.data","build/contrib/streaming/test/data")).getAbsolutePath().
          Hide
          Tsz Wo Nicholas Sze added a comment -

          What we do for hadoop core is setting up a java system property in build.xml and then get it in the tests

          • build.xml
              <property name="test.build.data" value="${test.build.dir}/data"/>
            
          • TestLocalFileSystem.java
            private static String TEST_ROOT_DIR
                = System.getProperty("test.build.data","build/test/data/work-dir/localfs");
            
          Show
          Tsz Wo Nicholas Sze added a comment - What we do for hadoop core is setting up a java system property in build.xml and then get it in the tests build.xml <property name= "test.build.data" value= "${test.build.dir}/data" /> TestLocalFileSystem.java private static String TEST_ROOT_DIR = System .getProperty( "test.build.data" , "build/test/data/work-dir/localfs" );
          Matei Zaharia made changes -
          Attachment HADOOP-4050-v2.patch [ 12389385 ]
          Hide
          Matei Zaharia added a comment -

          Here's a patch that calls the mkdir in the build file, see how this works.

          Show
          Matei Zaharia added a comment - Here's a patch that calls the mkdir in the build file, see how this works.
          Hide
          Matei Zaharia added a comment -

          You are right, calling mkdir on "." is strange, and the only reason I did it is because JUnit does not create the test/data directory for me automatically. Since you have a different Linux kernel from mine, this is probably why it's failing for you and not for me. I've looked carefully through the build.xml files for streaming, contrib and Hadoop itself, but I can't figure out where exactly streaming gets its data directory to be mkdir'ed. Maybe it does do it in Java after all. I will create a patch for build-contrib.xml that makes the data directory at the same time it makes logs, etc to see if that fixes it.

          Show
          Matei Zaharia added a comment - You are right, calling mkdir on "." is strange, and the only reason I did it is because JUnit does not create the test/data directory for me automatically. Since you have a different Linux kernel from mine, this is probably why it's failing for you and not for me. I've looked carefully through the build.xml files for streaming, contrib and Hadoop itself, but I can't figure out where exactly streaming gets its data directory to be mkdir'ed. Maybe it does do it in Java after all. I will create a patch for build-contrib.xml that makes the data directory at the same time it makes logs, etc to see if that fixes it.
          Tsz Wo Nicholas Sze made changes -
          Attachment 4050_20080902debug.patch [ 12389384 ]
          Hide
          Tsz Wo Nicholas Sze added a comment -

          With 4050_20080902debug.patch, I saw
          b=false
          TEST_DIR=/home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/.

          So the problem is that java.io.File.mkdir() returned false.

          Show
          Tsz Wo Nicholas Sze added a comment - With 4050_20080902debug.patch, I saw b=false TEST_DIR=/home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/. So the problem is that java.io.File.mkdir() returned false.
          Hide
          Tsz Wo Nicholas Sze added a comment -

          BTW, I think the problem might be in

            final static String TEST_DIR = new File(".").getAbsolutePath();
          

          In general, we should not use "." or current directory for testing.

          Show
          Tsz Wo Nicholas Sze added a comment - BTW, I think the problem might be in final static String TEST_DIR = new File(".").getAbsolutePath(); In general, we should not use "." or current directory for testing.
          Hide
          Tsz Wo Nicholas Sze added a comment -
          • The test still fails after the patch. Below is more details
            $ant clean
            ...
            BUILD SUCCESSFUL
            Total time: 9 seconds
            $ant test -Dtestcase=TestFairScheduler
            ...
            test:
                 [echo] contrib: fairscheduler
               [delete] Deleting directory /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/logs
                [mkdir] Created dir: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/logs
                [junit] Running org.apache.hadoop.mapred.TestFairScheduler
                [junit] Tests run: 13, Failures: 0, Errors: 13, Time elapsed: 0.03 sec
                [junit] Test org.apache.hadoop.mapred.TestFairScheduler FAILED
            
            BUILD FAILED
            /home/tsz/hadoop/testing/build.xml:662: The following error occurred while executing this line:
            /home/tsz/hadoop/testing/src/contrib/build.xml:48: The following error occurred while executing this line:
            /home/tsz/hadoop/testing/src/contrib/build-contrib.xml:224: Tests failed!
            
            Total time: 57 seconds
            $head build/contrib/fairscheduler/test/TEST-org.apache.hadoop.mapred.TestFairScheduler.txt
             
            Testsuite: org.apache.hadoop.mapred.TestFairScheduler
            Tests run: 13, Failures: 0, Errors: 13, Time elapsed: 0.03 sec
            
            Testcase: testAllocationFileParsing took 0.004 sec
                    Caused an ERROR
            /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/test-pools (No such file or directory)
            java.io.FileNotFoundException: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/test-pools (No such file or directory)
                    at java.io.FileOutputStream.open(Native Method)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:179)
                    at java.io.FileOutputStream.<init>(FileOutputStream.java:70)
            $ ls build/contrib/fairscheduler/test/data
            ls: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data: No such file or directory
            $ ls -ld build/contrib/fairscheduler/test
            drwxr-xr-x  4 tsz users 4096 Sep  2 22:45 build/contrib/fairscheduler/test
            $ ls -l build/contrib/fairscheduler/test
            total 16
            -rw-r--r--  1 tsz users 7959 Sep  2 22:45 TEST-org.apache.hadoop.mapred.TestFairScheduler.txt
            drwxr-xr-x  2 tsz users 4096 Sep  2 22:45 logs
            drwxr-xr-x  3 tsz users 4096 Sep  2 22:45 org
            
            
          • Here is more information about my envirnoment:
            $uname -a
            Linux nodename 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
            $ java -version
            java version "1.6.0"
            Java(TM) SE Runtime Environment (build 1.6.0-b105)
            Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
            
          Show
          Tsz Wo Nicholas Sze added a comment - The test still fails after the patch. Below is more details $ant clean ... BUILD SUCCESSFUL Total time: 9 seconds $ant test -Dtestcase=TestFairScheduler ... test: [echo] contrib: fairscheduler [delete] Deleting directory /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/logs [mkdir] Created dir: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/logs [junit] Running org.apache.hadoop.mapred.TestFairScheduler [junit] Tests run: 13, Failures: 0, Errors: 13, Time elapsed: 0.03 sec [junit] Test org.apache.hadoop.mapred.TestFairScheduler FAILED BUILD FAILED /home/tsz/hadoop/testing/build.xml:662: The following error occurred while executing this line: /home/tsz/hadoop/testing/src/contrib/build.xml:48: The following error occurred while executing this line: /home/tsz/hadoop/testing/src/contrib/build-contrib.xml:224: Tests failed! Total time: 57 seconds $head build/contrib/fairscheduler/test/TEST-org.apache.hadoop.mapred.TestFairScheduler.txt Testsuite: org.apache.hadoop.mapred.TestFairScheduler Tests run: 13, Failures: 0, Errors: 13, Time elapsed: 0.03 sec Testcase: testAllocationFileParsing took 0.004 sec Caused an ERROR /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/test-pools (No such file or directory) java.io.FileNotFoundException: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data/test-pools (No such file or directory) at java.io.FileOutputStream.open(Native Method) at java.io.FileOutputStream.<init>(FileOutputStream.java:179) at java.io.FileOutputStream.<init>(FileOutputStream.java:70) $ ls build/contrib/fairscheduler/test/data ls: /home/tsz/hadoop/testing/build/contrib/fairscheduler/test/data: No such file or directory $ ls -ld build/contrib/fairscheduler/test drwxr-xr-x 4 tsz users 4096 Sep 2 22:45 build/contrib/fairscheduler/test $ ls -l build/contrib/fairscheduler/test total 16 -rw-r--r-- 1 tsz users 7959 Sep 2 22:45 TEST-org.apache.hadoop.mapred.TestFairScheduler.txt drwxr-xr-x 2 tsz users 4096 Sep 2 22:45 logs drwxr-xr-x 3 tsz users 4096 Sep 2 22:45 org Here is more information about my envirnoment: $uname -a Linux nodename 2.6.9-55.ELsmp #1 SMP Fri Apr 20 16:36:54 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux $ java -version java version "1.6.0" Java(TM) SE Runtime Environment (build 1.6.0-b105) Java HotSpot(TM) Server VM (build 1.6.0-b105, mixed mode)
          Matei Zaharia made changes -
          Attachment HADOOP-4050.patch [ 12389361 ]
          Hide
          Matei Zaharia added a comment -

          Here's a patch that removes the dot in case you want to try that.

          Show
          Matei Zaharia added a comment - Here's a patch that removes the dot in case you want to try that.
          Hide
          steve_l added a comment -

          The java apis are a bit vague about handing a relative path to File("./test-pools"); its the same as File(null,,"./test-pools"), which again is pretty ambigious.

          It's better to have a base directory to work with, but you dont want to work relative to the processes 'current' dir as in IDEs or complex builds, that is rarely where you expect it to be.

          One option here is to add more diagnostics to the test case; check the file exists, that its parent dir exists, etc.

          Show
          steve_l added a comment - The java apis are a bit vague about handing a relative path to File("./test-pools"); its the same as File(null,,"./test-pools"), which again is pretty ambigious. It's better to have a base directory to work with, but you dont want to work relative to the processes 'current' dir as in IDEs or complex builds, that is rarely where you expect it to be. One option here is to add more diagnostics to the test case; check the file exists, that its parent dir exists, etc.
          Hide
          Matei Zaharia added a comment -

          Steve - the path is obtained by calling new File("./test-pools").getAbsolutePath(), rather than any concatenation done in the code, so it's a path returned by Java's own API. We can try doing just new File("test-pools").getAbsolutePath(), but I don't have any information about the environment to be able to reproduce this or to be sure that it's not due to permissions or other problems.

          Show
          Matei Zaharia added a comment - Steve - the path is obtained by calling new File("./test-pools").getAbsolutePath(), rather than any concatenation done in the code, so it's a path returned by Java's own API. We can try doing just new File("test-pools").getAbsolutePath(), but I don't have any information about the environment to be able to reproduce this or to be sure that it's not due to permissions or other problems.
          Hide
          steve_l added a comment -

          It looks like a path concatenation problem to me; by the time the Java file APIs see it, the path shouldn't contain the /./ sequence...things should be resolved at this point already.Either its a bad configuration file or the code to take path+subdir needs to handle relative references

          Show
          steve_l added a comment - It looks like a path concatenation problem to me; by the time the Java file APIs see it, the path shouldn't contain the /./ sequence...things should be resolved at this point already.Either its a bad configuration file or the code to take path+subdir needs to handle relative references
          Hide
          Matei Zaharia added a comment -

          By the way, you may also want to try changing the path in ALLOC_FILE from ./test-pools to just test-pools. This is the biggest difference I've seen between my test and others that use data files. I wouldn't expect this to be a problem, but I see some Java bugs such as http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4879985 which could potentially be related.

          I'm also a little confused as to how the test/data directory should be created. I thought that the build-contrib.xml ant file simply set the test's working dir to the data directory, but did not actually create it. However, for streaming for example, the data directory does seem to be created without the tests explicitly mkdir'ing it. Am I missing something in my build file?

          Show
          Matei Zaharia added a comment - By the way, you may also want to try changing the path in ALLOC_FILE from ./test-pools to just test-pools. This is the biggest difference I've seen between my test and others that use data files. I wouldn't expect this to be a problem, but I see some Java bugs such as http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4879985 which could potentially be related. I'm also a little confused as to how the test/data directory should be created. I thought that the build-contrib.xml ant file simply set the test's working dir to the data directory, but did not actually create it. However, for streaming for example, the data directory does seem to be created without the tests explicitly mkdir'ing it. Am I missing something in my build file?
          Hide
          Matei Zaharia added a comment -

          This is odd because TestFairScheduler worked on Linux both on my machine and on Hudson (check out the last Hudson reply for HADOOP-3746). I think there might be a permission problem or a JRE bug such as this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6182812. Can you tell me what version of Java you have, what command you used to run the test, whether the directory /home/tsz/hadoop/latest/build/contrib/fairscheduler/test/data exists, and what are its permissions? If you look at the code for the test, it first calls mkdirs on the test's working directory and then tries to write the file; so since the mkdirs didn't fail, it looks like the directory was created or already existed, but it failed to create the file.

          Show
          Matei Zaharia added a comment - This is odd because TestFairScheduler worked on Linux both on my machine and on Hudson (check out the last Hudson reply for HADOOP-3746 ). I think there might be a permission problem or a JRE bug such as this one: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6182812 . Can you tell me what version of Java you have, what command you used to run the test, whether the directory /home/tsz/hadoop/latest/build/contrib/fairscheduler/test/data exists, and what are its permissions? If you look at the code for the test, it first calls mkdirs on the test's working directory and then tries to write the file; so since the mkdirs didn't fail, it looks like the directory was created or already existed, but it failed to create the file.
          Owen O'Malley made changes -
          Assignee Matei Zaharia [ matei@eecs.berkeley.edu ]
          Hide
          Owen O'Malley added a comment -

          Matei, I assume it makes sense to assign this to you. Are you going to be able to address it this week?

          Show
          Owen O'Malley added a comment - Matei, I assume it makes sense to assign this to you. Are you going to be able to address it this week?
          Tsz Wo Nicholas Sze made changes -
          Link This issue relates to HADOOP-3746 [ HADOOP-3746 ]
          Tsz Wo Nicholas Sze made changes -
          Field Original Value New Value
          Environment Linux
          Hide
          Tsz Wo Nicholas Sze added a comment -

          The test works fine on Windows.

          Show
          Tsz Wo Nicholas Sze added a comment - The test works fine on Windows.
          Tsz Wo Nicholas Sze created issue -

            People

            • Assignee:
              Matei Zaharia
              Reporter:
              Tsz Wo Nicholas Sze
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development