Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-587

Test programs support only default queue.

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.21.0
    • Component/s: test
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Following test programs always run on "default" queue even when other queues are passed as job parameter.

      DFSCIOTest
      DistributedFSCheck
      TestDFSIO
      Filebench
      Loadgen
      Nnbench

      1. jira.HDFS-587.mapreduce.branch-0.22.patch
        28 kB
        Erik Steffl
      2. jira.HDFS-587.hdfs.branch-0.22.patch
        2 kB
        Erik Steffl
      3. jira.HDFS-587.branch-0.20-internal.1.patch
        73 kB
        Erik Steffl
      4. jira.HDFS-587.branch-0.21.patch
        30 kB
        Erik Steffl
      5. jira.HDFS-587.branch-0.20-internal.patch
        74 kB
        Erik Steffl

        Issue Links

          Activity

          Hide
          Sreekanth Ramakrishnan added a comment -

          Currently on the MAPREDUCE-945 we have made the corresponding classes which ProgramDriver uses to extend Configured and implement Tool. Also the configuration used to launch job are construced not by new Configuration() rather thro' getConf() method which is implemented in Configured and we use ToolRunner.run() method to launch program taking care of generic options.

          Show
          Sreekanth Ramakrishnan added a comment - Currently on the MAPREDUCE-945 we have made the corresponding classes which ProgramDriver uses to extend Configured and implement Tool . Also the configuration used to launch job are construced not by new Configuration() rather thro' getConf() method which is implemented in Configured and we use ToolRunner.run() method to launch program taking care of generic options.
          Hide
          Erik Steffl added a comment -

          List of test files for the test in description (some of them do not match test mentioned in description exactly):

          ./src/test/hdfs-with-mr/org/apache/hadoop/fs/DFSCIOTest.java
          ./src/test/hdfs-with-mr/org/apache/hadoop/fs/DistributedFSCheck.java
          ./src/test/hdfs-with-mr/org/apache/hadoop/fs/TestDFSIO.java
          ./src/test/hdfs-with-mr/org/apache/hadoop/io/FileBench.java
          ./src/test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java
          ./src/test/hdfs-with-mr/org/apache/hadoop/hdfs/NNBench.java
          ./src/test/hdfs-with-mr/org/apache/hadoop/hdfs/NNBenchWithoutMR.java

          Show
          Erik Steffl added a comment - List of test files for the test in description (some of them do not match test mentioned in description exactly): ./src/test/hdfs-with-mr/org/apache/hadoop/fs/DFSCIOTest.java ./src/test/hdfs-with-mr/org/apache/hadoop/fs/DistributedFSCheck.java ./src/test/hdfs-with-mr/org/apache/hadoop/fs/TestDFSIO.java ./src/test/hdfs-with-mr/org/apache/hadoop/io/FileBench.java ./src/test/hdfs/org/apache/hadoop/fs/loadGenerator/TestLoadGenerator.java ./src/test/hdfs-with-mr/org/apache/hadoop/hdfs/NNBench.java ./src/test/hdfs-with-mr/org/apache/hadoop/hdfs/NNBenchWithoutMR.java
          Hide
          Erik Steffl added a comment -

          This patch is for 0.20 (pre-split), also brings in junit-4.5 (see change in ivy/libraries.properties) because this issue requires classes to extend Configured and implement Tool so they cannot extend TestCase as required by junit-3.8.1

          Unit test (ant test) runs with same results with both 3.8.1 and 4.5 versions of junit, the only failed test is TestHdfsProxy in contrib which relies on a library that does not exist anymore (not related to junit).

          Show
          Erik Steffl added a comment - This patch is for 0.20 (pre-split), also brings in junit-4.5 (see change in ivy/libraries.properties) because this issue requires classes to extend Configured and implement Tool so they cannot extend TestCase as required by junit-3.8.1 Unit test (ant test) runs with same results with both 3.8.1 and 4.5 versions of junit, the only failed test is TestHdfsProxy in contrib which relies on a library that does not exist anymore (not related to junit).
          Hide
          Erik Steffl added a comment -

          Note: my comment above (posted 04/Oct/09 05:51 PM) applies to jira.HDFS-587.branch-0.20-internal.patch

          The jira.HDFS-587.branch-0.21.patch is for branch 0.21, before the MR tests in HDFS were moved to MR (they are being moved right now, thats why no patch for trunk at the moment).

          Show
          Erik Steffl added a comment - Note: my comment above (posted 04/Oct/09 05:51 PM) applies to jira. HDFS-587 .branch-0.20-internal.patch The jira. HDFS-587 .branch-0.21.patch is for branch 0.21, before the MR tests in HDFS were moved to MR (they are being moved right now, thats why no patch for trunk at the moment).
          Hide
          gary murry added a comment -

          As long as the results don't change from 3.8.1 to 4.5 and it makes back ports easier, +1.

          Show
          gary murry added a comment - As long as the results don't change from 3.8.1 to 4.5 and it makes back ports easier, +1.
          Hide
          Konstantin Boudnik added a comment -

          +1 Seems Ok from JUnit standpoint.

          Show
          Konstantin Boudnik added a comment - +1 Seems Ok from JUnit standpoint.
          Hide
          Sreekanth Ramakrishnan added a comment -

          +1 to the attached patches.

          Show
          Sreekanth Ramakrishnan added a comment - +1 to the attached patches.
          Hide
          Erik Steffl added a comment -

          HDFS-587 is blocked by HDFS-641 because HDFS-587 is related to the same set of test that are (probably) being moved to mapreduce in HDFS-641

          Show
          Erik Steffl added a comment - HDFS-587 is blocked by HDFS-641 because HDFS-587 is related to the same set of test that are (probably) being moved to mapreduce in HDFS-641
          Hide
          Erik Steffl added a comment -

          Reuslts of ant test-patch on revision 820532:

          [exec] +1 overall.
          [exec]
          [exec] +1 @author. The patch does not contain any @author tags.
          [exec]
          [exec] +1 tests included. The patch appears to include 22 new or modified tests.
          [exec]
          [exec] +1 javadoc. The javadoc tool did not generate any warning messages.
          [exec]
          [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
          [exec]
          [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
          [exec]
          [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.

          Show
          Erik Steffl added a comment - Reuslts of ant test-patch on revision 820532: [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 22 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
          Hide
          Erik Steffl added a comment -

          /home/steffl/jira.HDFS-587.branch-0.20-internal.1.patch is a new patch for 0.20 that does not have the two tests in TestLoadGenerator.java that use -scriptFile option of loadGenerator (that functionality is missing in 0.20 version of LoadGenerator).

          Show
          Erik Steffl added a comment - /home/steffl/jira. HDFS-587 .branch-0.20-internal.1.patch is a new patch for 0.20 that does not have the two tests in TestLoadGenerator.java that use -scriptFile option of loadGenerator (that functionality is missing in 0.20 version of LoadGenerator).
          Hide
          Erik Steffl added a comment -

          How to run these test from command line:

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar TestDFSIO -Dmapred.job.queue.name=anotherQueue -write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar DFSCIOTest -Dmapred.job.queue.name=anotherQueue -write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar DistributedFSCheck -Dmapred.job.queue.name=anotherQueue -write

          This test is not part of this bug, it did use Configured and Tools already (i.e. no need to run it): ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar filebench

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-0.21.0-dev.jar org.apache.hadoop.fs.loadGenerator.StructureGenerator -maxDepth 10 -minWidth 1 -maxWidth 10 -numOfFiles 5 -avgFileSize 2 -outDir /homes/steffl/tmp -seed 1000 # doesn't use queue, no mapreduce

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-0.21.0-dev.jar org.apache.hadoop.fs.loadGenerator.TestLoadGenerator # doesn't use queue, no mapreduce

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar nnbench -Dmapred.job.queue.name=anotherQueue -operation create_write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar NNBenchWithoutMR -Dmapred.job.queue.name=anotherQueue -operation create_write

          Version 0.20 specifics:

          • different jar: hadoop-0.20.2-dev-test.jar
          • as far as I can tell there is no way to run NNBenchWithoutMR in 0.20, even in 0.21 it had to be added to HdfsWithMRTestDriver.java (don't know how it was ran before)
          Show
          Erik Steffl added a comment - How to run these test from command line: ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar TestDFSIO -Dmapred.job.queue.name=anotherQueue -write ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar DFSCIOTest -Dmapred.job.queue.name=anotherQueue -write ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar DistributedFSCheck -Dmapred.job.queue.name=anotherQueue -write This test is not part of this bug, it did use Configured and Tools already (i.e. no need to run it): ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar filebench ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-0.21.0-dev.jar org.apache.hadoop.fs.loadGenerator.StructureGenerator -maxDepth 10 -minWidth 1 -maxWidth 10 -numOfFiles 5 -avgFileSize 2 -outDir /homes/steffl/tmp -seed 1000 # doesn't use queue, no mapreduce ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-0.21.0-dev.jar org.apache.hadoop.fs.loadGenerator.TestLoadGenerator # doesn't use queue, no mapreduce ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar nnbench -Dmapred.job.queue.name=anotherQueue -operation create_write ./hadoop/bin/hadoop jar ./hadoop/hadoop-hdfs-hdfswithmr-test-0.21.0-dev.jar NNBenchWithoutMR -Dmapred.job.queue.name=anotherQueue -operation create_write Version 0.20 specifics: different jar: hadoop-0.20.2-dev-test.jar as far as I can tell there is no way to run NNBenchWithoutMR in 0.20, even in 0.21 it had to be added to HdfsWithMRTestDriver.java (don't know how it was ran before)
          Hide
          Erik Steffl added a comment -

          Patches for 0.22 added:

          https://issues.apache.org/jira/secure/attachment/12425362/jira.HDFS-587.mapreduce.branch-0.22.patch
          https://issues.apache.org/jira/secure/attachment/12425361/jira.HDFS-587.hdfs.branch-0.22.patch

          Most of the Tests were moved to mapreduce, except of TestLoadGenerator.java which remains in hdfs, that's why there are two patches for 0.22.

          Patches were done against:

          URL: http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk
          Repository Root: http://svn.apache.org/repos/asf
          Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
          Revision: 880959
          Node Kind: directory
          Schedule: normal
          Last Changed Author: omalley
          Last Changed Rev: 836063
          Last Changed Date: 2009-11-13 15:58:21 -0800 (Fri, 13 Nov 2009)

          URL: http://svn.apache.org/repos/asf/hadoop/hdfs/trunk
          Repository Root: http://svn.apache.org/repos/asf
          Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
          Revision: 881058
          Node Kind: directory
          Schedule: normal
          Last Changed Author: cos
          Last Changed Rev: 881017
          Last Changed Date: 2009-11-16 14:38:46 -0800 (Mon, 16 Nov 2009)

          Running tests for 0.22:

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar TestDFSIO -Dmapred.job.queue.name=anotherQueue -write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar DFSCIOTest -Dmapred.job.queue.name=anotherQueue -write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar DistributedFSCheck -Dmapred.job.queue.name=anotherQueue -write

          mkdir /tmp/loadGenerator
          ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.StructureGenerator -maxDepth 3 -minWidth 1 -maxWidth 3 -numOfFiles 100 -avgFileSize 2 -outDir /tmp/loadGenerator -seed 1000
          ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.DataGenerator -inDir /tmp/loadGenerator -root /loadGenerator
          ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.LoadGenerator -readProbability 0.5 -writeProbability 0.5 -root /loadGenerator -maxDelayBetweenOps 0 -numOfThreads 10 -elapsedTime 15 -startTime 0 -seed 40

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar nnbench -Dmapred.job.queue.name=anotherQueue -operation create_write

          ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar NNBenchWithoutMR -Dmapred.job.queue.name=anotherQueue -operation createWrite -bytesPerBlock 512 -numFiles 2 -blocksPerFile 2 -baseDir /NNBenchWithoutMR

          Note: DFSCIOTest error: ot complains that $HADOOP_HOME/hadoop/libhdfs/libhdfs.so.1 does not exist, it's a bit weird, the test needs libhdfs.so.1, hdfs_read and hdfs_write in $HADOOP_HOME/libhdfs so you might want to touch those files (it copies them to /tmp/DFSCIOTest and uses them as data files, it does not load libhdfs.so.1 as a library).

          You need to add queue (name anotherQueue) to $HADOOP_HOME/conf/mapred-queues.xml (OR $HADOOP_CONF_DIR/mapred-queues.xml if you have a separate config dir)

          Show
          Erik Steffl added a comment - Patches for 0.22 added: https://issues.apache.org/jira/secure/attachment/12425362/jira.HDFS-587.mapreduce.branch-0.22.patch https://issues.apache.org/jira/secure/attachment/12425361/jira.HDFS-587.hdfs.branch-0.22.patch Most of the Tests were moved to mapreduce, except of TestLoadGenerator.java which remains in hdfs, that's why there are two patches for 0.22. Patches were done against: URL: http://svn.apache.org/repos/asf/hadoop/mapreduce/trunk Repository Root: http://svn.apache.org/repos/asf Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68 Revision: 880959 Node Kind: directory Schedule: normal Last Changed Author: omalley Last Changed Rev: 836063 Last Changed Date: 2009-11-13 15:58:21 -0800 (Fri, 13 Nov 2009) URL: http://svn.apache.org/repos/asf/hadoop/hdfs/trunk Repository Root: http://svn.apache.org/repos/asf Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68 Revision: 881058 Node Kind: directory Schedule: normal Last Changed Author: cos Last Changed Rev: 881017 Last Changed Date: 2009-11-16 14:38:46 -0800 (Mon, 16 Nov 2009) Running tests for 0.22: ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar TestDFSIO -Dmapred.job.queue.name=anotherQueue -write ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar DFSCIOTest -Dmapred.job.queue.name=anotherQueue -write ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar DistributedFSCheck -Dmapred.job.queue.name=anotherQueue -write mkdir /tmp/loadGenerator ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.StructureGenerator -maxDepth 3 -minWidth 1 -maxWidth 3 -numOfFiles 100 -avgFileSize 2 -outDir /tmp/loadGenerator -seed 1000 ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.DataGenerator -inDir /tmp/loadGenerator -root /loadGenerator ./hadoop/bin/hadoop org.apache.hadoop.fs.loadGenerator.LoadGenerator -readProbability 0.5 -writeProbability 0.5 -root /loadGenerator -maxDelayBetweenOps 0 -numOfThreads 10 -elapsedTime 15 -startTime 0 -seed 40 ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar nnbench -Dmapred.job.queue.name=anotherQueue -operation create_write ./hadoop/bin/hadoop jar ./hadoop/hadoop-mapred-test-0.22.0-dev.jar NNBenchWithoutMR -Dmapred.job.queue.name=anotherQueue -operation createWrite -bytesPerBlock 512 -numFiles 2 -blocksPerFile 2 -baseDir /NNBenchWithoutMR Note: DFSCIOTest error: ot complains that $HADOOP_HOME/hadoop/libhdfs/libhdfs.so.1 does not exist, it's a bit weird, the test needs libhdfs.so.1, hdfs_read and hdfs_write in $HADOOP_HOME/libhdfs so you might want to touch those files (it copies them to /tmp/DFSCIOTest and uses them as data files, it does not load libhdfs.so.1 as a library). You need to add queue (name anotherQueue) to $HADOOP_HOME/conf/mapred-queues.xml (OR $HADOOP_CONF_DIR/mapred-queues.xml if you have a separate config dir)
          Hide
          Suresh Srinivas added a comment -

          Committed the HDFS part of the patch to trunk. Thank you Erik.

          Show
          Suresh Srinivas added a comment - Committed the HDFS part of the patch to trunk. Thank you Erik.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk-Commit #168 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/168/)
          . Add support for specifying queue name in mapreduce tests. Contributed by Erik Steffl.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #168 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk-Commit/168/ ) . Add support for specifying queue name in mapreduce tests. Contributed by Erik Steffl.
          Hide
          Hudson added a comment -

          Integrated in Hadoop-Hdfs-trunk #199 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/199/)
          . Add support for specifying queue name in mapreduce tests. Contributed by Erik Steffl.

          Show
          Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #199 (See http://hudson.zones.apache.org/hudson/job/Hadoop-Hdfs-trunk/199/ ) . Add support for specifying queue name in mapreduce tests. Contributed by Erik Steffl.
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h2.grid.sp2.yahoo.net #96 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h2.grid.sp2.yahoo.net/96/ )
          Hide
          Sreekanth Ramakrishnan added a comment -

          +1 to the patch.

          Opened MAPREDUCE-1377 to run the patch thro' hudson on the MAPREDUCE branch. Also, since this is fixed on HDFS trunk can we close this JIRA as resolved fixed?

          @Erik, can you assign yourself the JIRA : MAPREDUCE-1377

          Show
          Sreekanth Ramakrishnan added a comment - +1 to the patch. Opened MAPREDUCE-1377 to run the patch thro' hudson on the MAPREDUCE branch. Also, since this is fixed on HDFS trunk can we close this JIRA as resolved fixed? @Erik, can you assign yourself the JIRA : MAPREDUCE-1377
          Hide
          Hudson added a comment -

          Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/)

          Show
          Hudson added a comment - Integrated in Hdfs-Patch-h5.grid.sp2.yahoo.net #196 (See http://hudson.zones.apache.org/hudson/job/Hdfs-Patch-h5.grid.sp2.yahoo.net/196/ )
          Hide
          Chris Douglas added a comment -

          This issue was resolved and committed to 0.22 from an HDFS perspective; it should be marked as such.

          Changes to MapReduce will be tracked in MAPREDUCE-1377.

          Show
          Chris Douglas added a comment - This issue was resolved and committed to 0.22 from an HDFS perspective; it should be marked as such. Changes to MapReduce will be tracked in MAPREDUCE-1377 .

            People

            • Assignee:
              Erik Steffl
              Reporter:
              Sreekanth Ramakrishnan
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development