Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-1818

RaidNode should specify a pool name incase the cluster is using FairScheduler

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 0.20.1
    • Fix Version/s: 0.22.0
    • Component/s: contrib/raid
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      contrib/fairscheduler (FairScheduler) supports scheduling based on pools. The RaidNode should specify a pool name based on configuration to make use of pools.

      1. MAPREDUCE-1818.2.patch
        2 kB
        Ramkumar Vadali
      2. MAPREDUCE-1818.patch
        66 kB
        Ramkumar Vadali

        Activity

        Ramkumar Vadali created issue -
        Hide
        Scott Chen added a comment -

        It will be nice if we can make this more general.
        Raid should allow users to configure raid job in raid.xml.
        This way people use CapacityScheduler can also specify how to run the raid job.
        What do you think?

        Show
        Scott Chen added a comment - It will be nice if we can make this more general. Raid should allow users to configure raid job in raid.xml. This way people use CapacityScheduler can also specify how to run the raid job. What do you think?
        Hide
        Ramkumar Vadali added a comment -

        That is a good idea. Any pointers on what sort of configuration is useful for Capacity Scheduler?

        Show
        Ramkumar Vadali added a comment - That is a good idea. Any pointers on what sort of configuration is useful for Capacity Scheduler?
        Hide
        Scott Chen added a comment -

        Capacity scheduler should have a configuration to specify which queue to use.
        I don't know the exact key.

        I think we should allow any kind of job configuration.
        And raid should not know about the details of the names and configuration keys.
        We can configure a parameter like raid.jobconf.location in each policy in raid.xml
        This raid.jobconf.location specify a path of a conf file for the raid job.
        For each policy, we can load the corresponding configuration file and passing them to the jobconf.

        This way we can support any parameters.
        And we don't have to create extra dependency between raid and other contrib projects.
        What do you think?

        Show
        Scott Chen added a comment - Capacity scheduler should have a configuration to specify which queue to use. I don't know the exact key. I think we should allow any kind of job configuration. And raid should not know about the details of the names and configuration keys. We can configure a parameter like raid.jobconf.location in each policy in raid.xml This raid.jobconf.location specify a path of a conf file for the raid job. For each policy, we can load the corresponding configuration file and passing them to the jobconf. This way we can support any parameters. And we don't have to create extra dependency between raid and other contrib projects. What do you think?
        Ramkumar Vadali made changes -
        Field Original Value New Value
        Assignee Ramkumar Vadali [ rvadali ]
        Hide
        Ramkumar Vadali added a comment -

        Attaching a patch that provides a generic way of specifying scheduler options. The conf variable raid.scheduleroption can be specified to have multiple key-value pairs in the form key:value,key:value...

        Show
        Ramkumar Vadali added a comment - Attaching a patch that provides a generic way of specifying scheduler options. The conf variable raid.scheduleroption can be specified to have multiple key-value pairs in the form key:value,key:value...
        Ramkumar Vadali made changes -
        Attachment MAPREDUCE-1818.patch [ 12458188 ]
        Ramkumar Vadali made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Ramkumar Vadali added a comment -

        Update after svn up

        Show
        Ramkumar Vadali added a comment - Update after svn up
        Ramkumar Vadali made changes -
        Attachment MAPREDUCE-1818.2.patch [ 12458192 ]
        Hide
        Ramkumar Vadali added a comment -

        Test results:

        ant test-patch

             [exec]
             [exec]
             [exec] +1 overall.
             [exec]
             [exec]     +1 @author.  The patch does not contain any @author tags.
             [exec]
             [exec]     +1 tests included.  The patch appears to include 2 new or modified tests.
             [exec]
             [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
             [exec]
             [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler warnings.
             [exec]
             [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
             [exec]
             [exec]     +1 release audit.  The applied patch does not increase the total number of release audit warnings.
             [exec]
             [exec]     +1 system tests framework.  The patch passed system tests framework compile.
             [exec]
             [exec]
             [exec]
             [exec]
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]     Finished build.
             [exec] ======================================================================
             [exec] ======================================================================
             [exec]
             [exec]
        
        BUILD SUCCESSFUL
        Total time: 16 minutes 6 seconds
        

        ant test under src/contrib/raid

        
        test-junit:
            [junit] WARNING: multiple versions of ant detected in path for junit 
            [junit]          jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class
            [junit]      and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class
            [junit] Running org.apache.hadoop.hdfs.TestRaidDfs
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 47.02 sec
            [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal
            [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 8.871 sec
            [junit] Running org.apache.hadoop.raid.TestErasureCodes
            [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 26.827 sec
            [junit] Running org.apache.hadoop.raid.TestGaloisField
            [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.4 sec
            [junit] Running org.apache.hadoop.raid.TestHarIndexParser
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec
            [junit] Running org.apache.hadoop.raid.TestRaidHar
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.009 sec
            [junit] Running org.apache.hadoop.raid.TestRaidNode
            [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 462.311 sec
            [junit] Running org.apache.hadoop.raid.TestRaidPurge
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 44.044 sec
            [junit] Running org.apache.hadoop.raid.TestRaidShell
            [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 21.899 sec
        
        test:
        
        BUILD SUCCESSFUL
        Total time: 11 minutes 37 seconds
        
        
        Show
        Ramkumar Vadali added a comment - Test results: ant test-patch [exec] [exec] [exec] +1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 2 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system tests framework. The patch passed system tests framework compile. [exec] [exec] [exec] [exec] [exec] ====================================================================== [exec] ====================================================================== [exec] Finished build. [exec] ====================================================================== [exec] ====================================================================== [exec] [exec] BUILD SUCCESSFUL Total time: 16 minutes 6 seconds ant test under src/contrib/raid test-junit: [junit] WARNING: multiple versions of ant detected in path for junit [junit] jar:file:/home/rvadali/local/external/ant/lib/ant.jar!/org/apache/tools/ant/Project.class [junit] and jar:file:/home/rvadali/.ivy2/cache/ant/ant/jars/ant-1.6.5.jar!/org/apache/tools/ant/Project.class [junit] Running org.apache.hadoop.hdfs.TestRaidDfs [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 47.02 sec [junit] Running org.apache.hadoop.raid.TestDirectoryTraversal [junit] Tests run: 2, Failures: 0, Errors: 0, Time elapsed: 8.871 sec [junit] Running org.apache.hadoop.raid.TestErasureCodes [junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 26.827 sec [junit] Running org.apache.hadoop.raid.TestGaloisField [junit] Tests run: 7, Failures: 0, Errors: 0, Time elapsed: 0.4 sec [junit] Running org.apache.hadoop.raid.TestHarIndexParser [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 0.052 sec [junit] Running org.apache.hadoop.raid.TestRaidHar [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 69.009 sec [junit] Running org.apache.hadoop.raid.TestRaidNode [junit] Tests run: 4, Failures: 0, Errors: 0, Time elapsed: 462.311 sec [junit] Running org.apache.hadoop.raid.TestRaidPurge [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 44.044 sec [junit] Running org.apache.hadoop.raid.TestRaidShell [junit] Tests run: 1, Failures: 0, Errors: 0, Time elapsed: 21.899 sec test: BUILD SUCCESSFUL Total time: 11 minutes 37 seconds
        Hide
        Scott Chen added a comment -

        +1 Looks good to me.

        Show
        Scott Chen added a comment - +1 Looks good to me.
        Hide
        Scott Chen added a comment -

        I just committed this. Thanks Ram.

        Show
        Scott Chen added a comment - I just committed this. Thanks Ram.
        Scott Chen made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Fix Version/s 0.22.0 [ 12314184 ]
        Resolution Fixed [ 1 ]
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #524 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/524/)
        MAPREDUCE-1818. RaidNode can specify scheduling parameters. (Ramkumar Vadali
        via schen)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #524 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk-Commit/524/ ) MAPREDUCE-1818 . RaidNode can specify scheduling parameters. (Ramkumar Vadali via schen)
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/)

        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #643 (See https://hudson.apache.org/hudson/job/Hadoop-Mapreduce-trunk/643/ )
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        154d 59m 1 Ramkumar Vadali 27/Oct/10 21:22
        Patch Available Patch Available Resolved Resolved
        1d 5h 13m 1 Scott Chen 29/Oct/10 02:36

          People

          • Assignee:
            Ramkumar Vadali
            Reporter:
            Ramkumar Vadali
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development