Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-453

Provide more flexibility in the way tasks are run


    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      The aim
      With HADOOP-3421 speaking about sharing a cluster among more than one organization (so potentially with non-cooperative users), and posts on the ML speaking about virtualization and the ability to re-use the TaskTracker's VM to run new tasks, it could be useful for admins to choose the way TaskRunners run their children.

      More specifically, it could be useful to provide a way to imprison a Task in its working directory, or in a virtual machine.
      In some cases, reusing the VM might be useful, since it seems that this feature is really wanted (HADOOP-249).

      What I propose is a new class, called called SeperateVMTaskWrapper which contains the current logic for running tasks in another JVM. This class extends another, called TaskWrapper, which could be inherited to provide new ways of running tasks.
      As part of this issue I would also like to provide two other TaskWrappers : the first would run the tasks as Thread of the TaskRunner's VM (if it is possible without too much changes), the second would use a fixed pool of local unix accounts to insulate tasks from each others (so potentially non-cooperating users will be hable to share a cluster, as described in HADOOP-3421).

      1. userBasedInsulator.sh
        2 kB
        Brice Arnould
      2. TaskWrapper_v0.patch
        25 kB
        Brice Arnould

        Issue Links


          Harsh J made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Owen O'Malley made changes -
          Project Hadoop Common [ 12310240 ] Hadoop Map/Reduce [ 12310941 ]
          Key HADOOP-3675 MAPREDUCE-453
          Component/s mapred [ 12310690 ]
          Brice Arnould made changes -
          Attachment userBasedInsulator.sh [ 12385191 ]
          Brice Arnould made changes -
          Link This issue blocks HADOOP-3421 [ HADOOP-3421 ]
          Brice Arnould made changes -
          Link This issue incorporates HADOOP-249 [ HADOOP-249 ]
          Brice Arnould made changes -
          Field Original Value New Value
          Attachment TaskWrapper_v0.patch [ 12385028 ]
          Brice Arnould created issue -


            • Assignee:
              Brice Arnould
              Brice Arnould
            • Votes:
              0 Vote for this issue
              16 Start watching this issue


              • Created: