Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      The job init thread currently initializes one job at a time. However, this is a lengthy and partly IO-bound process because all of the job's block locations need to be resolved through the namenode and a map of them needs to be built. It can take tens of seconds. As a result, the cluster sometimes initializes jobs too slowly for full utilization to be achieved, if there are many small jobs queued up. It would be better to have a pool of threads that initialize multiple jobs in parallel. One thing to be careful of, however, is not causing deadlocks or holding locks for too long in these threads.

      1. hadoop-4664-v1.patch
        13 kB
        Jothi Padmanabhan
      2. hadoop-4664-v2.patch
        14 kB
        Jothi Padmanabhan
      3. hadoop-4664-v3.patch
        14 kB
        Jothi Padmanabhan
      4. hadoop-4664-v4.patch
        14 kB
        Jothi Padmanabhan
      5. parallel-job-init-v1.patch
        5 kB
        Matei Zaharia

        Issue Links

          Activity

          Matei Zaharia created issue -
          Matei Zaharia made changes -
          Field Original Value New Value
          Attachment parallel-job-init-v1.patch [ 12394016 ]
          Matei Zaharia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Chris Douglas made changes -
          Assignee Matei Zaharia [ matei@eecs.berkeley.edu ]
          Tom White made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hemanth Yamijala made changes -
          Priority Major [ 3 ] Blocker [ 1 ]
          Fix Version/s 0.20.0 [ 12313438 ]
          Raghu Angadi made changes -
          Link This issue incorporates HADOOP-5286 [ HADOOP-5286 ]
          Hemanth Yamijala made changes -
          Assignee Matei Zaharia [ matei@eecs.berkeley.edu ] Jothi Padmanabhan [ jothipn ]
          Jothi Padmanabhan made changes -
          Attachment hadoop-4664-v1.patch [ 12401313 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment hadoop-4664-v2.patch [ 12401498 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Eric Yang made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Eric Yang made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Attachment hadoop-4664-v3.patch [ 12402042 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Jothi Padmanabhan made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Jothi Padmanabhan made changes -
          Attachment hadoop-4664-v4.patch [ 12402051 ]
          Jothi Padmanabhan made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Devaraj Das made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Hadoop Flags [Reviewed]
          Resolution Fixed [ 1 ]
          Nigel Daley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Owen O'Malley made changes -
          Component/s mapred [ 12310690 ]

            People

            • Assignee:
              Jothi Padmanabhan
              Reporter:
              Matei Zaharia
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development