Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6652

Add configuration property to prevent JHS from loading jobs with a task count greater than X

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha1
    • Component/s: jobhistoryserver
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Jobs with large number of tasks can have job history files that are large in size and resource-consuming(mainly memory) to parse in Job History Server. If there are many such jobs, the job history server can very easily hang.

      It would be a good usability feature if we added a new config property that could be set to X, where the JHS wouldn't load the details for a job with more than X tasks. The job would still show up on the list of jobs page, but clicking on it would give a warning message that the job is too big, instead of actually loading the job. This way we can prevent users from loading a job that's way too big for the JHS, which currently makes the JHS hang. The default value can be -1 so that it's disabled.

        Attachments

        1. mapreduce6652.009.branch2.patch
          29 kB
          Haibo Chen
        2. mapreduce6652.009.patch
          29 kB
          Haibo Chen
        3. mapreduce6652.008.branch2.patch
          29 kB
          Haibo Chen
        4. mapreduce6652.008.patch
          29 kB
          Haibo Chen
        5. mapreduce6652.007.patch
          29 kB
          Haibo Chen
        6. mapreduce6652.007.branch2.patch
          29 kB
          Haibo Chen
        7. mapreduce6652.005.patch
          29 kB
          Haibo Chen
        8. mapreduce6652.004.patch
          27 kB
          Haibo Chen
        9. mapreduce6652.003.patch
          27 kB
          Haibo Chen
        10. mapreduce6652.002.patch
          27 kB
          Haibo Chen
        11. mapreduce6652.001.patch
          26 kB
          Haibo Chen

          Activity

            People

            • Assignee:
              haibochen Haibo Chen
              Reporter:
              haibochen Haibo Chen
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: