Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2489

Jobsplits with random hostnames can make the queue unusable

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0, 0.23.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else.

      I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve.

      1. MAPREDUCE-2489-mapred-v7.patch
        2 kB
        Jeffrey Naisbitt
      2. MAPREDUCE-2489-0.20s-v6.patch
        7 kB
        Jeffrey Naisbitt
      3. MAPREDUCE-2489-mapred-v6.patch
        6 kB
        Jeffrey Naisbitt
      4. MAPREDUCE-2489-0.20s-v5.patch
        9 kB
        Jeffrey Naisbitt
      5. MAPREDUCE-2489-mapred-v5.patch
        4 kB
        Jeffrey Naisbitt
      6. MAPREDUCE-2489-0.20s-v4.patch
        8 kB
        Jeffrey Naisbitt
      7. MAPREDUCE-2489-mapred-v4.patch
        6 kB
        Jeffrey Naisbitt
      8. MAPREDUCE-2489-0.20s-v3.patch
        6 kB
        Jeffrey Naisbitt
      9. MAPREDUCE-2489-mapred-v3.patch
        6 kB
        Jeffrey Naisbitt
      10. MAPREDUCE-2489-0.20s-v2.patch
        16 kB
        Jeffrey Naisbitt
      11. MAPREDUCE-2489-0.20s.patch
        17 kB
        Jeffrey Naisbitt
      12. MAPREDUCE-2489-mapred-v2.patch
        6 kB
        Jeffrey Naisbitt
      13. MAPREDUCE-2489-mapred.patch
        6 kB
        Jeffrey Naisbitt

        Issue Links

          Activity

            People

            • Assignee:
              Jeffrey Naisbitt
              Reporter:
              Jeffrey Naisbitt
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development