Hadoop Map/Reduce
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2489

Jobsplits with random hostnames can make the queue unusable


    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s:, 0.23.0
    • Fix Version/s:, 0.23.0
    • Component/s: jobtracker
    • Labels:
    • Hadoop Flags:


      We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else.

      I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve.

      1. MAPREDUCE-2489-mapred-v7.patch
        2 kB
        Jeffrey Naisbitt
      2. MAPREDUCE-2489-mapred-v6.patch
        6 kB
        Jeffrey Naisbitt
      3. MAPREDUCE-2489-mapred-v5.patch
        4 kB
        Jeffrey Naisbitt
      4. MAPREDUCE-2489-mapred-v4.patch
        6 kB
        Jeffrey Naisbitt
      5. MAPREDUCE-2489-mapred-v3.patch
        6 kB
        Jeffrey Naisbitt
      6. MAPREDUCE-2489-mapred-v2.patch
        6 kB
        Jeffrey Naisbitt
      7. MAPREDUCE-2489-mapred.patch
        6 kB
        Jeffrey Naisbitt
      8. MAPREDUCE-2489-0.20s-v6.patch
        7 kB
        Jeffrey Naisbitt
      9. MAPREDUCE-2489-0.20s-v5.patch
        9 kB
        Jeffrey Naisbitt
      10. MAPREDUCE-2489-0.20s-v4.patch
        8 kB
        Jeffrey Naisbitt
      11. MAPREDUCE-2489-0.20s-v3.patch
        6 kB
        Jeffrey Naisbitt
      12. MAPREDUCE-2489-0.20s-v2.patch
        16 kB
        Jeffrey Naisbitt
      13. MAPREDUCE-2489-0.20s.patch
        17 kB
        Jeffrey Naisbitt

        Issue Links


          No work has yet been logged on this issue.


            • Assignee:
              Jeffrey Naisbitt
              Jeffrey Naisbitt
            • Votes:
              0 Vote for this issue
              9 Start watching this issue


              • Created: