Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2489

Jobsplits with random hostnames can make the queue unusable

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.20.205.0, 0.23.0
    • Fix Version/s: 0.20.205.0, 0.23.0
    • Component/s: jobtracker
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      We saw an issue where a custom InputSplit was returning invalid hostnames for the splits that were then causing the JobTracker to attempt to excessively resolve host names. This caused a major slowdown for the JobTracker. We should prevent invalid InputSplit hostnames from affecting everyone else.

      I propose we implement some verification for the hostnames to try to ensure that we only do DNS lookups on valid hostnames (and fail otherwise). We could also fail the job after a certain number of failures in the resolve.

        Attachments

        1. MAPREDUCE-2489-mapred-v7.patch
          2 kB
          Jeffrey Naisbitt
        2. MAPREDUCE-2489-mapred-v6.patch
          6 kB
          Jeffrey Naisbitt
        3. MAPREDUCE-2489-mapred-v5.patch
          4 kB
          Jeffrey Naisbitt
        4. MAPREDUCE-2489-mapred-v4.patch
          6 kB
          Jeffrey Naisbitt
        5. MAPREDUCE-2489-mapred-v3.patch
          6 kB
          Jeffrey Naisbitt
        6. MAPREDUCE-2489-mapred-v2.patch
          6 kB
          Jeffrey Naisbitt
        7. MAPREDUCE-2489-mapred.patch
          6 kB
          Jeffrey Naisbitt
        8. MAPREDUCE-2489-0.20s-v6.patch
          7 kB
          Jeffrey Naisbitt
        9. MAPREDUCE-2489-0.20s-v5.patch
          9 kB
          Jeffrey Naisbitt
        10. MAPREDUCE-2489-0.20s-v4.patch
          8 kB
          Jeffrey Naisbitt
        11. MAPREDUCE-2489-0.20s-v3.patch
          6 kB
          Jeffrey Naisbitt
        12. MAPREDUCE-2489-0.20s-v2.patch
          16 kB
          Jeffrey Naisbitt
        13. MAPREDUCE-2489-0.20s.patch
          17 kB
          Jeffrey Naisbitt

          Issue Links

            Activity

              People

              • Assignee:
                naisbitt Jeffrey Naisbitt
                Reporter:
                naisbitt Jeffrey Naisbitt
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: