Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-2657 TaskTracker should handle disk failures
  3. MAPREDUCE-2415

Distribute TaskTracker userlogs onto multiple disks

    XMLWordPrintableJSON

Details

    • Reviewed

    Description

      Currently, userlogs directory in TaskTracker is placed under hadoop.log.dir like <hadoop.log.dir>/userlogs. I am proposing to spread these userlogs onto multiple configured mapred.local.dirs to strengthen TaskTracker reliability w.r.t disk failures.

      Attachments

        1. MAPREDUCE-2415-1.patch
          18 kB
          Bharath Mundlapudi
        2. TaskTracker Userlogs Design.pdf
          46 kB
          Bharath Mundlapudi
        3. MAPREDUCE-2415-2.patch
          36 kB
          Bharath Mundlapudi
        4. MAPREDUCE-2415-3.patch
          36 kB
          Bharath Mundlapudi

        Issue Links

          breaks

          Sub-task - The sub-task of the issue MAPREDUCE-2920 Local log dir links in the JT web UI are broken

          • Major - Major loss of function.
          • Closed
          is blocked by

          Sub-task - The sub-task of the issue MAPREDUCE-2413 TaskTracker should handle disk failures at both startup and runtime

          • Major - Major loss of function.
          • Closed
          is related to

          Sub-task - The sub-task of the issue MAPREDUCE-2851 Add test for userlogs on multiple disks (MR-2415)

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2846 a small % of all tasks fail with DefaultTaskController

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. MAPREDUCE-2804 "Creation of symlink to attempt log dir failed." message is not useful

          • Blocker - Blocks development and/or testing work, production could not run
          • Closed

          Improvement - An improvement or enhancement to an existing feature or task. MAPREDUCE-4481 User Log Retention across TT restarts

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Reopened

          Activity

            People

              bharathm Bharath Mundlapudi
              bharathm Bharath Mundlapudi
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: