Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-479

Add reduce ID to shuffle clienttrace

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 0.21.0
    • 0.21.0
    • None
    • None
    • Incompatible change, Reviewed
    • Adds Reduce Attempt ID to ClientTrace log messages, and adds Reduce Attempt ID to HTTP query string sent to mapOutputServlet. Extracts partition number from attempt ID.

    Description

      Current clienttrace messages from shuffles note only the destination map ID but not the source reduce ID. Having both source and destination ID of each shuffle enables full tracing of execution.

      Attachments

        1. HADOOP-6013.patch
          1 kB
          Jiaqi Tan
        2. MAPREDUCE-479.patch
          2 kB
          Jiaqi Tan
        3. MAPREDUCE-479-1.patch
          7 kB
          Jiaqi Tan
        4. MAPREDUCE-479-2.patch
          3 kB
          Jiaqi Tan
        5. MAPREDUCE-479-3.patch
          3 kB
          Jiaqi Tan
        6. MAPREDUCE-479-4.patch
          3 kB
          Jiaqi Tan

        Issue Links

          relates to

          Improvement - An improvement or enhancement to an existing feature or task. HADOOP-5876 Shuffling information logged to userlogs/attempt_####_###_r_###_#/syslogs

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              tanjiaqi Jiaqi Tan
              tanjiaqi Jiaqi Tan
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: