Uploaded image for project: 'Hadoop Map/Reduce'
  1. Hadoop Map/Reduce
  2. MAPREDUCE-6974

Add standard configuration keys for HTrace values, propagate across to MR committers if set

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 3.0.0-beta1
    • Fix Version/s: None
    • Component/s: job submission, performance
    • Labels:
      None

      Description

      HDFS &c support HTrace logging; HBase sets up spans.

      What doesn't do spans is MR jobs or other frameworks with use the MR committers.

      That can be addressed by defining some standard configuration keys for HTrace hi/low numbers, setting them in job submission, then passing them over the wire in the Configuration, where setupJob and setupTask can extract them & use for the tracing span. They could also add their own span for taskCommit and jobCommit for performance measurement there.

      Although the core code would be in MR, I'd propose putting the keys into Hadoop common,
      with some code in org.apache.hadoop.tracing.TraceUtils, to set it up. That way its possible to use more broadly.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: