Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15554

Improve JIT performance for Configuration parsing

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.0
    • 3.2.0, 3.1.1, 3.0.4
    • conf, performance
    • None

    Description

      In investigating a performance regression for small tasks between Hadoop 2 and Hadoop 3, we found that the amount of time spent in JIT was significantly higher. Using jitwatch we were able to determine that, due to a combination of switching from DOM to SAX style parsing and just having more configuration key/value pairs, Configuration.loadResource is now getting compiled with the C2 compiler and taking quite some time. Breaking that very large function up into several smaller ones and eliminating some redundant bits of code improves the JIT performance measurably.

      Attachments

        1. HADOOP-15554.branch-3.0.patch
          16 kB
          Gergo Repas
        2. HADOOP-15554.branch-3.0.patch
          16 kB
          Gergo Repas
        3. hadoop-15554.patch
          18 kB
          Todd Lipcon
        4. hadoop-15554.patch
          17 kB
          Todd Lipcon

        Issue Links

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: