Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16641

RPC: Heavy contention on Configuration.getClassByNameOrNull

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: conf
    • Labels:

      Description

      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589

              map = Collections.synchronizedMap(
                new WeakHashMap<String, WeakReference<Class<?>>>());
      

      This synchronizes all lookups across the same class-loader across all threads & yields rpc threads.

      When reading from HDFS with good locality, this fills up the contended lock profile with almost no other contributors to the locking - see llap-rpc-locks.svg

        Attachments

        1. llap-rpc-locks.svg
          35 kB
          Gopal Vijayaraghavan
        2. config-get-class-by-name.png
          72 kB
          Gopal Vijayaraghavan

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              gopalv Gopal Vijayaraghavan
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated: