Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16641

RPC: Heavy contention on Configuration.getClassByNameOrNull

Add voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • conf

    Description

      https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/conf/Configuration.java#L2589

              map = Collections.synchronizedMap(
                new WeakHashMap<String, WeakReference<Class<?>>>());
      

      This synchronizes all lookups across the same class-loader across all threads & yields rpc threads.

      When reading from HDFS with good locality, this fills up the contended lock profile with almost no other contributors to the locking - see llap-rpc-locks.svg

      Attachments

        1. config-get-class-by-name.png
          72 kB
          Gopal Vijayaraghavan
        2. llap-rpc-locks.svg
          35 kB
          Gopal Vijayaraghavan

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            gopalv Gopal Vijayaraghavan

            Dates

              Created:
              Updated:

              Slack

                Issue deployment