Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-11770 [Umbrella] locate static synchronized blocks in hadoop-common
  3. HADOOP-12475

Replace guava Cache with ConcurrentHashMap for caching Connection in ipc Client

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • conf, io, ipc
    • None
    • Reviewed

    Description

      quote daryn from HADOOP-11772:

      CacheBuilder is obscenely expensive for concurrent map, and it requires generating unnecessary garbage even just to look up a key. Replace it with ConcurrentHashMap.

      I identified this issue that impaired my own perf testing under load. The slowdown isn't just the sync. It's the expensive of Connection's ctor stalling other connections. The expensive of ConnectionId#equals causes delays. Synch'ing on connections causes unfair contention unlike a sync'ed method. Concurrency simply hides this.

      BTW, guava Cache is heavyweight. Per local test, ConcurrentHashMap has better overal performance.

      Attachments

        1. HADOOP-12475.03.patch
          5 kB
          Walter Su
        2. HADOOP-12475.02.patch
          5 kB
          Walter Su
        3. HADOOP-12475.01.patch
          5 kB
          Walter Su

        Activity

          People

            walter.k.su Walter Su
            walter.k.su Walter Su
            Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: