Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-1140

Speedup INode.getPathComponents

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 0.22.0
    • 0.22.0
    • namenode
    • None
    • Reviewed

    Description

      When the namenode is loading the image there is a significant amount of time being spent in the DFSUtil.string2Bytes. We have a very specific workload here. The path that namenode does getPathComponents for shares N - 1 component with the previous path this method was called for (assuming current path has N components).
      Hence we can improve the image load time by caching the result of previous conversion.
      We thought of using some simple LRU cache for components, but the reality is, String.getBytes gets optimized during runtime and LRU cache doesn't perform as well, however using just the latest path components and their translation to bytes in two arrays gives quite a performance boost.
      I could get another 20% off of the time to load the image on our cluster (30 seconds vs 24) and I wrote a simple benchmark that tests performance with and without caching.

      Attachments

        1. HDFS-1140.patch
          4 kB
          Dmytro Molkov
        2. HDFS-1140.2.patch
          13 kB
          Dmytro Molkov
        3. HDFS-1140.3.patch
          13 kB
          Dmytro Molkov
        4. HDFS-1140.4.patch
          13 kB
          Dmytro Molkov

        Activity

          People

            dms Dmytro Molkov
            dms Dmytro Molkov
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: