Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-13703

Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.2.0, 3.1.1, 3.0.4
    • Component/s: performance
    • Labels:
      None

      Description

      The DFSClient creates a CorruptedBlocks object, which contains a HashMap, on every read call. In most cases, a read will not hit any corrupted blocks, and this hashmap is not used. It seems the JIT isn't smart enough to eliminate this allocation. We would be better off avoiding it and only allocating in the rare case when a corrupt block is hit.

      Removing this allocation reduced CPU usage of a TeraValidate job by about 10%.

        Attachments

        1. hdfs-13703.patch
          3 kB
          Todd Lipcon
        2. hdfs-13703.patch
          3 kB
          Todd Lipcon

          Activity

            People

            • Assignee:
              tlipcon Todd Lipcon
              Reporter:
              tlipcon Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: