Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15160

ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock

    XMLWordPrintableJSON

Details

    Description

      Now we have HDFS-15150, we can start to move some DN operations to use the read lock rather than the write lock to improve concurrence. The first step is to make the changes to ReplicaMap, as many other methods make calls to it.

      This Jira switches read operations against the volume map to use the readLock rather than the write lock.

      Additionally, some methods make a call to replicaMap.replicas() (eg getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result in a read only fashion, so they can also be switched to using a readLock.

      Next is the directory scanner and disk balancer, which only require a read lock.

      Finally (for this Jira) are various "low hanging fruit" items in BlockSender and fsdatasetImpl where is it fairly obvious they only need a read lock.

      For now, I have avoided changing anything which looks too risky, as I think its better to do any larger refactoring or risky changes each in their own Jira.

      Attachments

        1. HDFS-15160.001.patch
          4 kB
          Stephen O'Donnell
        2. HDFS-15160.002.patch
          11 kB
          Stephen O'Donnell
        3. HDFS-15160.003.patch
          11 kB
          Stephen O'Donnell
        4. HDFS-15160.004.patch
          14 kB
          Stephen O'Donnell
        5. HDFS-15160.005.patch
          14 kB
          Stephen O'Donnell
        6. HDFS-15160.006.patch
          14 kB
          Stephen O'Donnell
        7. HDFS-15160.007.patch
          24 kB
          Stephen O'Donnell
        8. HDFS-15160.008.patch
          24 kB
          Stephen O'Donnell
        9. HDFS-15160.branch-3-3.001.patch
          23 kB
          Stephen O'Donnell
        10. HDFS-15160-branch-3.3-001.patch
          23 kB
          Akira Ajisaka
        11. image-2020-04-10-17-18-08-128.png
          795 kB
          Qi Zhu
        12. image-2020-04-10-17-18-55-938.png
          702 kB
          Qi Zhu

        Issue Links

          Activity

            People

              sodonnell Stephen O'Donnell
              sodonnell Stephen O'Donnell
              Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 7h 20m
                  7h 20m