Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-15160

ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Now we have HDFS-15150, we can start to move some DN operations to use the read lock rather than the write lock to improve concurrence. The first step is to make the changes to ReplicaMap, as many other methods make calls to it.

      This Jira switches read operations against the volume map to use the readLock rather than the write lock.

      Additionally, some methods make a call to replicaMap.replicas() (eg getBlockReports, getFinalizedBlocks, deepCopyReplica) and only use the result in a read only fashion, so they can also be switched to using a readLock.

      Next is the directory scanner and disk balancer, which only require a read lock.

      Finally (for this Jira) are various "low hanging fruit" items in BlockSender and fsdatasetImpl where is it fairly obvious they only need a read lock.

      For now, I have avoided changing anything which looks too risky, as I think its better to do any larger refactoring or risky changes each in their own Jira.

      Attachments

        1. HDFS-15160.001.patch
          4 kB
          Stephen O'Donnell
        2. HDFS-15160.002.patch
          11 kB
          Stephen O'Donnell
        3. HDFS-15160.003.patch
          11 kB
          Stephen O'Donnell
        4. HDFS-15160.004.patch
          14 kB
          Stephen O'Donnell
        5. HDFS-15160.005.patch
          14 kB
          Stephen O'Donnell
        6. HDFS-15160.006.patch
          14 kB
          Stephen O'Donnell
        7. HDFS-15160.007.patch
          24 kB
          Stephen O'Donnell
        8. HDFS-15160.008.patch
          24 kB
          Stephen O'Donnell
        9. HDFS-15160.branch-3-3.001.patch
          23 kB
          Stephen O'Donnell
        10. HDFS-15160-branch-3.3-001.patch
          23 kB
          Akira Ajisaka
        11. image-2020-04-10-17-18-08-128.png
          795 kB
          Qi Zhu
        12. image-2020-04-10-17-18-55-938.png
          702 kB
          Qi Zhu

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sodonnell Stephen O'Donnell
            sodonnell Stephen O'Donnell
            Votes:
            0 Vote for this issue
            Watchers:
            21 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 7h 10m
                7h 10m

                Issue deployment