Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-5341

Reduce fsdataset lock duration during directory scanning.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.3.0
    • Component/s: datanode
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When DirectoryScanner doing the scan function, it holding the dataset to diff the block info between memory and disk.But it do a lot of disk operation because it call the file's getlength funciton.

      Once the dataset is locked,heartbeat thread and all DataXceiver thread would be blocked.

      So,such disk operation should move to the async disk scan

        Attachments

        1. HDFS-5341.004.patch
          2 kB
          qus-jiawei
        2. HDFS-5341.003.patch
          6 kB
          qus-jiawei
        3. HDFS-5341.002.patch
          5 kB
          qus-jiawei
        4. HDFS-5341.001.patch
          5 kB
          qus-jiawei
        5. HDFS-5341.000.patch
          2 kB
          qus-jiawei

          Activity

            People

            • Assignee:
              gtzygtzy qus-jiawei
              Reporter:
              gtzygtzy qus-jiawei
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: