Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2379

0.20: Allow block reports to proceed without holding FSDataset lock

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.

      This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.

        Attachments

        1. hdfs-2379.txt
          11 kB
          Todd Lipcon
        2. hdfs-2379.txt
          31 kB
          Todd Lipcon
        3. hdfs-2379.txt
          32 kB
          Todd Lipcon
        4. hdfs-2379.txt
          32 kB
          Todd Lipcon
        5. hdfs-2379.txt
          32 kB
          Todd Lipcon
        6. hdfs-2379.txt
          32 kB
          Todd Lipcon

          Issue Links

            Activity

              People

              • Assignee:
                tlipcon Todd Lipcon
                Reporter:
                tlipcon Todd Lipcon
              • Votes:
                0 Vote for this issue
                Watchers:
                17 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: