Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-2379

0.20: Allow block reports to proceed without holding FSDataset lock

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Fixed
    • 1.0.0
    • 1.0.1
    • datanode
    • None
    • Reviewed

    Description

      As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.

      This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.

      Attachments

        1. hdfs-2379.txt
          32 kB
          Todd Lipcon
        2. hdfs-2379.txt
          32 kB
          Todd Lipcon
        3. hdfs-2379.txt
          32 kB
          Todd Lipcon
        4. hdfs-2379.txt
          32 kB
          Todd Lipcon
        5. hdfs-2379.txt
          31 kB
          Todd Lipcon
        6. hdfs-2379.txt
          11 kB
          Todd Lipcon

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. HDFS-159 Block reports should be processed offline

          • Major - Major loss of function.
          • Resolved
          is related to

          Improvement - An improvement or enhancement to an existing feature or task. HDFS-2384 Improve speed of block report generation

          • Major - Major loss of function.
          • Open
          relates to

          Bug - A problem which impairs or prevents the functions of the product. HDFS-1280 SocketTimeoutException during HDFS Block report generation

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. HDFS-2282 Semi-harmless race between block reports and block invalidation

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. HADOOP-4584 Slow generation of blockReport at DataNode causes delay of sending heartbeat to NameNode

          • Major - Major loss of function.
          • Closed

          Activity

            People

              tlipcon Todd Lipcon
              tlipcon Todd Lipcon
              Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: