Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-2379

0.20: Allow block reports to proceed without holding FSDataset lock

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.0.1
    • Component/s: datanode
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      As disks are getting larger and more plentiful, we're seeing DNs with multiple millions of blocks on a single machine. When page cache space is tight, block reports can take multiple minutes to generate. Currently, during the scanning of the data directories to generate a report, the FSVolumeSet lock is held. This causes writes and reads to block, timeout, etc, causing big problems especially for clients like HBase.

      This JIRA is to explore some of the ideas originally discussed in HADOOP-4584 for the 0.20.20x series.

      1. hdfs-2379.txt
        32 kB
        Todd Lipcon
      2. hdfs-2379.txt
        32 kB
        Todd Lipcon
      3. hdfs-2379.txt
        32 kB
        Todd Lipcon
      4. hdfs-2379.txt
        32 kB
        Todd Lipcon
      5. hdfs-2379.txt
        31 kB
        Todd Lipcon
      6. hdfs-2379.txt
        11 kB
        Todd Lipcon

        Issue Links

          Activity

          Matt Foley made changes -
          Link This issue relates to HDFS-2282 [ HDFS-2282 ]
          Matt Foley made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Matt Foley made changes -
          Fix Version/s 1.1.0 [ 12317959 ]
          Affects Version/s 1.0.0 [ 12318243 ]
          Affects Version/s 1.1.0 [ 12317959 ]
          Suresh Srinivas made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Fix Version/s 1.0.1 [ 12319502 ]
          Resolution Fixed [ 1 ]
          Matt Foley made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Harsh J made changes -
          Link This issue is duplicated by HDFS-159 [ HDFS-159 ]
          Todd Lipcon made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Hadoop Flags Reviewed [ 10343 ]
          Fix Version/s 0.20.206.0 [ 12317959 ]
          Resolution Fixed [ 1 ]
          Todd Lipcon made changes -
          Assignee Todd Lipcon [ tlipcon ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12499493 ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12498656 ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12498514 ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12498389 ]
          Todd Lipcon made changes -
          Link This issue is related to HDFS-2384 [ HDFS-2384 ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12496960 ]
          Todd Lipcon made changes -
          Attachment hdfs-2379.txt [ 12496871 ]
          Todd Lipcon made changes -
          Link This issue relates to HADOOP-4584 [ HADOOP-4584 ]
          Todd Lipcon made changes -
          Field Original Value New Value
          Link This issue relates to HDFS-1280 [ HDFS-1280 ]
          Todd Lipcon created issue -

            People

            • Assignee:
              Todd Lipcon
              Reporter:
              Todd Lipcon
            • Votes:
              0 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development