Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-6325

Append should fail if the last block has insufficient number of replicas

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.4.1
    • Component/s: namenode
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      I have committed the fix to the trunk, branch-2, and branch-2.4 respectively. Thanks Keith!

      Description

      Currently append() succeeds on a file with the last block that has no replicas. But the subsequent updatePipeline() fails as there are no replicas with the exception "Unable to retrieve blocks locations for last block". This leaves the file unclosed, and others can not do anything with it until its lease expires.
      The solution is to check replicas of the last block on the NameNode and fail during append() rather than during updatePipeline().
      How many replicas should be present before NN allows to append? I see two options:

      1. min-replication: allow append if the last block is minimally replicated (1 by default)
      2. full-replication: allow append if the last block is fully replicated (3 by default)

        Attachments

        1. appendTest.patch
          2 kB
          Plamen Jeliazkov
        2. HDFS-6325_test.patch
          3 kB
          Keith Pak
        3. HDFS-6325.patch
          6 kB
          Keith Pak
        4. HDFS-6325.patch
          6 kB
          Keith Pak
        5. HDFS-6325.patch
          7 kB
          Keith Pak
        6. HDFS-6325.patch
          7 kB
          Keith Pak

          Activity

            People

            • Assignee:
              KeithPak Keith Pak
              Reporter:
              shv Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: