Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-8531

Append failed due to unreleased lease from previous appender with quota exceeded exception

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 2.7.0
    • Fix Version/s: None
    • Component/s: namenode
    • Labels:
      None

      Description

      Append operation fails if
      1. Set SpaceQuota to 3G for /user/hrt_qa/heterogenous
      2. Copy 1GB file to /user/hrt_qa/heterogenous
      3. Run 'hdfs dfs -appendToFile' to append 1GB file to the already copied 1GB file in /user/hrt_qa/heterogenous
      4. Append fails with Quota exceed message
      5. Increase the Quota to 5G for /user/hrt_qa/heterogenous
      6. Run the Append message again and the following error message comes.

      appendToFile: Failed to APPEND_FILE /user/hrt_qa/quotaPerHerterogenousStorage/1GBFile_1/part-m-00000 for DFSClient_NONMAPREDUCE_1431576997_1 on 172.31.37.190 because this file lease is currently owned by DFSClient_NONMAPREDUCE_-231994503_1 on 172.31.37.190
      

      7. Wait for a while (lease soft limit – 1 min) and retry append it succeeds after the previous lease expire.

      This seems to relate to HDFS-7587
      "
      When a client was trying to append to the file, the remaining space quota was very small. This caused a failure in prepareFileForWrite(), but after the inode was already converted for writing and a lease added. Since these were not undone when the quota violation was detected, the file was left in under-construction with an active lease without edit logging OP_ADD.
      A subsequent append() eventually caused a lease recovery after the soft limit period. This resulted in commitBlockSynchronization(), which closed the file with OP_CLOSE being logged."

      The fix in HDFS-7587 addressed the edit log corruption issue but does not handle the problem within soft limit period where other clients can't append.

        Attachments

        1. HDFS-8531.00.patch
          6 kB
          Xiaoyu Yao

          Issue Links

            Activity

              People

              • Assignee:
                xyao Xiaoyu Yao
                Reporter:
                ssathish@hortonworks.com Sumana Sathish
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: