Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-4504

DFSOutputStream#close doesn't always release resources (such as leases)

    Details

    • Type: Bug
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      DFSOutputStream#close can throw an IOException in some cases. One example is if there is a pipeline error and then pipeline recovery fails. Unfortunately, in this case, some of the resources used by the DFSOutputStream are leaked. One particularly important resource is file leases.

      So it's possible for a long-lived HDFS client, such as Flume, to write many blocks to a file, but then fail to close it. Unfortunately, the LeaseRenewerThread inside the client will continue to renew the lease for the "undead" file. Future attempts to close the file will just rethrow the previous exception, and no progress can be made by the client.

        Attachments

        1. HDFS-4504.016.patch
          68 kB
          Colin P. McCabe
        2. HDFS-4504.015.patch
          70 kB
          Colin P. McCabe
        3. HDFS-4504.014.patch
          66 kB
          Colin P. McCabe
        4. HDFS-4504.011.patch
          33 kB
          Colin P. McCabe
        5. HDFS-4504.010.patch
          31 kB
          Colin P. McCabe
        6. HDFS-4504.009.patch
          30 kB
          Colin P. McCabe
        7. HDFS-4504.008.patch
          30 kB
          Colin P. McCabe
        8. HDFS-4504.007.patch
          28 kB
          Colin P. McCabe
        9. HDFS-4504.002.patch
          10 kB
          Colin P. McCabe
        10. HDFS-4504.001.patch
          12 kB
          Colin P. McCabe

          Issue Links

            Activity

              People

              • Assignee:
                cmccabe Colin P. McCabe
                Reporter:
                cmccabe Colin P. McCabe
              • Votes:
                1 Vote for this issue
                Watchers:
                38 Start watching this issue

                Dates

                • Created:
                  Updated: