Hadoop HDFS
  1. Hadoop HDFS
  2. HDFS-4504

DFSOutputStream#close doesn't always release resources (such as leases)

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      DFSOutputStream#close can throw an IOException in some cases. One example is if there is a pipeline error and then pipeline recovery fails. Unfortunately, in this case, some of the resources used by the DFSOutputStream are leaked. One particularly important resource is file leases.

      So it's possible for a long-lived HDFS client, such as Flume, to write many blocks to a file, but then fail to close it. Unfortunately, the LeaseRenewerThread inside the client will continue to renew the lease for the "undead" file. Future attempts to close the file will just rethrow the previous exception, and no progress can be made by the client.

      1. HDFS-4504.001.patch
        12 kB
        Colin Patrick McCabe
      2. HDFS-4504.002.patch
        10 kB
        Colin Patrick McCabe
      3. HDFS-4504.007.patch
        28 kB
        Colin Patrick McCabe
      4. HDFS-4504.008.patch
        30 kB
        Colin Patrick McCabe
      5. HDFS-4504.009.patch
        30 kB
        Colin Patrick McCabe
      6. HDFS-4504.010.patch
        31 kB
        Colin Patrick McCabe
      7. HDFS-4504.011.patch
        33 kB
        Colin Patrick McCabe
      8. HDFS-4504.014.patch
        66 kB
        Colin Patrick McCabe
      9. HDFS-4504.015.patch
        70 kB
        Colin Patrick McCabe
      10. HDFS-4504.016.patch
        68 kB
        Colin Patrick McCabe

        Issue Links

          Activity

          Juan Yu made changes -
          Link This issue is related to HDFS-6652 [ HDFS-6652 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.016.patch [ 12599081 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.015.patch [ 12598294 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.014.patch [ 12598096 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.011.patch [ 12597800 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.010.patch [ 12597529 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.009.patch [ 12597191 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.008.patch [ 12597009 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.007.patch [ 12596972 ]
          Colin Patrick McCabe made changes -
          Link This issue is duplicated by HDFS-4855 [ HDFS-4855 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.002.patch [ 12579431 ]
          Colin Patrick McCabe made changes -
          Assignee Colin Patrick McCabe [ cmccabe ]
          Colin Patrick McCabe made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Colin Patrick McCabe made changes -
          Attachment HDFS-4504.001.patch [ 12579233 ]
          Andrew Wang made changes -
          Field Original Value New Value
          Link This issue blocks HDFS-4688 [ HDFS-4688 ]
          Colin Patrick McCabe created issue -

            People

            • Assignee:
              Colin Patrick McCabe
              Reporter:
              Colin Patrick McCabe
            • Votes:
              1 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

              • Created:
                Updated:

                Development