Avro
  1. Avro
  2. AVRO-775

File handle leak in DataFileReader

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.0
    • Component/s: java
    • Labels:
      None
    • Environment:

      all

    • Hadoop Flags:
      Reviewed

      Description

      When a DataFileReader is constructed by passing it a java.io.File and the is subsequently closed, the underlying FileInputStream is not getting closed. It manifests as an error in mapreduce tests in Windows machines. The files produced by mappers/reducers are opened as DataFiles for checking and, due to this bug, the handles are not closed. So subsequent delete operation on the file fails. And future mapper/reducer complains that the output already exists.

      1. AVRO-775.patch
        0.8 kB
        Thiruvalluvan M. G.
      2. AVRO-775.patch
        1 kB
        Doug Cutting

        Activity

        Thiruvalluvan M. G. created issue -
        Hide
        Thiruvalluvan M. G. added a comment -

        This patch addresses the problem.

        Show
        Thiruvalluvan M. G. added a comment - This patch addresses the problem.
        Thiruvalluvan M. G. made changes -
        Field Original Value New Value
        Attachment AVRO-775.patch [ 12472731 ]
        Thiruvalluvan M. G. made changes -
        Status Open [ 1 ] Patch Available [ 10002 ]
        Hide
        Doug Cutting added a comment -

        Here's another version of the patch that instead changes the SeekableInput interface to implement Closeable.

        Show
        Doug Cutting added a comment - Here's another version of the patch that instead changes the SeekableInput interface to implement Closeable.
        Doug Cutting made changes -
        Attachment AVRO-775.patch [ 12472844 ]
        Hide
        Doug Cutting added a comment -

        I prefer using Closeable, as otherwise non-InputStream implementations of SeekableInput would not be closed. This is a slightly incompatible API change, so it would be good to get into 1.5.0. Unless there are objections, I'll commit this and roll another release candidate.

        Show
        Doug Cutting added a comment - I prefer using Closeable, as otherwise non-InputStream implementations of SeekableInput would not be closed. This is a slightly incompatible API change, so it would be good to get into 1.5.0. Unless there are objections, I'll commit this and roll another release candidate.
        Doug Cutting made changes -
        Fix Version/s 1.5.0 [ 12315282 ]
        Fix Version/s 1.5.1 [ 12316197 ]
        Hide
        Thiruvalluvan M. G. added a comment -

        +1.

        Doug's implementation is cleaner and more general.

        Show
        Thiruvalluvan M. G. added a comment - +1. Doug's implementation is cleaner and more general.
        Hide
        Doug Cutting added a comment -

        I committed this.

        Show
        Doug Cutting added a comment - I committed this.
        Doug Cutting made changes -
        Status Patch Available [ 10002 ] Resolved [ 5 ]
        Hadoop Flags [Reviewed]
        Assignee Thiruvalluvan M. G. [ thiru_mg ] Doug Cutting [ cutting ]
        Resolution Fixed [ 1 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Patch Available Patch Available
        2m 31s 1 Thiruvalluvan M. G. 05/Mar/11 03:00
        Patch Available Patch Available Resolved Resolved
        2d 21h 57m 1 Doug Cutting 08/Mar/11 00:57
        Resolved Resolved Closed Closed
        3d 23h 35m 1 Doug Cutting 12/Mar/11 00:32

          People

          • Assignee:
            Doug Cutting
            Reporter:
            Thiruvalluvan M. G.
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development