Avro
  1. Avro
  2. AVRO-828

The 16-byte, sync marker for file not generated till first record written.

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Invalid
    • Affects Version/s: 1.5.1
    • Fix Version/s: None
    • Component/s: java
    • Environment:

      Java 1.6

      Description

      When opening an avro file to be written, the 16-byte, sync marker for file not generated till first record written or the file properly closed. Our usecase has the logs being rotated if they have no open filehandles which means if we open a file and experience an unclean shutdown we get files that throw exceptions on processing. When records are written the sync marker is written on each write which prevents this from being a problem on non-empty files. Can we write the sync marker when the header is written instead of on first record write?

        Activity

        Douglas Britsch created issue -
        Douglas Britsch made changes -
        Field Original Value New Value
        Description When opening an avro file to be written, the 16-byte, sync marker for file not generated till first record written or the file properly closed. Our usecase has the logs being rotated if they have no open filehandles which means if we open a file and experience an unclean shutdown we get files that throw exceptions on processing. When records are written the sync marker is written on each write which prevents this from being a problem on non-empty files. Can we write the sync marker when the header is written instead of on first write? When opening an avro file to be written, the 16-byte, sync marker for file not generated till first record written or the file properly closed. Our usecase has the logs being rotated if they have no open filehandles which means if we open a file and experience an unclean shutdown we get files that throw exceptions on processing. When records are written the sync marker is written on each write which prevents this from being a problem on non-empty files. Can we write the sync marker when the header is written instead of on first record write?
        Hide
        Doug Cutting added a comment -

        From my examination of the source code it appears that a sync marker is written when the header is written, currently on line 146 of DataFileWriter.java.

        http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java?view=markup#l146

        Can you please provide a test case that illustrates the problem?

        Show
        Doug Cutting added a comment - From my examination of the source code it appears that a sync marker is written when the header is written, currently on line 146 of DataFileWriter.java. http://svn.apache.org/viewvc/avro/trunk/lang/java/avro/src/main/java/org/apache/avro/file/DataFileWriter.java?view=markup#l146 Can you please provide a test case that illustrates the problem?
        Hide
        Douglas Britsch added a comment -

        Sorry, I had a transitive dependency bringing in 1.4.0, I have verified that this is fixed in 1.5.1.
        Thanks much for looking into it!

        Show
        Douglas Britsch added a comment - Sorry, I had a transitive dependency bringing in 1.4.0, I have verified that this is fixed in 1.5.1. Thanks much for looking into it!
        Douglas Britsch made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 1.5.1 [ 12316197 ]
        Resolution Invalid [ 6 ]
        Doug Cutting made changes -
        Fix Version/s 1.5.1 [ 12316197 ]
        Doug Cutting made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        19h 2m 1 Douglas Britsch 26/May/11 18:43
        Resolved Resolved Closed Closed
        134d 2h 39m 1 Doug Cutting 07/Oct/11 21:23

          People

          • Assignee:
            Unassigned
            Reporter:
            Douglas Britsch
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development