Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-1685

Allow specifying sync in DataFileWriter.create

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • None
    • 1.8.0
    • java
    • None
    • Reviewed
    • Allows specifying the 16-byte sync when creating an Avro container file via DataFileWriter

    Description

      Currently DataFileWriter generates a random 16-byte sync each time a new file is created. This means that even if you write the exact same data in a new file writer, the file itself will be slightly different (specifically the sync will be different).

      I'd like to be able to generate the exact same file multiple times. To do so, I need a way to specify the 16-byte sync.

      I've created a patch that adds this functionality by adding an overload of the create() that takes a byte[] array as the third parameter. If the byte array is null then a random sync is generated using the same internal static generateSync() method as before. If it's not null then the length is checked and it's used as the sync. The other two overloads of create(...) have been modified to call the three parameter version with a null sync.

      The patch includes three additional tests to check the error cases (invalid length) and verify that generating the same file twice results in the same byte array (i.e. exact match).

      Attachments

        1. AVRO-1685.patch
          5 kB
          Sehrope Sarkuni

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            sehrope Sehrope Sarkuni
            sehrope Sehrope Sarkuni
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment