Uploaded image for project: 'Avro'
  1. Avro
  2. AVRO-805

Java: add ProtobufDatumReader and ProtobufDatumWriter

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: java
    • Labels:
      None

      Description

      Add a ProtobufDatumWriter that accepts ProtocolBuffer-generated classes and writes them in Avro format, and a ProtobufDatumReader that reads Avro data into a Protobuf-generated class. This would permit storage of Protobuf-generated data structures in Avro data files.

      1. AVRO-805.patch
        25 kB
        Doug Cutting
      2. AVRO-805.patch
        28 kB
        Doug Cutting
      3. AVRO-805.patch
        29 kB
        Doug Cutting

        Issue Links

          Activity

          Hide
          cutting Doug Cutting added a comment -

          I committed this.

          Show
          cutting Doug Cutting added a comment - I committed this.
          Hide
          cutting Doug Cutting added a comment -

          I think this is ready to commit.

          Show
          cutting Doug Cutting added a comment - I think this is ready to commit.
          Hide
          cutting Doug Cutting added a comment -

          Here's a version that handles repeated fields and that caches some expensive operations.

          Show
          cutting Doug Cutting added a comment - Here's a version that handles repeated fields and that caches some expensive operations.
          Hide
          cutting Doug Cutting added a comment -

          Here's an updated version that handles default values and enums. Still need to add support for repeated fields, plus probably cache a few things for decent performance.

          Show
          cutting Doug Cutting added a comment - Here's an updated version that handles default values and enums. Still need to add support for repeated fields, plus probably cache a few things for decent performance.
          Hide
          cutting Doug Cutting added a comment -

          Here's an initial implementation of this. It still doesn't have full support & tests for enums, repeated fields, etc.

          Probably all protobuf 'optional' fields should be interpreted as unions with null.

          Show
          cutting Doug Cutting added a comment - Here's an initial implementation of this. It still doesn't have full support & tests for enums, repeated fields, etc. Probably all protobuf 'optional' fields should be interpreted as unions with null.
          Hide
          cutting Doug Cutting added a comment -

          Protobuf-generated classes extend Message, which permits efficient access to field values. The list of Protobuf fields can also be obtained through Protobuf's Message interface, permitting construction of an equivalent Avro schema.

          Show
          cutting Doug Cutting added a comment - Protobuf-generated classes extend Message, which permits efficient access to field values. The list of Protobuf fields can also be obtained through Protobuf's Message interface, permitting construction of an equivalent Avro schema.

            People

            • Assignee:
              cutting Doug Cutting
              Reporter:
              cutting Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development