Avro
  1. Avro
  2. AVRO-805

Java: add ProtobufDatumReader and ProtobufDatumWriter

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: java
    • Labels:
      None

      Description

      Add a ProtobufDatumWriter that accepts ProtocolBuffer-generated classes and writes them in Avro format, and a ProtobufDatumReader that reads Avro data into a Protobuf-generated class. This would permit storage of Protobuf-generated data structures in Avro data files.

      1. AVRO-805.patch
        25 kB
        Doug Cutting
      2. AVRO-805.patch
        28 kB
        Doug Cutting
      3. AVRO-805.patch
        29 kB
        Doug Cutting

        Issue Links

          Activity

          Hide
          Doug Cutting added a comment -

          Protobuf-generated classes extend Message, which permits efficient access to field values. The list of Protobuf fields can also be obtained through Protobuf's Message interface, permitting construction of an equivalent Avro schema.

          Show
          Doug Cutting added a comment - Protobuf-generated classes extend Message, which permits efficient access to field values. The list of Protobuf fields can also be obtained through Protobuf's Message interface, permitting construction of an equivalent Avro schema.
          Hide
          Doug Cutting added a comment -

          Here's an initial implementation of this. It still doesn't have full support & tests for enums, repeated fields, etc.

          Probably all protobuf 'optional' fields should be interpreted as unions with null.

          Show
          Doug Cutting added a comment - Here's an initial implementation of this. It still doesn't have full support & tests for enums, repeated fields, etc. Probably all protobuf 'optional' fields should be interpreted as unions with null.
          Hide
          Doug Cutting added a comment -

          Here's an updated version that handles default values and enums. Still need to add support for repeated fields, plus probably cache a few things for decent performance.

          Show
          Doug Cutting added a comment - Here's an updated version that handles default values and enums. Still need to add support for repeated fields, plus probably cache a few things for decent performance.
          Hide
          Doug Cutting added a comment -

          Here's a version that handles repeated fields and that caches some expensive operations.

          Show
          Doug Cutting added a comment - Here's a version that handles repeated fields and that caches some expensive operations.
          Hide
          Doug Cutting added a comment -

          I think this is ready to commit.

          Show
          Doug Cutting added a comment - I think this is ready to commit.
          Hide
          Doug Cutting added a comment -

          I committed this.

          Show
          Doug Cutting added a comment - I committed this.

            People

            • Assignee:
              Doug Cutting
              Reporter:
              Doug Cutting
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development