Uploaded image for project: 'Apache Avro'
  1. Apache Avro
  2. AVRO-762

Better schema resolution

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • None
    • 1.5.0
    • c
    • None

    Description

      I've been working on a pretty major patch that changes the way the C library implements schema resolution. Before, we would compare the writer and reader schemas each time we try to read a record from an Avro file. This is a fair bit of wasted effort. The approach I'm taking with the new implementation is to separate schema resolution and binary parsing into separate operations. There's a new "consumer" API, which defines a set of callbacks for processing Avro data that conforms to a schema. The new avro_consume_binary function reads binary-encoded Avro data from a buffer or file, and passes that data into a consumer instance. Each consumer instance is associated with the writer schema of the data that it expects to process.

      Schema resolution is now implemented in the new avro_resolver_new function, which returns a consumer instance that knows how to translate from the writer schema to the reader schema. As the resolver receives data via the consumer API, it fills in the contents of a destination avro_datum_t (which should be an instance of the reader schema).

      This work isn't complete yet — I still have to implement promotion (int->long and friends), and have to add support for recursive schemas (via the AVRO_LINK schema type). But I wanted to get the patch out there for people to view and test in the meantime. This patch depends on a few other of my patches, that haven't made it into SVN yet; if you want to test the code without patching yourself, I have a tracking branch on github.

      Attachments

        1. 0003-Recursive-schema-resolution.patch
          30 kB
          Douglas Creager
        2. 0002-Promotion-of-values-during-schema-resolution.patch
          16 kB
          Douglas Creager
        3. 0001-Better-schema-resolution.patch
          70 kB
          Douglas Creager

        Activity

          People

            dcreager Douglas Creager
            dcreager Douglas Creager
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: