Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None
    • Release Note:
      Implemented Schema#isSuperset, Schema#unify, and a Tool for generating unified Schemas from avro data files.

      Description

      From my post on the mailing list, and Doug's response:

      On 05/05/2011 10:29 AM, Joe Crobak wrote:
      > We've recently come across a situation where we have two data files with
      > different schemas that we'd like to process together using
      > GenericDatumReader. One schema is promotable to the other, but not vice
      > versa. We'd like to programmatically determine which of the schemas to
      > use. I did a brief look through javadoc and tests, and I couldn't find
      > any examples of checking if one schema is promotable to the other. Has
      > anyone else come across this?
      >
      > For some context, we're considering patching AvroStorage [1] to remove
      > the assumption that all files have the same schema. In our case, our
      > schema has evolved in that a field that was an int was promoted to a long.

      A boolean method that tells you if one schema is promotable to another
      would work in this case, but would not help in cases where, e.g.,
      different fields had changed in different versions. For example, in
      branched development, two branches might each add a distinct symbol to
      an enum. So I think you might be better off with a method that, given
      two schemas, returns their superset, a schema that can read data written
      by either.

      Such a method does not yet exist in Avro, but should not be difficult to
      add. Please file an issue in Jira if this sounds of interest.

      Doug

      I think it would be useful to have both of the methods that Doug mentioned in some sort of schema utils class.

        Attachments

        1. AVRO-816.patch
          10 kB
          Joe Crobak
        2. AVRO-816.patch
          28 kB
          Joe Crobak
        3. AVRO-816.patch
          30 kB
          Joe Crobak
        4. AVRO-816.patch
          33 kB
          Joe Crobak

          Activity

            People

            • Assignee:
              joecrobak Joe Crobak
              Reporter:
              joecrobak Joe Crobak
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: