[AVRO-3026] Allow custom annotations in IDL files and support translating them to AVSC Avro.

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.9.0, 1.9.1, 1.9.2, 1.10.1
• Component/s: spec
• Assignee: Unassigned
• Reporter: Feroze Daud
• Created: 19/Jan/21 02:17
• Updated: 23/Nov/21 11:11
• Resolved: 23/Nov/21 11:11

Description

Introduction

Our company has standardized on Avro schemas for all Data intestion and storage. As part of this, and to satisfy CCPA, we need to be able to tag the records and fields appropriately if they have PI, or Non PI information, etc.

Avro AVSC files, being valid json, can easily be modified to add tags that will be used by downstream processors, and also wont interfere with Avro itself ( to generate POJO, serialization, deserialization etc)

One such key we chose is simply called tags. It's example usage is shown below.

{
   "type": "record",
   "name": "PropertyOwner",
   "namespace": "com.acme.Property", 
   "tags": ["PI", "PII" ],
   "fields": [
   {
      "name": "FullName",
      "type": "string",
      "tags": ["Name"]
   },
   {
       "name": "PhoneNumber",
       "type": "string",
       "tags": ["Phone"]
   }],
}

 

These tags can be processed by downstream processors and the data landing in datalake, or database can be tagged appropriately.

 

Problem Description

While tagging will work fine for AVSC because adding extra fields doesnt make it invalid, we will have a problem when using IDL to author schemas. IDL spec does not allow a way to add extra tags that are copied over to the Avro schema.

 

Proposal

I propose that we allow a special @annotation tag . And, this tag can be applied to records and fields. Whatever is in this annotation should be copied verbatim to the output AVSC.

For eg:

@annotation("tags", "[\"PI\", \"Non PI\"]"
record Employee {
  @annotation("tags", "[\"Name\"]"
  string fullName;
  boolean active = true;
  long salary;
  @annotation("tags", "[\"Phone\"]"
  string phone;
} 

 

would generate an avro schema as folllows:

 

 {
 "type": "record",
 "name": "Employee",
 "tags": ["PI", "PII" ],
 "fields": [
 {
 "name": "FullName",
 "type": "string",
 "tags": ["Name"]
 },
 { 
  "name": "PhoneNumber", 
  "type": "string", 
  "tags": ["Phone"] 
 }],
}

 

As you can see, we dont need to support any wellformed JSONness in the @annotation . It just takes a string and we just render it into the output json.

@annotation("foo", "[\"bar\"]") -> "tags": ["bar"]

@annotation("foo", "{\"bar\": \"jar\"}") -> "tags":

{"bar": "jar"}