Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-12197

Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

    XMLWordPrintableJSON

    Details

      Description

      The original problem: Serializing GenericRecords in Spark Core results in a very high overhead, as the schema is serialized per record. (When in the actual input data of HDFS it's stored once per file. )

      The extended problem: Spark 1.5 introduced the ability to register Avro schemas ahead of time using SparkConf. This solution is partial as some applications may not know exactly which schemas they're going to read ahead of time.

      Extended solution:
      Adding a schema repository to the Serializer. Assuming the generic record has schemaId on them, it's possible to extract them dynamically from the read records and serialize only the schemaId.
      Upon deserialization the schemaRepo will be queried once again.

      The local caching mechanism will remain in tact - so in fact each Task will query the schema repo only once per schemaId.

      The previous static registering of schemas will remain in place, as it is more efficient when the schemas are known ahead of time.

      New flow of serializing generic record:
      1) check the pre-registered schema list, if found the schema, serialize only its finger print
      2) if not found, and schema repo has been set, attempt to extract the schemaId from record and check if repo contains the id. If so - serialize only the schema id
      3) if no schema repo set or didn't find the schemaId in repo - compress and send the entire schema.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Rotem Shaul Rotem Shaul
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - 72h
                72h
                Remaining:
                Remaining Estimate - 72h
                72h
                Logged:
                Time Spent - Not Specified
                Not Specified