Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.9.0
    • java
    • None

    Description

      New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag (org.apache.avro.specific.use_custom_coders) and (for now) turned off by default. See Getting Started (Java) for instructions.

      (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is slower for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)

      Attachments

        1. customcoders.md
          10 kB
          Raymie Stata
        2. perf-data.txt
          10 kB
          Raymie Stata

        Issue Links

        Activity

          iroh Chris Wells made changes -
          Workflow patch-available, re-open possible [ 13911495 ] no-reopen-closed, patch-avail [ 13915311 ]
          iroh Chris Wells made changes -
          Workflow no-reopen-closed, patch-avail [ 13780678 ] patch-available, re-open possible [ 13911495 ]
          gmcdonald Gavin McDonald made changes -
          Workflow patch-available, re-open possible [ 13751968 ] no-reopen-closed, patch-avail [ 13780678 ]
          gmcdonald Gavin McDonald made changes -
          Workflow no-reopen-closed, patch-avail [ 13366991 ] patch-available, re-open possible [ 13751968 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          cutting Doug Cutting made changes -
          Fix Version/s 1.9.0 [ 12333394 ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          raymie Raymie Stata made changes -
          Description New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag ({{org.apache.avro.specific.use_custom_coders}}) and (for now) turned off by default. See [Getting Started (Java)|https://avro.apache.org/docs/current/gettingstartedjava.html#Beta+feature:+Generating faster+code] for instructions.

          (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)
          New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag ({{org.apache.avro.specific.use_custom_coders}}) and (for now) turned off by default. See [Getting Started (Java)|https://avro.apache.org/docs/current/gettingstartedjava.html#Beta+feature:+Generating+faster+code] for instructions.

          (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          raymie Raymie Stata made changes -
          Description New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag ({{org.apache.avro.specific.use_custom_coders}}) and (for now) turned off by default. See [Getting Started (Java)|https://avro.apache.org/docs/current/gettingstartedjava.html#Beta+feature:+Faster+code+generation] for instructions.

          (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)
          New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag ({{org.apache.avro.specific.use_custom_coders}}) and (for now) turned off by default. See [Getting Started (Java)|https://avro.apache.org/docs/current/gettingstartedjava.html#Beta+feature:+Generating faster+code] for instructions.

          (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)
          raymie Raymie Stata made changes -
          Description Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord).

          This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords. We've run a public benchmark showing that the new code reduces serialization time by 2/3 and deserialization time by close to 50%.
          New implementation for generation of code for SpecificRecord that improves decoding by over 10% and encoding over 30% (more improvements are on the way). This feature is behind a feature flag ({{org.apache.avro.specific.use_custom_coders}}) and (for now) turned off by default. See [Getting Started (Java)|https://avro.apache.org/docs/current/gettingstartedjava.html#Beta+feature:+Faster+code+generation] for instructions.

          (A bit more info: Compared to GenericRecords, SpecificRecords offer type-safety plus the performance of traditional getters/setters/instance variables. But these are only beneficial to Java code accessing those records. SpecificRecords inherit serialization and deserialization code from GenericRecords, which is dynamic and thus slow (in fact, benchmarks show that serialization and deserialization is _slower_ for SpecificRecord than for GenericRecord). This patch extends record.vm to generate custom, higher-performance encoder and decoder functions for SpecificRecords.)
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          raymie Raymie Stata made changes -
          Attachment perf-data.txt [ 12944486 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ] This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #350 (Web Link)" [ 133428 ]
          raymie Raymie Stata made changes -
          Link This issue is blocked by AVRO-2235 [ AVRO-2235 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ] This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 110168 ]
          thiru_mg Thiruvalluvan M. G. made changes -
          Assignee Raymie Stata [ raymie ]
          githubbot ASF GitHub Bot made changes -
          Remote Link This issue links to "GitHub Pull Request #256 (Web Link)" [ 93135 ]
          raymie Raymie Stata made changes -
          Attachment customcoders.md [ 12890882 ]
          raymie Raymie Stata made changes -
          Link This issue blocks AVRO-2093 [ AVRO-2093 ]
          raymie Raymie Stata made changes -
          Link This issue blocks AVRO-2092 [ AVRO-2092 ]
          raymie Raymie Stata made changes -
          Link This issue blocks AVRO-2091 [ AVRO-2091 ]
          raymie Raymie Stata made changes -
          Link This issue blocks AVRO-2094 [ AVRO-2094 ]
          raymie Raymie Stata made changes -
          Link This issue is related to AVRO-2092 [ AVRO-2092 ]
          raymie Raymie Stata made changes -
          Link This issue is related to AVRO-2091 [ AVRO-2091 ]
          raymie Raymie Stata made changes -
          Link This issue is related to AVRO-2092 [ AVRO-2092 ]
          raymie Raymie Stata made changes -
          Field Original Value New Value
          Link This issue is related to AVRO-2091 [ AVRO-2091 ]
          raymie Raymie Stata created issue -

          People

            raymie Raymie Stata
            raymie Raymie Stata
            Votes:
            1 Vote for this issue
            Watchers:
            Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack