Avro
  1. Avro
  2. AVRO-985

Remove byte-by-byte copying in RecordBuilderBase.defaultValue for non-complex types

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.1
    • Fix Version/s: None
    • Component/s: java
    • Labels:
      None

      Description

      In one section of RecordBuilderBase.defaultValue(Field) (quoted below) a bytewise copy of the default object is created based on the JSON value provided. However, this is an extremely inefficient operation and causes large slowdowns when building large object sets, including latency spikes when the binary encoder flushes.

      A simple workaround for a majority of cases would be to have a separate code path for "primitives" (fixed, string, boolean, int, double, enum, float, bytes) that allows direct creation rather than a full bytewise copy (and subsequent deep copy).

      RecordBuilderBase.java:

          // If not cached, get the default Java value by encoding the default JSON
          // value and then decoding it:
          if (defaultValue == null) {
            ByteArrayOutputStream baos = new ByteArrayOutputStream();
            encoder = EncoderFactory.get().binaryEncoder(baos, encoder);
            ResolvingGrammarGenerator.encode(
                encoder, field.schema(), defaultJsonValue);
            encoder.flush();
            decoder = DecoderFactory.get().binaryDecoder(
                baos.toByteArray(), decoder);
            defaultValue = new GenericDatumReader(
                field.schema()).read(null, decoder);
            defaultSchemaValues.putIfAbsent(field.pos(), defaultValue);
          }
      

        Issue Links

          Activity

          Douglas Kaminsky created issue -
          Douglas Kaminsky made changes -
          Field Original Value New Value
          Affects Version/s 1.6.1 [ 12318847 ]
          Douglas Kaminsky made changes -
          Summary Remove byte-by-byte copying in RecordBuilderBase.defaultValue Remove byte-by-byte copying in RecordBuilderBase.defaultValue for non-complex types
          Douglas Kaminsky made changes -
          Component/s java [ 12312780 ]
          James Baldassari made changes -
          Link This issue is related to AVRO-839 [ AVRO-839 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Douglas Kaminsky
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development