Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4574

Avro Plugin: Flatten does not work correctly on record items

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.8.0
    • Component/s: Storage - Avro
    • Labels:
      None

      Description

      If the schema looks like this:

      {
        "type" : "record",
        "name" : "MainRecord",
        "namespace" : "drizz.WriteAvroTestFileForDrill$",
        "fields" : [ {
          "name" : "elements",
          "type" : {
            "type" : "array",
            "items" : {
              "type" : "record",
              "name" : "NestedRecord",
              "fields" : [ {
                "name" : "field1",
                "type" : "int"
              } ]
            },
            "java-class" : "java.util.List"
          }
        } ]
      }
      

      and the contents looks like this (according to avro tojson command line utility)

      {"elements":[{"field1":0},{"field1":1},{"field1":2},{"field1":3},{"field1":4},{"field1":5},{"field1":6},{"field1":7},{"field1":8},{"field1":9}]}
      {"elements":[{"field1":0},{"field1":1},{"field1":2},{"field1":3},{"field1":4},{"field1":5},{"field1":6},{"field1":7},{"field1":8},{"field1":9}]}
      

      a query like

      select flatten(elements) from dfs.`/Users/j.schulte/data/avro-drill/no-union/`;

      yields exactly two rows:

      +---------------+
      |    EXPR$0     |
      +---------------+
      | {"field1":9}  |
      | {"field1":9}  |
      +---------------+
      

      as if only the last element in the array would survive.

      It works for primitive arrays.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                baunz Johannes Schulte
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: