Apache Drill
  1. Apache Drill
  2. DRILL-282

Support 'repeated' type in Json reader

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.4.0
    • Component/s: None
    • Labels:
      None

      Description

      Trying to scan the following json file

      {
      "Bigname": [

      {"name": "hello", "id": "1"}

      ,

      {"name": "world", "id": "2"}

      ]
      }

      using a simple query (via sqlline): select * from "filename.json"; results in the following error

      java.lang.IndexOutOfBoundsException
      at io.netty.buffer.EmptyByteBuf.checkIndex(EmptyByteBuf.java:857)
      at io.netty.buffer.EmptyByteBuf.getBytes(EmptyByteBuf.java:321)
      at org.apache.drill.exec.vector.VarCharVector$Accessor.get(VarCharVector.java:241)
      at org.apache.drill.exec.vector.VarCharVector$Accessor.getObject(VarCharVector.java:258)
      at org.apache.drill.exec.vector.NullableVarCharVector$Accessor.getObject(NullableVarCharVector.java:245)
      at org.apache.drill.sql.client.full.JsonHelper.get(JsonHelper.java:41)
      at org.apache.drill.sql.client.full.BatchLoaderMap.getCurrentObject(BatchLoaderMap.java:112)
      at org.apache.drill.sql.client.full.ResultEnumerator.moveNext(ResultEnumerator.java:51)
      at net.hydromatic.optiq.runtime.ObjectEnumeratorCursor.next(ObjectEnumeratorCursor.java:44)
      at net.hydromatic.optiq.jdbc.OptiqResultSet.next(OptiqResultSet.java:162)
      at sqlline.SqlLine$BufferedRows.<init>(SqlLine.java:2499)
      at sqlline.SqlLine.print(SqlLine.java:1886)
      at sqlline.SqlLine$Commands.execute(SqlLine.java:3835)
      at sqlline.SqlLine$Commands.sql(SqlLine.java:3738)
      at sqlline.SqlLine.dispatch(SqlLine.java:882)
      at sqlline.SqlLine.begin(SqlLine.java:717)
      at sqlline.SqlLine.mainWithInputRedirection(SqlLine.java:460)
      at sqlline.SqlLine.main(SqlLine.java:443)

      The same query works if the above json file is modified to be the following:

      {
      "Bigname": [

      {"name": "hello", "id": "1"}

      ]
      }

        Issue Links

          Activity

          Hide
          Ramana Inukonda Nagaraj added a comment -

          The behavior now is if repeated values exist the first value in the array is outputted .

          Show
          Ramana Inukonda Nagaraj added a comment - The behavior now is if repeated values exist the first value in the array is outputted .
          Hide
          Jacques Nadeau added a comment -

          We don't actually support the repeated map type yet in Milestone one. We're missing that datatype. I think Tim already created the functionality on the reader side but since we don't support in general execution layer, it doesn't work. Repeated map is used when you have an array of sub maps within a top level map. I think this is the problem you are encountering given the data described. I think this should be targeted for milestone 3.

          Show
          Jacques Nadeau added a comment - We don't actually support the repeated map type yet in Milestone one. We're missing that datatype. I think Tim already created the functionality on the reader side but since we don't support in general execution layer, it doesn't work. Repeated map is used when you have an array of sub maps within a top level map. I think this is the problem you are encountering given the data described. I think this should be targeted for milestone 3.
          Hide
          Ben Becker added a comment -

          This requires more work than just modifying the JSON reader; the repeated map type needs to be implemented across the board. I created DRILL-283 to encompass this work.

          Show
          Ben Becker added a comment - This requires more work than just modifying the JSON reader; the repeated map type needs to be implemented across the board. I created DRILL-283 to encompass this work.
          Hide
          Dhruv Gohil added a comment -

          Yes, easily reproducible with sample-data/donuts.json
          Kind of blocker.

          I can spend time, can anybody point me at core things to look at?

          Show
          Dhruv Gohil added a comment - Yes, easily reproducible with sample-data/donuts.json Kind of blocker. I can spend time, can anybody point me at core things to look at?

            People

            • Assignee:
              Timothy Chen
              Reporter:
              Mehant Baid
            • Votes:
              1 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development