Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-1816

Scan Error with JSON on large no of records with Complex Types

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Cannot Reproduce
    • Affects Version/s: None
    • Fix Version/s: 1.2.0
    • Component/s: Storage - JSON
    • Labels:
      None

      Description

      git.commit.id.abbrev=4a4f54a
      Memory Settings

      DRILL_MAX_DIRECT_MEMORY="32G"
      DRILL_MAX_HEAP="4G"
      

      Dataset :

      {
        "data" : {
          "col1" : {
            "one" : [1,2,3,4],
            "two" : [{"a":"b"},{"c":"d"}]
          }
        }
      }
      

      The below query works fine for the above record. However if we copy the same record 100,000 times, it fails with IOOB exception

      select data from `json_kvgenflatten/kvgen-complex-large.json`;
      

      Attached the logs. Let me know if you need anything more.

        Attachments

        1. complex.log
          142 kB
          Rahul Challapalli

          Issue Links

            Activity

              People

              • Assignee:
                rkins Rahul Challapalli
                Reporter:
                rkins Rahul Challapalli
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: