[DRILL-1816] Scan Error with JSON on large no of records with Complex Types

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Fix Version/s: 1.2.0
• Component/s: Storage - JSON
• Assignee: Rahul Kumar Challapalli
• Reporter: Rahul Kumar Challapalli
• Created: 05/Dec/14 21:03
• Updated: 06/Oct/15 23:27
• Resolved: 07/Jul/15 01:17

Description

git.commit.id.abbrev=4a4f54a
Memory Settings

DRILL_MAX_DIRECT_MEMORY="32G"
DRILL_MAX_HEAP="4G"

Dataset :

{
  "data" : {
    "col1" : {
      "one" : [1,2,3,4],
      "two" : [{"a":"b"},{"c":"d"}]
    }
  }
}

The below query works fine for the above record. However if we copy the same record 100,000 times, it fails with IOOB exception

select data from `json_kvgenflatten/kvgen-complex-large.json`;

Attached the logs. Let me know if you need anything more.

Issue Links

• is depended upon by: DRILL-1810 Scalability issues with kvgen