[DRILL-5382] Error: Missing function implementation: [isnotnull(MAP-REQUIRED)]

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.11.0
• Component/s: Execution - Flow
• Assignee: Unassigned
• Reporter: Khurram Faraaz
• Created: 24/Mar/17 18:56
• Updated: 30/Mar/17 04:56

Description

Projecting a map from JSON data and filter non null values results in SchemaChangeException, Error: Missing function implementation: [isnotnull(MAP-REQUIRED)].

Data used in test is available here - wget http://data.githubarchive.org/2015-01-01-15.json.gz
Drill 1.11.0 commit id: adbf363

0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json` where org is not null;
Error: SYSTEM ERROR: SchemaChangeException: Failure while trying to materialize incoming schema.  Errors:

Error in expression at index -1.  Error: Missing function implementation: [isnotnull(MAP-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--..

Fragment 0:0

[Error Id: 3a776c68-6476-4bd8-a9e6-928bfc2ef5bd on centos-01.qa.lab:31010] (state=,code=0)
0: jdbc:drill:schema=dfs.tmp>

Select without IS NOT NULL filter, returns correct results.

apache drill 1.11.0-SNAPSHOT
"just drill it"
0: jdbc:drill:schema=dfs.tmp> select org from `2015-01-01-15.json`;
+-----+
| org |
+-----+
| {} |
| {} |
| {} |
| {"id":9285252,"login":"visionmedia","gravatar_id":"","url":"https://api.github.com/orgs/visionmedia","avatar_url":"https://avatars.githubusercontent.com/u/9285252?"} |
| {} |
| {} |
| {} |
...
...
| {} |
| {} |
| {} |
| {"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"} |
| {} |
| {"id":296074,"login":"zendframework","gravatar_id":"","url":"https://api.github.com/orgs/zendframework","avatar_url":"https://avatars.githubusercontent.com/u/296074?"} |
| {} |
| {} |
| {} |
| {} |
| {"id":9216151,"login":"osp","gravatar_id":"","url":"https://api.github.com/orgs/osp","avatar_url":"https://avatars.githubusercontent.com/u/9216151?"} |
| {} |
| {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} |
| {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} |
| {} |
| {"id":5822862,"login":"SuaveIO","gravatar_id":"","url":"https://api.github.com/orgs/SuaveIO","avatar_url":"https://avatars.githubusercontent.com/u/5822862?"} |
| {"id":2918581,"login":"twbs","gravatar_id":"","url":"https://api.github.com/orgs/twbs","avatar_url":"https://avatars.githubusercontent.com/u/2918581?"} |
| {"id":1104713,"login":"s9y","gravatar_id":"","url":"https://api.github.com/orgs/s9y","avatar_url":"https://avatars.githubusercontent.com/u/1104713?"} |
| {} |
| {} |
| {} |
| {} |
+-----+
11,351 rows selected (0.865 seconds)

Stack trace from drillbit.log

Caused by: org.apache.drill.exec.exception.SchemaChangeException: Failure while trying to materialize incoming schema.  Errors:

Error in expression at index -1.  Error: Missing function implementation: [isnotnull(MAP-REQUIRED)].  Full expression: --UNKNOWN EXPRESSION--..
        at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.generateSV2Filterer(FilterRecordBatch.java:186) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.filter.FilterRecordBatch.setupNewSchema(FilterRecordBatch.java:111) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:78) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.svremover.RemovingRecordBatch.innerNext(RemovingRecordBatch.java:93) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:119) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:109) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractSingleRecordBatch.innerNext(AbstractSingleRecordBatch.java:51) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:135) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:162) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:81) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:232) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:226) ~[drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        at java.security.AccessController.doPrivileged(Native Method) ~[na:1.8.0_91]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[na:1.8.0_91]
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1595) ~[hadoop-common-2.7.0-mapr-1607.jar:na]
        at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:226) [drill-java-exec-1.11.0-SNAPSHOT.jar:1.11.0-SNAPSHOT]
        ... 4 common frames omitted