[DRILL-405] Not able to query directory containing json files

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0
• Assignee: Unassigned
• Reporter: Ramana Inukonda Nagaraj
• Created: 10/Mar/14 20:19
• Updated: 17/Apr/15 00:07
• Resolved: 20/Apr/14 04:11

Description

Trying to query a directory which contains json files fails. The same test works for parquet files. Also if entire path upto the file is specified the json file can be queried successfully.

0: jdbc:drill:schema=dfs> select * from dfs.`/user/root/testJsonDir`;
Query failed: org.apache.drill.exec.rpc.RpcException: Remote failure while running query.[error_id: "afb20373-1010-4d2d-8a88-d89015db5801"
endpoint {
address: "perfnode166.perf.lab"
user_port: 31010
control_port: 31011
data_port: 31012
}
error_type: 0
message: "Failure while parsing sql. < ValidationException:[ org.eigenbase.util.EigenbaseContextException: From line 1, column 15 to line 1, column 42 ] < EigenbaseContextException:[ From line 1, column 15 to line 1, column 42 ] < SqlValidatorException:[ Table \'dfs./user/root/testJsonDir\' not found ]"
]
Error: exception while executing query (state=,code=0)

Log file has

13:16:57.398 [WorkManager-2] DEBUG o.a.d.e.w.fragment.FragmentExecutor - Fragment runner complete. 0:0
13:17:03.797 [WorkManager Event Thread] DEBUG o.apache.drill.exec.work.WorkManager - Starting pending task org.apache.drill.exec.work.foreman.Foreman@771fa2d1
13:17:03.806 [WorkManager-3] DEBUG o.a.d.e.s.dfs.WorkspaceSchemaFactory - File read failed.
java.io.IOException: Open failed for file: /user/root/testJsonDir, error: Invalid argument (22)
at com.mapr.fs.MapRClientImpl.open(MapRClientImpl.java:141) ~[maprfs-core-1.0.3-mapr-3.0.2.jar:na]
at com.mapr.fs.MapRFileSystem.open(MapRFileSystem.java:499) ~[maprfs-core-1.0.3-mapr-3.0.2.jar:na]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:463) ~[hadoop-0.20.2-dev-core.jar:na]
at org.apache.drill.exec.store.dfs.shim.fallback.FallbackFileSystem.open(FallbackFileSystem.java:89) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.dfs.BasicFormatMatcher$MagicStringMatcher.matches(BasicFormatMatcher.java:104) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable(BasicFormatMatcher.java:77) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.dfs.BasicFormatMatcher.isReadable(BasicFormatMatcher.java:64) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.parquet.ParquetFormatPlugin$ParquetFormatMatcher.isReadable(ParquetFormatPlugin.java:163) ~[drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:77) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.create(WorkspaceSchemaFactory.java:35) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.getNewEntry(ExpandingConcurrentMap.java:83) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap.containsKey(ExpandingConcurrentMap.java:64) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.planner.sql.ExpandingConcurrentMap$DelegatingKeySet.contains(ExpandingConcurrentMap.java:160) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at com.google.common.collect.Maps$AsMapView.containsKey(Maps.java:788) [guava-14.0.1.jar:na]
at net.hydromatic.optiq.util.CompositeMap.get(CompositeMap.java:91) [optiq-core-0.4.18.jar:na]
at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTableFrom(OptiqCatalogReader.java:89) [optiq-core-0.4.18.jar:na]
at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:77) [optiq-core-0.4.18.jar:na]
at net.hydromatic.optiq.prepare.OptiqCatalogReader.getTable(OptiqCatalogReader.java:43) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.IdentifierNamespace.validateImpl(IdentifierNamespace.java:67) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:80) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:753) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:742) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:2672) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:2899) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:60) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:80) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:753) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:742) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.SqlSelect.validate(SqlSelect.java:135) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:707) [optiq-core-0.4.18.jar:na]
at org.eigenbase.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:423) [optiq-core-0.4.18.jar:na]
at net.hydromatic.optiq.prepare.PlannerImpl.validate(PlannerImpl.java:153) [optiq-core-0.4.18.jar:na]
at org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan(DrillSqlWorker.java:76) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.runSQL(Foreman.java:350) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at org.apache.drill.exec.work.foreman.Foreman.run(Foreman.java:175) [drill-java-exec-1.0.0-m2-incubating-SNAPSHOT-rebuffed.jar:1.0.0-m2-incubating-SNAPSHOT]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_45]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_45]
at java.lang.Thread.run(Thread.java:744) [na:1.7.0_45]

Issue Links

• is duplicated by: DRILL-453 Query on a directory throws exception