Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
It looks like a number of doc pages can be improved by referencing some
changes made recently.
With the inclusion of the needed jars for s3a with Drill, there is no
longer a need to download jets3t [1]. In addition to setting your
credentials, this option for allowing more concurrent connections
(necessary to allow reads of wider parquet files) can also be set in this
block instead of a core-site.xml file [2].
This config block can actually be used to set any filesystem properties.
Some of these are custom to a particular filesystem like S3, but a number
of them are used by a variety of implementations of the HDFS interface. Any
properties like these [3] should be able to be set in this config block.
[1] -
https://drill.apache.org/blog/2014/12/09/running-sql-queries-on-amazon-s3/
[2] -
https://drill.apache.org/docs/s3-storage-plugin/#quering-parquet-format-files-on-s3
[3] -
https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/core-default.xml
Jason Altekruse
Software Engineer at Dremio
Apache Drill Committer
See email thread: Drill v1.6 and s3n connection