Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
-
None
-
ghx-label-8
Description
Today, the catalog synthesizes block metadata for S3/ADLS by just breaking up splittable files into "blocks" with the FileSystem's default block size. Rather than carrying these blocks around in the catalog and distributing them to all impalad's, we might as well generate the scan ranges on-the-fly during planning. That would save the memory and network bandwidth of blocks.
That does mean that the planner will have to instantiate and call the filesystem to get the default block size, but for these FileSystem's, that's just a matter of reading the config.
Perhaps the same can be done for HDFS erasure coding, though that depends on what a block location actually means in that context and whether they contain useful info.
Attachments
Issue Links
- is related to
-
IMPALA-6729 Provide startup option to disable file and block location cache
- Open
- relates to
-
IMPALA-8944 Update and re-enable S3PlannerTest
- Resolved
Commit c84782e7466862e85e9ba534faa9f2baf85cc45e in impala's branch refs/heads/master from vercego
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c84782e ]
IMPALA-5931: Generates scan ranges in planner for s3/adlsCurrently, for filesystems that do not include physical
block information (e.g., block replica locations, caching),
synthetic blocks are generated and stored in the catalog
when metadata is loaded. Example file systems for which this is done
includes S3, ADLS, and local fs.
This change avoids generating these blocks when metadata is loaded.
Instead, scan ranges are directly generated from such files by the
backend coordinator. Previously, all scan ranges were produced by
the planner in HDFSScanNode in the frontend. Now, those files without
block information are sent to the coordinator represented by a split
specification that determines how the coordinator will create scan ranges
to send to executors.
This change reduces the space needed in the catalog and reduces the
scan range data structures that are passed from the frontend to the
backend when planning and coordinating a query.
In addition a bug is avoided where non-splittable files were being
split anyways to support the query parameter that places a limit on
scan ranges.
Testing:
partitions and observed the same scan ranges.
Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5
Reviewed-on: http://gerrit.cloudera.org:8080/8523
Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com>
Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>