Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5931

Don't synthesize block metadata in the catalog for S3/ADLS

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • Impala 2.13.0, Impala 3.1.0
    • Catalog
    • None

    Description

      Today, the catalog synthesizes block metadata for S3/ADLS by just breaking up splittable files into "blocks" with the FileSystem's default block size. Rather than carrying these blocks around in the catalog and distributing them to all impalad's, we might as well generate the scan ranges on-the-fly during planning. That would save the memory and network bandwidth of blocks.

      That does mean that the planner will have to instantiate and call the filesystem to get the default block size, but for these FileSystem's, that's just a matter of reading the config.

      Perhaps the same can be done for HDFS erasure coding, though that depends on what a block location actually means in that context and whether they contain useful info.

      Attachments

        Issue Links

          Activity

            Commit c84782e7466862e85e9ba534faa9f2baf85cc45e in impala's branch refs/heads/master from vercego
            [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c84782e ]

            IMPALA-5931: Generates scan ranges in planner for s3/adls

            Currently, for filesystems that do not include physical
            block information (e.g., block replica locations, caching),
            synthetic blocks are generated and stored in the catalog
            when metadata is loaded. Example file systems for which this is done
            includes S3, ADLS, and local fs.

            This change avoids generating these blocks when metadata is loaded.
            Instead, scan ranges are directly generated from such files by the
            backend coordinator. Previously, all scan ranges were produced by
            the planner in HDFSScanNode in the frontend. Now, those files without
            block information are sent to the coordinator represented by a split
            specification that determines how the coordinator will create scan ranges
            to send to executors.

            This change reduces the space needed in the catalog and reduces the
            scan range data structures that are passed from the frontend to the
            backend when planning and coordinating a query.
            In addition a bug is avoided where non-splittable files were being
            split anyways to support the query parameter that places a limit on
            scan ranges.

            Testing:

            • added backend scheduler tests
            • mixed-filesystems test covers tables/queries with multiple fs's.
            • local fs tests cover the code paths in this change
            • all core tests pass when configured with s3
            • manually tried larger local filesystem tables (tpch) with multiple
              partitions and observed the same scan ranges.
            • TODO: adls testing

            Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5
            Reviewed-on: http://gerrit.cloudera.org:8080/8523
            Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com>
            Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            jira-bot ASF subversion and git services added a comment - Commit c84782e7466862e85e9ba534faa9f2baf85cc45e in impala's branch refs/heads/master from vercego [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=c84782e ] IMPALA-5931 : Generates scan ranges in planner for s3/adls Currently, for filesystems that do not include physical block information (e.g., block replica locations, caching), synthetic blocks are generated and stored in the catalog when metadata is loaded. Example file systems for which this is done includes S3, ADLS, and local fs. This change avoids generating these blocks when metadata is loaded. Instead, scan ranges are directly generated from such files by the backend coordinator. Previously, all scan ranges were produced by the planner in HDFSScanNode in the frontend. Now, those files without block information are sent to the coordinator represented by a split specification that determines how the coordinator will create scan ranges to send to executors. This change reduces the space needed in the catalog and reduces the scan range data structures that are passed from the frontend to the backend when planning and coordinating a query. In addition a bug is avoided where non-splittable files were being split anyways to support the query parameter that places a limit on scan ranges. Testing: added backend scheduler tests mixed-filesystems test covers tables/queries with multiple fs's. local fs tests cover the code paths in this change all core tests pass when configured with s3 manually tried larger local filesystem tables (tpch) with multiple partitions and observed the same scan ranges. TODO: adls testing Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Reviewed-on: http://gerrit.cloudera.org:8080/8523 Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            Commit 11554a17c75b242767d5a50d66bc2874aa545c77 in impala's branch refs/heads/2.x from vercego
            [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=11554a1 ]

            IMPALA-5931: Generates scan ranges in planner for s3/adls

            Currently, for filesystems that do not include physical
            block information (e.g., block replica locations, caching),
            synthetic blocks are generated and stored in the catalog
            when metadata is loaded. Example file systems for which this is done
            includes S3, ADLS, and local fs.

            This change avoids generating these blocks when metadata is loaded.
            Instead, scan ranges are directly generated from such files by the
            backend coordinator. Previously, all scan ranges were produced by
            the planner in HDFSScanNode in the frontend. Now, those files without
            block information are sent to the coordinator represented by a split
            specification that determines how the coordinator will create scan ranges
            to send to executors.

            This change reduces the space needed in the catalog and reduces the
            scan range data structures that are passed from the frontend to the
            backend when planning and coordinating a query.
            In addition a bug is avoided where non-splittable files were being
            split anyways to support the query parameter that places a limit on
            scan ranges.

            Testing:

            • added backend scheduler tests
            • mixed-filesystems test covers tables/queries with multiple fs's.
            • local fs tests cover the code paths in this change
            • all core tests pass when configured with s3
            • manually tried larger local filesystem tables (tpch) with multiple
              partitions and observed the same scan ranges.
            • TODO: adls testing

            Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5
            Reviewed-on: http://gerrit.cloudera.org:8080/8523
            Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com>
            Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
            Reviewed-on: http://gerrit.cloudera.org:8080/10692
            Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            jira-bot ASF subversion and git services added a comment - Commit 11554a17c75b242767d5a50d66bc2874aa545c77 in impala's branch refs/heads/2.x from vercego [ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=11554a1 ] IMPALA-5931 : Generates scan ranges in planner for s3/adls Currently, for filesystems that do not include physical block information (e.g., block replica locations, caching), synthetic blocks are generated and stored in the catalog when metadata is loaded. Example file systems for which this is done includes S3, ADLS, and local fs. This change avoids generating these blocks when metadata is loaded. Instead, scan ranges are directly generated from such files by the backend coordinator. Previously, all scan ranges were produced by the planner in HDFSScanNode in the frontend. Now, those files without block information are sent to the coordinator represented by a split specification that determines how the coordinator will create scan ranges to send to executors. This change reduces the space needed in the catalog and reduces the scan range data structures that are passed from the frontend to the backend when planning and coordinating a query. In addition a bug is avoided where non-splittable files were being split anyways to support the query parameter that places a limit on scan ranges. Testing: added backend scheduler tests mixed-filesystems test covers tables/queries with multiple fs's. local fs tests cover the code paths in this change all core tests pass when configured with s3 manually tried larger local filesystem tables (tpch) with multiple partitions and observed the same scan ranges. TODO: adls testing Change-Id: I326065adbb2f7e632814113aae85cb51ca4779a5 Reviewed-on: http://gerrit.cloudera.org:8080/8523 Reviewed-by: Vuk Ercegovac <vercegovac@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Reviewed-on: http://gerrit.cloudera.org:8080/10692 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            Commit feed25084a999fe0a4e7b58b5264fce5829c43e7 in impala's branch refs/heads/master from stakiar
            [ https://gitbox.apache.org/repos/asf?p=impala.git;h=feed250 ]

            IMPALA-8944: Update and re-enable S3PlannerTest

            Addresses several test infra issues that were preventing the
            S3PlannerTest from running successfully. Disables a few tests that are
            no longer working, and removes some planner checks that are no longer
            applicable when running on S3. Specifically, this patch removes the
            checks in PlannerTestBase#checkScanRangeLocations when running against
            S3, because the planner no longer generates scan ranges; generation is
            deferred to the scheduler (IMPALA-5931).

            Replaces the old logic of specifying S3-specific fe/ tests with a
            combination of JUnit Categories and Maven Profiles. The previous method
            was broken and assumed that all S3-specific fe/ tests started with S3*.
            The new approach removes that restriction and only requires S3-specific
            JUnit tests to be tagged with the Java annotation
            '@Category(S3Tests.class)' (entire classes or individual tests can be
            tagged with the annotation).

            Testing:

            • Ran fe/ tests with TARGET_FILESYSTEM=s3

            Change-Id: I1690b6c5346376c1111fd4845c72062cc237e0f9
            Reviewed-on: http://gerrit.cloudera.org:8080/14248
            Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>
            Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            jira-bot ASF subversion and git services added a comment - Commit feed25084a999fe0a4e7b58b5264fce5829c43e7 in impala's branch refs/heads/master from stakiar [ https://gitbox.apache.org/repos/asf?p=impala.git;h=feed250 ] IMPALA-8944 : Update and re-enable S3PlannerTest Addresses several test infra issues that were preventing the S3PlannerTest from running successfully. Disables a few tests that are no longer working, and removes some planner checks that are no longer applicable when running on S3. Specifically, this patch removes the checks in PlannerTestBase#checkScanRangeLocations when running against S3, because the planner no longer generates scan ranges; generation is deferred to the scheduler ( IMPALA-5931 ). Replaces the old logic of specifying S3-specific fe/ tests with a combination of JUnit Categories and Maven Profiles. The previous method was broken and assumed that all S3-specific fe/ tests started with S3*. The new approach removes that restriction and only requires S3-specific JUnit tests to be tagged with the Java annotation '@Category(S3Tests.class)' (entire classes or individual tests can be tagged with the annotation). Testing: Ran fe/ tests with TARGET_FILESYSTEM=s3 Change-Id: I1690b6c5346376c1111fd4845c72062cc237e0f9 Reviewed-on: http://gerrit.cloudera.org:8080/14248 Reviewed-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com> Tested-by: Impala Public Jenkins <impala-public-jenkins@cloudera.com>

            People

              vukercegovac Vuk Ercegovac
              dhecht Daniel Hecht
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: