Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.12.0
    • Component/s: S3
    • Labels:
      None

      Description

      AWS S3 provides bulk listing API. It takes the common prefix of all input paths as a parameter and returns all the objects whose prefixes start with the common prefix in blocks of 1000.

      If we will use AmazonS3Client for listing S3 files instead of using S3A, this will improve performance. To prove this idea, I adopted PrestoFileSystem instead of S3AFileSystem. When pruning partition filters, PrestoFileSystem was faster much more than S3AFileSystem.

      Here is my benchmark results for the following queries:

      1 partition : select count(*) from lineitem where l_shipdate = '1992-01-02';
      30 partitions: select count(*) from lineitem  where l_shipdate > '1992-01-01' and l_shipdate < '1992-02-01';
      90 partitions: select count(*) from lineitem  where l_shipdate >= '1992-01-01' and l_shipdate < '1992-04-01';
      151 partitions: select count(*) from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-06-01';
      
      (#) of partitions PrestoFileSystem(ms) S3AFileSystem(ms)
      1 677 800
      30 2753 6977
      90 6825 13772
      151 13834 25701

      For the reference, I used tpc-h 1g dataset and set l_shipdate column of lineitem table to partition column.

      I think there are ways to resolve this as following:

      • Borrow PrestoFileSystem and related codes from Presto
      • Implement necessary codes to S3TableSpace by referencing Presto

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user blrunner opened a pull request:

        https://github.com/apache/tajo/pull/932

        TAJO-2030: Use list S3 files using AmazonS3Client instead of using S3A.

        The code for S3 bulk listing is fully implemented in ``TajoS3FileSystem``. Honestly, my code is heavily based on ``PrestoS3FileSystem``. And ``TajoS3FileSystem`` extends ``S3AFileSystem`` because ``PrestoS3FileSystem`` doesn't support some methods for file writing, for example, ``FileSystem::mkdir``.

        Here is my benchmark results as follows.

        1. Configuration
        • EC2 instance type : c3.xlarge
        • Tajo version : 0.12.0-SNAPSHOT
        • Cluster: 1 master, 1 worker
        • partitions had been generated by Hive
        1. Queries

        ```
        1 partition: select count from lineitem where l_shipdate = '1992-01-02';
        30 partitions: select count from lineitem where l_shipdate > '1992-01-01' and l_shipdate < '1992-02-01';
        90 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-04-01';
        151 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-06-01';
        334 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-12-01';
        ```

        1. Results : Partition Pruning

        #of partitions | S3AFileSystem | TajoS3FileSystem | Improvement
        ------------------|--------------------|------------------------|------------------
        1 | 1088 ms | 607 ms | 1.79x
        30 | 5421 ms | 3414 ms | 1.58x
        90 | 15776 ms | 7927 ms | 1.99x
        151 | 24060 ms | 14912 ms | 1.61x
        334 | 45397 ms | 32247 ms | 1.40x

        1. Results : Query Finished time

        #of partitions | S3AFileSystem | TajoS3FileSystem | Improvement
        ------------------|--------------------|------------------------|------------------
        1 | 3.99 sec | 2.726 sec | 1.46x
        30 | 15.447 sec | 12.416 sec | 1.24
        90 | 40.153 sec | 31.593 sec | 1.27x
        151 | 66.038 sec | 44.604 sec | 1.48x
        334 | 137.137 sec | 90.419 sec | 1.51x

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/blrunner/tajo TAJO-2030

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/932.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #932


        commit 1a90ad1688e763b7a1c52bdf7600c206094a10b6
        Author: JaeHwa Jung <blrunner@apache.org>
        Date: 2016-01-07T05:10:07Z

        TAJO-2030: Use list S3 files using AmazonS3Client instead of using S3A.


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user blrunner opened a pull request: https://github.com/apache/tajo/pull/932 TAJO-2030 : Use list S3 files using AmazonS3Client instead of using S3A. The code for S3 bulk listing is fully implemented in ``TajoS3FileSystem``. Honestly, my code is heavily based on ``PrestoS3FileSystem``. And ``TajoS3FileSystem`` extends ``S3AFileSystem`` because ``PrestoS3FileSystem`` doesn't support some methods for file writing, for example, ``FileSystem::mkdir``. Here is my benchmark results as follows. Configuration EC2 instance type : c3.xlarge Tajo version : 0.12.0-SNAPSHOT Cluster: 1 master, 1 worker partitions had been generated by Hive Queries ``` 1 partition: select count from lineitem where l_shipdate = '1992-01-02'; 30 partitions: select count from lineitem where l_shipdate > '1992-01-01' and l_shipdate < '1992-02-01'; 90 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-04-01'; 151 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-06-01'; 334 partitions: select count from lineitem where l_shipdate >= '1992-01-01' and l_shipdate < '1992-12-01'; ``` Results : Partition Pruning #of partitions | S3AFileSystem | TajoS3FileSystem | Improvement ------------------ | -------------------- | ------------------------ | ------------------ 1 | 1088 ms | 607 ms | 1.79x 30 | 5421 ms | 3414 ms | 1.58x 90 | 15776 ms | 7927 ms | 1.99x 151 | 24060 ms | 14912 ms | 1.61x 334 | 45397 ms | 32247 ms | 1.40x Results : Query Finished time #of partitions | S3AFileSystem | TajoS3FileSystem | Improvement ------------------ | -------------------- | ------------------------ | ------------------ 1 | 3.99 sec | 2.726 sec | 1.46x 30 | 15.447 sec | 12.416 sec | 1.24 90 | 40.153 sec | 31.593 sec | 1.27x 151 | 66.038 sec | 44.604 sec | 1.48x 334 | 137.137 sec | 90.419 sec | 1.51x You can merge this pull request into a Git repository by running: $ git pull https://github.com/blrunner/tajo TAJO-2030 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/932.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #932 commit 1a90ad1688e763b7a1c52bdf7600c206094a10b6 Author: JaeHwa Jung <blrunner@apache.org> Date: 2016-01-07T05:10:07Z TAJO-2030 : Use list S3 files using AmazonS3Client instead of using S3A.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner commented on the pull request:

        https://github.com/apache/tajo/pull/932#issuecomment-183187420

        This patch depends on hadoop-aws. I'm going to implement it afresh after resolving https://github.com/apache/tajo/pull/953.

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/932#issuecomment-183187420 This patch depends on hadoop-aws. I'm going to implement it afresh after resolving https://github.com/apache/tajo/pull/953 .
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user blrunner closed the pull request at:

        https://github.com/apache/tajo/pull/932

        Show
        githubbot ASF GitHub Bot added a comment - Github user blrunner closed the pull request at: https://github.com/apache/tajo/pull/932
        Hide
        stevel@apache.org Steve Loughran added a comment -

        probably a bit late for you now, but HADOOP-13208 does the speedup you want in S3a. You know, if you do think there is some opportunity for speedup in that hadoop s3a code, happy to take it on provided it comes with decent set of integration tests.

        Show
        stevel@apache.org Steve Loughran added a comment - probably a bit late for you now, but HADOOP-13208 does the speedup you want in S3a. You know, if you do think there is some opportunity for speedup in that hadoop s3a code, happy to take it on provided it comes with decent set of integration tests.

          People

          • Assignee:
            blrunner Jaehwa Jung
            Reporter:
            blrunner Jaehwa Jung
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:

              Development