Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-19595

Flink SQL support S3 select

    XMLWordPrintableJSON

    Details

      Description

      Summarize

      Flink is based on S3AInputStream.java to select datas stored in Tencent COS, it will call the getObject function of AmazonS3Client.java. 

      Now, Tencent COS  have already support to pushdown the CSV and Parquert file format.

      In these cases, using getObject to select datas will wastes a lots of bandwidth.

      So, I think Flink SQL should support S3 Select, to reduce the waste of bandwidth.

       

      Design

      1. In HiveMapredSplitReader.java , we used int[] selectedFields to construct S3 SELECT SQL. And we created a new Class named S3SelectCsvReader which used AmazonS3Client.selectObjectContent function to readLine CSV File.

       

       

      2.  Flink Demo Table:

      1) Table schema

      Flink SQL> desc cos.test_s3a;
      root

      – name: STRING (col1)
      – age: INT           (col2)
      – dt: STRING      (col3,it's a partition column)

       

      2) Conversion relationship (FLINK SQL Convert To S3 SELECT SQL)

      FlinkSQL                                                                                              S3 SELECT SQL

      select name from cos.test_s3a;                                             =>       SELECT s._1, null FROM S3Object s

      select age from cos.test_s3a;                                                 =>      SELECT null, s._2 FROM S3Object s

      select dt, name, age from cos.test_s3a;                                =>       SELECT s._1, s._2 FROM S3Object s

      select dt from cos.test_s3a;                                                    =>      SELECT null, null FROM S3Object s

      select * from cos.test_s3a;                                                      =>      SELECT s._1, s._2 FROM S3Object s

      select name from cos.test_s3a where dt='2020-07-15';      =>      SELECT s._1, null FROM S3Object s

       

      3) Patch Commit

      https://github.com/Coderlxl/flink/commit/b211f4830a7301bf9283a6d37209000b176913ad

        Attachments

        1. image-2020-11-02-18-08-11-461.png
          252 kB
          liuxiaolong
        2. image-2020-11-02-18-18-14-961.png
          57 kB
          liuxiaolong

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              lxl liuxiaolong
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: