Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-24052

Flink SQL reads S3 bucket data.

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Not a Priority
    • Resolution: Not A Problem
    • None
    • None
    • Table SQL / Ecosystem
    • None

    Description

      I wanna use Flink SQL reads S3 bucket data. But now I found it ONLY supports absolute path, which means I can not read all content in the bucket.
      My SQL statements write as below:

      CREATE TABLE file_data (
          a BIGINT, b STRING, c STRING, d DOUBLE, e BOOLEAN, f DATE, g STRING,h STRING,
          i STRING, j STRING, k STRING, l STRING, m STRING, n STRING, o STRING, p FLOAT
      ) WITH (
          'connector' = 'filesystem',
          'path' = 's3a://my-bucket',
          'format' = 'parquet'
      );
      
      SELECT COUNT(*) FROM file_data;
      

      The exception info:

      Caused by: java.lang.IllegalArgumentException: path must be absolute
      	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88) ~[flink-s3-fs-hadoop-1.13.1.jar:1.13.1]
      	at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:68) ~[flink-s3-fs-hadoop-1.13.1.jar:1.13.1]
      	at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:60) ~[flink-s3-fs-hadoop-1.13.1.jar:1.13.1]
      	at org.apache.hadoop.fs.s3a.s3guard.PathMetadata.<init>(PathMetadata.java:56) ~[flink-s3-fs-hadoop-1.13.1.jar:1.13.1]
      	at org.apache.hadoop.fs.s3a.s3guard.S3Guard.putAndReturn(S3Guard.java:149) ~[flink-s3-fs-hadoop-1.13.1.jar:1.13.1]
      

      Is there any solution to meet my requirement ?

      Attachments

        Activity

          People

            Unassigned Unassigned
            ZhangChaoming Moses
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: