Uploaded image for project: 'Beam'
  1. Beam
  2. BEAM-6874

HCatalogTableProvider supports filter pushdown

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: P3
    • Resolution: Unresolved
    • Affects Version/s: 2.11.0
    • Fix Version/s: None
    • Component/s: dsl-sql, io-java-hcatalog
    • Labels:
      None

      Description

      Hi,

      I'm using HCatalogTableProvider while doing SqlTransform.query. The query is something like "select * from `hive`.`table_name` limit 10". Despite of the limit clause, the data source still reads much more rows (the data of Hive table are files on S3), even more than the number of rows in one file (or partition).

       

      Some more details:

      1. It is running on Flink.
      2. I actually implemented my own HiveTableProvider because HCatalogBeamSchema only supports primitive types. However, the table provider works when I query a small table with ~1k rows.

        Attachments

        1. limit.png
          444 kB
          Near

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Near Near
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: