Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Table API & SQL
    • Labels:
      None

      Description

      This issue aims to support register one or more ExternalCatalog (which is referred in https://issues.apache.org/jira/browse/FLINK-5568) to TableEnvironment. After registration, SQL and TableAPI queries could access to tables in the external catalogs without register those tables one by one to TableEnvironment beforehand.

      We plan to add two APIs in TableEnvironment:
      1. register externalCatalog

      def registerExternalCatalog(name: String, externalCatalog: ExternalCatalog): Unit
      

      2. scan a table from registered catalog and returns the resulting Table, the API is very useful in TableAPI queries.

      def scan(catalogName: String, tableIdentifier: TableIdentifier): Table
      

        Activity

        Hide
        wheat9 Haohui Mai added a comment -

        We are interested in this feature as well. Just curious about the current progress – are there anything that I can help?

        Show
        wheat9 Haohui Mai added a comment - We are interested in this feature as well. Just curious about the current progress – are there anything that I can help?
        Hide
        ykt836 Kurt Young added a comment -

        Hi Haohui Mai, thanks for the offering, i think we are implementing this right now. Will let you know if we need some help.

        Show
        ykt836 Kurt Young added a comment - Hi Haohui Mai , thanks for the offering, i think we are implementing this right now. Will let you know if we need some help.
        Hide
        fhueske Fabian Hueske added a comment -

        Hi Haohui Mai, we got a PR for this issue: #3409.
        You can have a look, help reviewing and check if the solution addresses your needs. Would be great to have your feedback on this. Thanks.

        Show
        fhueske Fabian Hueske added a comment - Hi Haohui Mai , we got a PR for this issue: #3409 . You can have a look, help reviewing and check if the solution addresses your needs. Would be great to have your feedback on this. Thanks.
        Hide
        wheat9 Haohui Mai added a comment -

        Thanks Fabian Hueske for the pointer. Just took a very quick skim – if I understand correctly, #3409 allows other users to manipulate the catalogs within Flink.

        This is different from having a Metastore-like API that gives the statistics of the tables. Personally I think it might make sense to look into both issues holistically and come up with a design that works well with both the Metastore and HCatalog use cases.

        Show
        wheat9 Haohui Mai added a comment - Thanks Fabian Hueske for the pointer. Just took a very quick skim – if I understand correctly, #3409 allows other users to manipulate the catalogs within Flink. This is different from having a Metastore-like API that gives the statistics of the tables. Personally I think it might make sense to look into both issues holistically and come up with a design that works well with both the Metastore and HCatalog use cases.
        Hide
        fhueske Fabian Hueske added a comment -

        Hi Haohui Mai, the issue is actually about adding an interface to connect to external catalogs. HCatalog or Metastore could be one of these catalogs.

        If you look at the PR #3406 (#3409 builds on the commits of #3406), you'll find an interface ExternalCatalog that would need to be implemented for HCatalog and act as a bridge between HCatalog and the Flink's catalog in Calcite. Do you think the ExternalCatalog interface is sufficient to address your use case?

        Show
        fhueske Fabian Hueske added a comment - Hi Haohui Mai , the issue is actually about adding an interface to connect to external catalogs. HCatalog or Metastore could be one of these catalogs. If you look at the PR #3406 (#3409 builds on the commits of #3406), you'll find an interface ExternalCatalog that would need to be implemented for HCatalog and act as a bridge between HCatalog and the Flink's catalog in Calcite. Do you think the ExternalCatalog interface is sufficient to address your use case?
        Hide
        wheat9 Haohui Mai added a comment -

        Thanks for the ping, Fabian Hueske. Please allow me for a day or two to go through the PR again.

        Show
        wheat9 Haohui Mai added a comment - Thanks for the ping, Fabian Hueske . Please allow me for a day or two to go through the PR again.
        Hide
        wheat9 Haohui Mai added a comment -

        I think the APIs of the PR looks good to me overall.

        One question – Does the PR need additional fixes on to make functions like isRegistered and getRowType be aware of databases?

        Show
        wheat9 Haohui Mai added a comment - I think the APIs of the PR looks good to me overall. One question – Does the PR need additional fixes on to make functions like isRegistered and getRowType be aware of databases?
        Hide
        fhueske Fabian Hueske added a comment -

        Thanks for the feedback Haohui Mai.

        I think we are fine at the moment, but could update the isRegistered() and getRowType() methods to also take an Array[String] parameter.
        At the moment, they isRegistered() is only called from methods that manage the top-level tables (not those in the catalog). getRowType() was only called from scan() and is now unused so it could be removed.

        Show
        fhueske Fabian Hueske added a comment - Thanks for the feedback Haohui Mai . I think we are fine at the moment, but could update the isRegistered() and getRowType() methods to also take an Array [String] parameter. At the moment, they isRegistered() is only called from methods that manage the top-level tables (not those in the catalog). getRowType() was only called from scan() and is now unused so it could be removed.
        Hide
        fhueske Fabian Hueske added a comment -

        Implemented with 135a57c4bb37eaa9cb85faaff1cc694f9448fabd

        Show
        fhueske Fabian Hueske added a comment - Implemented with 135a57c4bb37eaa9cb85faaff1cc694f9448fabd

          People

          • Assignee:
            jinyu.zj jingzhang
            Reporter:
            ykt836 Kurt Young
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development