Uploaded image for project: 'Phoenix'
  1. Phoenix
  2. PHOENIX-5442 Change Data Capture
  3. PHOENIX-6227

Option for DDL changes to export to external schema repository

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.17.0, 5.2.0
    • None
    • None
    • Hide
      External_Schema_Id is now added to SYSTEM.CATALOG. When CHANGE_DETECTION_ENABLED = true, CREATE and ALTER statements to a table or view cause a call to an external schema registry is made to save a string representation of the schema to be made. The schema registry returns a schema id which is saved in the new SYSTEM.CATALOG field.

      When change detection is enabled, the WAL is now annotated with the external schema id, rather than the tuple of tenant id / schema name / logical table name / last DDL timestamp it was previously.

      The implementation to generate the string representation can be set via the Configuration property org.apache.phoenix.export.schemawriter.impl . By default it is a String representation of the PTable protobuf.

      The implementation of the call to the schema repository can be set via the Configuration property org.apache.phoenix.export.schemaregistry.impl . By default it is an in-memory data structure unsuitable for production use. In production it is intended to be used with a standalone schema registry service.
      Show
      External_Schema_Id is now added to SYSTEM.CATALOG. When CHANGE_DETECTION_ENABLED = true, CREATE and ALTER statements to a table or view cause a call to an external schema registry is made to save a string representation of the schema to be made. The schema registry returns a schema id which is saved in the new SYSTEM.CATALOG field. When change detection is enabled, the WAL is now annotated with the external schema id, rather than the tuple of tenant id / schema name / logical table name / last DDL timestamp it was previously. The implementation to generate the string representation can be set via the Configuration property org.apache.phoenix.export.schemawriter.impl . By default it is a String representation of the PTable protobuf. The implementation of the call to the schema repository can be set via the Configuration property org.apache.phoenix.export.schemaregistry.impl . By default it is an in-memory data structure unsuitable for production use. In production it is intended to be used with a standalone schema registry service.

    Description

      When a user creates or drops a table or view, or adds/removes a column from one, there should be the option for Phoenix to notify an external schema repository. This should be a configurable plugin so that core Phoenix is not coupled to any particular repository implementation.

      This will also store a schema id generated by the external schema registry in a new field in System.Catalog so that a future JIRA can switch to using the schema id in change detection WAL annotations. Because of this this JIRA will not be able to be backported to 5.1.x or 4.16.x

      Attachments

        Issue Links

          contains

          Sub-task - The sub-task of the issue PHOENIX-6584 Annotate HBase WALs with Schema Id

          • Major - Major loss of function.
          • Resolved
          depends upon

          New Feature - A new feature of the product, which has yet to be developed. PHOENIX-6457 Optionally store schema version string in SYSTEM.CATALOG

          • Major - Major loss of function.
          • Resolved
          is depended upon by

          Sub-task - The sub-task of the issue PHOENIX-6584 Annotate HBase WALs with Schema Id

          • Major - Major loss of function.
          • Resolved
          links to

          Web Link GitHub Pull Request #1341

          Web Link GitHub Pull Request #1345

          Activity

            People

              gjacoby Geoffrey Jacoby
              gjacoby Geoffrey Jacoby
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: