Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-45511

SPIP: State Data Source - Reader

    XMLWordPrintableJSON

Details

    Description

      State Store has been a black box from the introduction of the stateful operator. It has been the “internal” data to the streaming query, and Spark does not expose the data outside of the streaming query. There is no feature/tool for users to read and modify the content of state stores.

      Specific to the ability to read the state, the lack of feature brings up various limitations like following:

      • Users are unable to see the content in the state store, leading to inability to debug.
      • Users have to perform some indirect approach on verifying the content of the state store in unit tests. The only option they can take is relying on the output of the query.

      Given that, we propose to introduce a feature which enables users to read the state from the outside of the streaming query.

      SPIP: https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing

       

       

      Attachments

        Issue Links

          is depended upon by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-45671 Implement an option similar to corrupt record column in State Data Source Reader

          • Major - Major loss of function.
          • Open

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-45672 Provide a unified user-facing schema for state format versions in state data source - reader

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. SPARK-45889 Implement push-down filter with partition ID and grouping key (if possible) for state data source reader

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. SPARK-45890 Implement limit push down for state data source reader

          • Major - Major loss of function.
          • Open

          Task - A task that needs to be done. SPARK-45833 Document state data source - reader

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved

          Task - A task that needs to be done. SPARK-45888 Apply error class framework to state data source & state metadata data source

          • Blocker - Blocks development and/or testing work, production could not run
          • Resolved
          links to

          Web Link GitHub Pull Request #43425

          Web Link GitHub Pull Request #43831

          Web Link GitHub Pull Request #43945

          Activity

            People

              kabhwan Jungtaek Lim
              kabhwan Jungtaek Lim
              Jungtaek Lim Jungtaek Lim
              Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: