Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-9180

IgniteSparkSession Should Copy State on cloneSession()

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.6
    • Fix Version/s: 2.7
    • Component/s: spark
    • Labels:
      None

      Description

      The IgniteSparkSession class extends SparkSession and overrides the cloneSession() method. The contract for cloneSession() explicitly states that it should clone all state (ie. the sharedState and sessionState fields), however the IgniteSparkSession implementation doesn't clone its state fields.

      This has the side-effect of breaking stateful operations for anything which uses cloneSession(), for example a Spark streaming job will not be able to use cached data across streaming microbatches, which is a significant issue for such applications.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stuartmacd Stuart Macdonald
                Reporter:
                stuartmacd Stuart Macdonald
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: