Uploaded image for project: 'Ignite'
  1. Ignite
  2. IGNITE-7092

Deprecate embedded mode in IgniteRDD

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.3
    • Fix Version/s: 2.4
    • Component/s: spark
    • Labels:
      None

      Description

      Currently we claim to support IgniteRDD in two modes: standalone and embedded. Standalone means there is a separately running Ignite cluster, and Spark start client node(s) to interact with it. In embedded node everything runs within Spark, including Ignite server nodes that are started embedded into Spark executors.

      The latter case doesn't really work, mainly because the lifecycle of Spark executors is not very predictable - Spark can start and stop them while application is running. In case Ignite cluster is used to store data (which is usually the case), this causes unnecessary rebalancing or even unexpected data loss.

      I propose to deprecate and eventually discontinue the embedded mode. Luckily, standalone mode is the default one, so we can simply print out a clear warning if one switches to embedded mode, and also mention this in the docs.

        Attachments

          Activity

            People

            • Assignee:
              vkulichenko Valentin Kulichenko
              Reporter:
              vkulichenko Valentin Kulichenko
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: