Uploaded image for project: 'Phoenix Tephra'
  1. Phoenix Tephra
  2. TEPHRA-39

Support Nested Transactions

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • api, core, manager

    Description

      Tephra transactions are used not only for real-time processing, but also in batch.

      There are cases when during processing data in batches, the single unit of processing (a job) consists of many subunits (tasks), which have its own lifecycle. Such subunits can fail, can run in parallel and attempted multiple times to recover from failure or for performance optimization reasons. One example of such batch data processing is MapReduce: the job may consist of multiple Map and Reduce tasks that can be re-attempted individually after failure or attempted multiple times during speculative execution.

      At the moment Tephra supports "long-running" transactions that can be used to cover a batch job. Using it guarantees that no incomplete changes of a job that is not finished properly (failed, killed, cancelled) are visible to others.

      At the same time this does not guarantee that changes made by multiple attempts of the same subtask cannot break the data consistency, unless data operations in subtasks are idempotent because same data operations may be done multiple times by different subtask attempts.

      Having nested transactions would allow relaxing this constraint by:

      • running each individual subtask in separate "nested transaction"
      • make changes of subtasks made in nested transactions visible only if the parent transaction succeeds

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            ghelmling Gary Helmling
            alexb Alex Baranau

            Dates

              Created:
              Updated:

              Slack

                Issue deployment