Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2271 Follow-up items for timeline-server-based marker files
  3. HUDI-2865

Enable timeline-server-based markers for Spark structured streaming

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • spark
    • None

    Description

      For Spark structured streaming, after the end of first micro-batch, the write client is closed and hence triggers closure of timeline service. But subsequent micro-batches do succeed though.

      Currently, we explicitly override the marker type to be DIRECT in Spark structured streaming.  We need to revisit the behavior of timeline service before we can enable timeline-server-based markers for Spark structured streaming.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            guoyihua Ethan Guo

            Dates

              Created:
              Updated:

              Slack

                Issue deployment