Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-2271 Follow-up items for timeline-server-based marker files
  3. HUDI-2865

Enable timeline-server-based markers for Spark structured streaming

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • None
    • spark
    • None

    Description

      For Spark structured streaming, after the end of first micro-batch, the write client is closed and hence triggers closure of timeline service. But subsequent micro-batches do succeed though.

      Currently, we explicitly override the marker type to be DIRECT in Spark structured streaming.  We need to revisit the behavior of timeline service before we can enable timeline-server-based markers for Spark structured streaming.

      Attachments

        Activity

          People

            Unassigned Unassigned
            guoyihua Ethan Guo
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: