Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-3177

Expose snapshotTimestampMicros to Spark Read Options

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.13.0
    • spark

    Description

      If a spark application needs to read from the same table multiple times and that table has new records that may come in during the life of the application, you may get inconsistent scan results unless you persist the DataFrame. I'd like to expose snapshotTimestampMicros to the spark read options so I can set a timestamp before the first scan and use that for READ_AT_SNAPSHOT to keep all scans on the same table consistent throughout the run of the application. 

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            kmccarthy Kevin J McCarthy
            kmccarthy Kevin J McCarthy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment