Uploaded image for project: 'Apache Hudi'
  1. Apache Hudi
  2. HUDI-1765

disabling auto commit does not work with spark ds

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • None
    • 0.11.0
    • spark

    Description

      commitAndPerformPostOperations in HoodieSparkSqlWriter commits all operations w/o checking for autoCommitProp. So, this prop is not being honored at spark ds layer. 

       

      Added below test to TestCOWDataSource and it fails. 

      https://gist.github.com/nsivabalan/479320351bec95ba3e0c0dfa5abb2b06

       

      Attachments

        Activity

          vinoth Vinoth Chandar added a comment -

          Whats the use-case for disabling autocommit with the spark ds? does the user want to commit by themselves, somehow figuring out the instanttime of the commit?

          vinoth Vinoth Chandar added a comment - Whats the use-case for disabling autocommit with the spark ds? does the user want to commit by themselves, somehow figuring out the instanttime of the commit?

          oh, good point. I will check w/ customer. my bad. didn't think about it. 

          shivnarayan sivabalan narayanan added a comment - oh, good point. I will check w/ customer. my bad. didn't think about it. 

          It may not make sense to expose this config via spark-ds writer. Hence closing it out. 

          shivnarayan sivabalan narayanan added a comment - It may not make sense to expose this config via spark-ds writer. Hence closing it out. 

          People

            shivnarayan sivabalan narayanan
            shivnarayan sivabalan narayanan
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: