Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21596

Audit the places calling HDFSMetadataLog.get

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.1.1
    • 2.2.1, 2.3.0
    • Structured Streaming
    • None

    Description

      When I was investigating a flaky test, I realized that many places don't check the return value of `HDFSMetadataLog.get(batchId: Long): Option[T]`. When a batch is supposed to be there, the caller just ignores None rather than throwing an error. If some bug causes a query doesn't generate a batch metadata file, this behavior will hide it and allow the query continuing to run.

      Attachments

        Issue Links

          is duplicated by

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21760 Structured streaming terminates with Exception

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #18799 [Github] Pull Request #18799 (zsxwing)

          Activity

            People

              zsxwing Shixiong Zhu
              zsxwing Shixiong Zhu
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: