Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21525

ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.2.0
    • 2.4.0
    • DStreams
    • None

    Description

      AddBlock returns an error code related to whether writing the block to the WAL was successful or not. In cases where a WAL may be unavailable temporarily, the write would fail but it seems like we are not using the return code (see here).

      For example, when using the Flume Receiver, we should be sending a n'ack back to Flume if the block wasn't written to the WAL. I haven't gone through the full code path yet but at least from looking at the ReceiverSupervisorImpl, it doesn't seem like that return code is being used.

      Attachments

        Activity

          People

            vanzin Marcelo Masiero Vanzin
            mgrover Mark Grover
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: