Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-21525

ReceiverSupervisorImpl seems to ignore the error code when writing to the WAL

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.4.0
    • Component/s: DStreams
    • Labels:
      None

      Description

      AddBlock returns an error code related to whether writing the block to the WAL was successful or not. In cases where a WAL may be unavailable temporarily, the write would fail but it seems like we are not using the return code (see here).

      For example, when using the Flume Receiver, we should be sending a n'ack back to Flume if the block wasn't written to the WAL. I haven't gone through the full code path yet but at least from looking at the ReceiverSupervisorImpl, it doesn't seem like that return code is being used.

        Attachments

          Activity

            People

            • Assignee:
              vanzin Marcelo Vanzin
              Reporter:
              mgrover Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: