Flume
  1. Flume
  2. FLUME-883

Flume E2E sink could send incorrect ACKs if there are HDFS file close errors

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v0.9.4
    • Fix Version/s: v0.9.5
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      The E2E collector sink saves the batch tags as the batches are passed to the downstream sinks. The ACKs are flushed when the roller close the file. Currently for the HDFS sink, the close is the only operation that guarantees that data is safely stored. Hence the acks are sent on close. If for some reason, the writes fail then we don't send the acks assuming the data is lost. The E2E mechanism then resends the data.
      The problem is that if the close fails then we don't clear the accumulated acks for that current rolltag. Hence its possible that the next successful roll could send those acks and hence the batch will not be resent.

      1. Flume-883.patch.1
        7 kB
        Prasad Mujumdar

        Activity

        Prasad Mujumdar created issue -
        Prasad Mujumdar made changes -
        Field Original Value New Value
        Attachment Flume-883.patch.1 [ 12507636 ]
        Prasad Mujumdar made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Prasad Mujumdar
            Reporter:
            Prasad Mujumdar
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development