Flume
  1. Flume
  2. FLUME-1378

Sink falls into infinite lifecycle with EventDeliveryException

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Channel, Sinks+Sources
    • Labels:
      None

      Description

      I tested the Flume and found a critical issue. HDFSEventSink with 'txnEventMax' parameter > MemoryChannel 'transactionCapacity' parameter, Sink falls into infinite lifecycle and all Events has been hold in MemoryChannel with error message:

      "Take list for MemoryTransaction, capacity <n> full, consider committing more frequently,increasing capacity, or increasing thread count"

        Issue Links

          Activity

          Hide
          Mike Percy added a comment -

          Not sure this is a critical issue. Just a misconfiguration.

          Show
          Mike Percy added a comment - Not sure this is a critical issue. Just a misconfiguration.
          Hide
          Denny Ye added a comment -

          OK, it can be fixed with parameter checking. The impact of this bug is terrible, infinite lifecycle of Sink, and huge heap with full GC in 90% time, causes the 'GC overhead limit exceeded'

          Show
          Denny Ye added a comment - OK, it can be fixed with parameter checking. The impact of this bug is terrible, infinite lifecycle of Sink, and huge heap with full GC in 90% time, causes the 'GC overhead limit exceeded'
          Hide
          Denny Ye added a comment -

          I am thinking about the point or method that I should take place the validation between channel and sink. Most possibility in my thought is validation progress in FlumeConfiguration, unfortunately, they used different parameter name in each Sink.

          Could somebody give me your advice?

          Show
          Denny Ye added a comment - I am thinking about the point or method that I should take place the validation between channel and sink. Most possibility in my thought is validation progress in FlumeConfiguration, unfortunately, they used different parameter name in each Sink. Could somebody give me your advice?

            People

            • Assignee:
              Denny Ye
              Reporter:
              Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development