Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1378

Sink falls into infinite lifecycle with EventDeliveryException

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Channel, Sinks+Sources
    • Labels:
      None

      Description

      I tested the Flume and found a critical issue. HDFSEventSink with 'txnEventMax' parameter > MemoryChannel 'transactionCapacity' parameter, Sink falls into infinite lifecycle and all Events has been hold in MemoryChannel with error message:

      "Take list for MemoryTransaction, capacity <n> full, consider committing more frequently,increasing capacity, or increasing thread count"

        Issue Links

          Activity

          Hide
          mpercy Mike Percy added a comment -

          Not sure this is a critical issue. Just a misconfiguration.

          Show
          mpercy Mike Percy added a comment - Not sure this is a critical issue. Just a misconfiguration.
          Hide
          dennyy Denny Ye added a comment -

          OK, it can be fixed with parameter checking. The impact of this bug is terrible, infinite lifecycle of Sink, and huge heap with full GC in 90% time, causes the 'GC overhead limit exceeded'

          Show
          dennyy Denny Ye added a comment - OK, it can be fixed with parameter checking. The impact of this bug is terrible, infinite lifecycle of Sink, and huge heap with full GC in 90% time, causes the 'GC overhead limit exceeded'
          Hide
          dennyy Denny Ye added a comment -

          I am thinking about the point or method that I should take place the validation between channel and sink. Most possibility in my thought is validation progress in FlumeConfiguration, unfortunately, they used different parameter name in each Sink.

          Could somebody give me your advice?

          Show
          dennyy Denny Ye added a comment - I am thinking about the point or method that I should take place the validation between channel and sink. Most possibility in my thought is validation progress in FlumeConfiguration, unfortunately, they used different parameter name in each Sink. Could somebody give me your advice?

            People

            • Assignee:
              dennyy Denny Ye
              Reporter:
              dennyy Denny Ye
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development