Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-4155

NotificationHookConsumer: Large Compressed Message Processing Problem

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.0
    • Fix Version/s: None
    • Component/s: atlas-core
    • Labels:

      Description

      Background

      Notification messages can be large in size. To get over Kafka's limitation on message size, Atlas has compressed and split messages. If message size goes beyond stipulated threshold, the message is compressed. If compressed message goes beyond the size, it is split into multiple messages.

      Situation

      Consider a message that is so large that uncompressing it takes longer than Kafka's timeout for message. This causes the problem where the large message offset is not committed in time and that causes Kafka to present the same message again.

      Message Description:
      Number of splits: 8
      Compressed message size: 7,452,640
      Uncompressed message size: 520,803,946
      Time taken to uncompress and stitch messages: > 90 seconds

      Sequence:
      2021-02-10 14:57:24,221: first message received
      2021-02-10 14:58:36,052: all splits combined – 72 seconds
      2021-02-10 15:01:06,971: message processing completed – 90 seconds
      2021-02-10 15:01:17,158: Kafka commit failed. Elapsed time since first message: 197 seconds
      2021-02-10 15:01:19,857: attempt #2: first message received
      2021-02-10 15:03:01,993: attempt #2: all splits combined – 102 seconds
      2021-02-10 15:04:44,896: attempt #2: Kafka commit failed. Elapsed time since first message: 205 seconds
      Back to #5

      Solution

      Maintain last offset received. If the same offset is presented, commit the offset and move on to the next message.

       

       

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                amestry Ashutosh Mestry
                Reporter:
                amestry Ashutosh Mestry
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: