Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22729

Truncated Messages in Python workers

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Critical
    • Resolution: Cannot Reproduce
    • statefun-2.2.2
    • statefun-3.1.0
    • Stateful Functions
    • None
    • The Stateful Function version is 2.2.2, java8. The Java App as well as
      the external Python workers are deployed in the same kubernetes cluster.

    Description

      Recently we started seeing the following faulty behavior in the Flink
      Stateful Functions HTTP communication towards external Python workers.
      This is only occurring when the system is under heavy load.

      The Java Application will send HTTP Messages to an external Python
      Function but the external Function fails to parse the message with a
      "Truncated Message Error". Printouts show that the truncated message
      looks as follows:

      <Start of Message>
      
      my.protobuf.MyClass: <Protobuf Content>
      
      my.protobuf.MyClass: <Protobuf Content>
      
      my.protobuf.MyClass: <Protobuf Content>
      
      my.protobuf.MyClass: <Protob
      

      Which leads to the following Error in the Python worker:

      Error Parsing Message: Truncated Message
      

      Either the sender or the receiver (or something in between) seems to be
      truncacting some (not all) messages at some random point in the payload.
      The source code in both Flink SDKs looks to be correct. We temporarily
      solved this by setting the "maxNumBatchRequests" parameter in the
      external function definition really low. But this is not an ideal
      solution as we believe this adds considerable communication overhead
      between the Java and the Python Functions.

      The Stateful Function version is 2.2.2, java8. The Java App as well as
      the external Python workers are deployed in the same kubernetes cluster.


      This was reported on the Mailing List in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Statefun-Truncated-Messages-in-Python-workers-td43831.html

      Attachments

        Activity

          People

            Unassigned Unassigned
            sewen Stephan Ewen
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: