Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14500 Support Flink Python User-Defined Stateless Function for Table - Phase 2
  3. FLINK-14020

User Apache Arrow as the serializer for data transmission between Java operator and Python harness

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 1.11.0
    • Component/s: API / Python
    • Labels:
      None

      Description

      Apache Arrow is "a cross-language development platform for in-memory data. It specifies a standardized language-independent columnar memory format for flat and hierarchical data, organized for efficient analytic operations on modern hardware". It has been widely used in many notable projects, such as Spark, Parquet, Pandas, etc.

      We should firstly benchmark whether it could improve the performance a lot for non-vectorized Python UDFs. If we see significant performance improvements, it would be great to use it for the Java/Python communication. Otherwise, record by record serializer will be used.

        Attachments

          Activity

            People

            • Assignee:
              dian.fu Dian Fu
              Reporter:
              dian.fu Dian Fu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: