Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14398

Further split input unboxing code into separate methods

    XMLWordPrintableJSON

    Details

      Description

      In one of our production pipelines, we have a table with 1200+ columns.  At runtime, it failed due to a method inside the generated code exceeding 64kb when compiled to bytecode.

      After we investigated the generated code, it appeared that the map method inside a generated RichMapFunction was too long. See attached file (codegen.example.txt).

      In the problematic map method, result setters were correctly split into individual methods and did not have the largest footprint.

      However, there were also 1000+ input unboxing expressions inside reusableInputUnboxingExprs, which, individually were not trivial and when flattened linearly in the map function here, pushed the method size beyond 64kb in bytecode.

      We think it is worthwhile to split these input unboxing code snippets into individual methods.  We were able to verify, in our production environment, that splitting input unboxing code snippets into individual methods resolves the issue.  Would love to hear thoughts from the team and find the best path to fix it.

        Attachments

        1. codegen.example.txt
          731 kB
          Hao Dang

          Activity

            People

            • Assignee:
              haodang Hao Dang
              Reporter:
              haodang Hao Dang
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 40m
                40m