Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14133 Improve batch sql and hive integrate performance milestone-1
  3. FLINK-14722

Optimize mapred.HadoopInputSplit to not serialize conf when split is not configurable

    XMLWordPrintableJSON

    Details

      Description

      JobConf may very big, contains hundreds of configurations, if it is serialized by every split, that will significantly reduce performance.

      Consider thousands of splits, the akka thread of JobMaster will all on the serialization of conf. That may will lead to various akka timeouts too.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                lzljs3620320 Jingsong Lee
                Reporter:
                lzljs3620320 Jingsong Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 20m
                  20m