Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-25318 Improvement of scheduler and execution for Flink OLAP
  3. FLINK-33532

Move the serialization of ShuffleDescriptorGroup out of the RPC main thread

    XMLWordPrintableJSON

Details

    Description

      Currently, the serilization of ShuffleDescriptorGroup would happen in the RPC main thread even for light jobs. That would harm the QPS of OLAP session, e.g. the deserialization will take 3ms at TM side for each task of a 128*128 wordcount job. We propose to move it out of the rpc main thread. For heavy jobs, it still worth to cache the serialized value for memory efficiency. At the same time we need to make the serialization thread pool configurable.

      Attachments

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. FLINK-34105 Akka timeout happens in TPC-DS benchmarks

          • Critical - Crashes, loss of data, severe memory leak.
          • Closed
          links to

          Web Link GitHub Pull Request #23707

          Web Link GitHub Pull Request #24441

          Web Link GitHub Pull Request #24442

          Activity

            People

              lsdy dizhou cao
              guoyangze Yangze Guo
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: