Uploaded image for project: 'Apache Nemo'
  1. Apache Nemo
  2. NEMO-144 Improve Data Plane Code
  3. NEMO-149

Integrate BlockTransferThrottler and backgroundExecutorService

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      There exists two different configuration knobs for throttling remote block I/Os in executors.

      • IORequestHandleThreadsTotal: BlockManagerWorker#backgroundExecutorService
      • MaxNumDownloadsForARuntimeEdge: BlockTransferThrottler

      Things can be simpler if the two are integrated into a single mechanism.

      I like IORequestHandleThreadsTotal (the first one) better for the following reasons

      • This sets the number of threads used in Executors.newFixedThreadPool. When a thread fails for whatever reason, a new thread is spawned to handle next operations. Thus it is more resilient against unforseen exceptions creating zombies that prevent next operations from running.
      • Throttling should be configured in an executor-centric way, and not be affected by the shape of the IR. In case of the second one, if we happen to have 100 incoming runtime edges for the task, then we allow for (100 * MaxNumDownloadsForARuntimeEdge concurrent) data transfers, which may put the executor under significant concurrent load.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              johnyangk John Yang
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: