Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-8414

Gelly performance seriously decreases when using the suggested parallelism configuration

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Feedback Received
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      I am running Gelly examples with different datasets in a cluster of 5 machines (1 Jobmanager and 4 Taskmanagers) of 32 cores each.

      The number of Slots parameter is set to 32 (as suggested) and the parallelism to 128 (32 cores*4 taskmanagers).

      I observe a vast performance degradation using these suggested settings than setting parallelism.default to 16 for example were the same job completes at ~60 seconds vs ~140 in the 128 parallelism case.

      Is there something wrong in my configuration? Should I decrease parallelism and if so will this inevitably decrease CPU utilization?

      Another matter that may be related to this is the number of partitions of the data. Is this somehow related to parallelism? How many partitions are created in the case of parallelism.default=128?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              flora_k flora karniav
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: