Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6128

Optimize JVM options for improve test performance

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: None
    • Fix Version/s: 1.3.0
    • Component/s: Tests
    • Labels:
      None
    • Environment:

      maven-surefire-plugin

      Description

      Tune JVM options for run tests by maven-surefire-plugin at travis

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user DmytroShkvyra opened a pull request:

          https://github.com/apache/flink/pull/3571

          FLINK-6128 Optimize JVM options for improve test performance

          The main goal of this PR improve latency of running tests with travis and prevent OOM.
          1. Increased heap size
          2. Use UseSerialGC
          3. Remove UseGCOverheadLimit option

          • [ ] General
          • The pull request references the related JIRA issue ("FLINK-6128 Optimize JVM options for improve test performance")

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/DmytroShkvyra/flink FLINK-6128

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3571.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3571


          commit 36b3a627c86482872d59cf99c14a807f14c5be6c
          Author: DmytroShkvyra <dshkvyra@gmail.com>
          Date: 2017-03-20T11:12:37Z

          FLINK-6128 Optimize JVM options for improve test performance


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user DmytroShkvyra opened a pull request: https://github.com/apache/flink/pull/3571 FLINK-6128 Optimize JVM options for improve test performance The main goal of this PR improve latency of running tests with travis and prevent OOM. 1. Increased heap size 2. Use UseSerialGC 3. Remove UseGCOverheadLimit option [ ] General The pull request references the related JIRA issue (" FLINK-6128 Optimize JVM options for improve test performance") You can merge this pull request into a Git repository by running: $ git pull https://github.com/DmytroShkvyra/flink FLINK-6128 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3571.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3571 commit 36b3a627c86482872d59cf99c14a807f14c5be6c Author: DmytroShkvyra <dshkvyra@gmail.com> Date: 2017-03-20T11:12:37Z FLINK-6128 Optimize JVM options for improve test performance
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DmytroShkvyra commented on the issue:

          https://github.com/apache/flink/pull/3571

          Hi @rmetzger could you review this PR?

          Show
          githubbot ASF GitHub Bot added a comment - Github user DmytroShkvyra commented on the issue: https://github.com/apache/flink/pull/3571 Hi @rmetzger could you review this PR?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DmytroShkvyra commented on the issue:

          https://github.com/apache/flink/pull/3571

          I don't think that:
          **Results :
          Failed tests:
          ZooKeeperLeaderElectionTest.testZooKeeperReelection:197 null**
          is mine fault.

          Show
          githubbot ASF GitHub Bot added a comment - Github user DmytroShkvyra commented on the issue: https://github.com/apache/flink/pull/3571 I don't think that: **Results : Failed tests: ZooKeeperLeaderElectionTest.testZooKeeperReelection:197 null** is mine fault.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DmytroShkvyra closed the pull request at:

          https://github.com/apache/flink/pull/3571

          Show
          githubbot ASF GitHub Bot added a comment - Github user DmytroShkvyra closed the pull request at: https://github.com/apache/flink/pull/3571
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user DmytroShkvyra reopened a pull request:

          https://github.com/apache/flink/pull/3571

          FLINK-6128 Optimize JVM options for improve test performance

          The main goal of this PR improve latency of running tests with travis and prevent OOM.
          1. Increased heap size
          2. Use UseSerialGC
          3. Remove UseGCOverheadLimit option

          • [ ] General
          • The pull request references the related JIRA issue ("FLINK-6128 Optimize JVM options for improve test performance")

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/DmytroShkvyra/flink FLINK-6128

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3571.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3571


          commit 36b3a627c86482872d59cf99c14a807f14c5be6c
          Author: DmytroShkvyra <dshkvyra@gmail.com>
          Date: 2017-03-20T11:12:37Z

          FLINK-6128 Optimize JVM options for improve test performance


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user DmytroShkvyra reopened a pull request: https://github.com/apache/flink/pull/3571 FLINK-6128 Optimize JVM options for improve test performance The main goal of this PR improve latency of running tests with travis and prevent OOM. 1. Increased heap size 2. Use UseSerialGC 3. Remove UseGCOverheadLimit option [ ] General The pull request references the related JIRA issue (" FLINK-6128 Optimize JVM options for improve test performance") You can merge this pull request into a Git repository by running: $ git pull https://github.com/DmytroShkvyra/flink FLINK-6128 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3571.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3571 commit 36b3a627c86482872d59cf99c14a807f14c5be6c Author: DmytroShkvyra <dshkvyra@gmail.com> Date: 2017-03-20T11:12:37Z FLINK-6128 Optimize JVM options for improve test performance
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user twalthr commented on the issue:

          https://github.com/apache/flink/pull/3571

          Do you have some numbers about how much your changes improve tests?

          Show
          githubbot ASF GitHub Bot added a comment - Github user twalthr commented on the issue: https://github.com/apache/flink/pull/3571 Do you have some numbers about how much your changes improve tests?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user DmytroShkvyra commented on the issue:

          https://github.com/apache/flink/pull/3571

          @twalthr Yes of course! Following data I have sent to dev list
          >
          > # Test|Present JVM options |"-Xms256m -Xmx1536m -XX:+UseSerialGC"
          > -----------------------------------------------------------------
          > 1 35 min 53 sec |35 min 35 sec
          > 2 38 min 49 sec |34 min 18 sec
          > 3 35 min 34 sec |29 min 38 sec
          > 4 34 min 38 sec |31 min 14 sec
          > 5 35 min 41 sec |35 min 11 sec
          > 6 36 min 41 sec |33 min 52 sec
          > 7 49 min 59 sec |31 min 35 sec
          > 8 37 min 0 sec |36 min 20 sec
          > 9 32 min 28 sec |31 min 48 sec
          > 10 38 min 25 sec |33 min 28 sec
          > 11 36 min 19 sec |38 min 24 sec
          > 12 25 min 0 sec |24 min 3 sec
          > ------------------------------------------------------------------
          > Total 26187 sec |23726 sec
          > ------------------------------------------------------------------
          > Acceleration |9.40%

          Also Evgeny Kincharov have done some tests with different options https://travis-ci.org/kenmy/flink/builds (see builds with "fix unittests time" description). He and I sure that using serial GC more suitable for tests than use default GCs of Java 7 and 8 (we have only one core per fork)

          Show
          githubbot ASF GitHub Bot added a comment - Github user DmytroShkvyra commented on the issue: https://github.com/apache/flink/pull/3571 @twalthr Yes of course! Following data I have sent to dev list > > # Test|Present JVM options |"-Xms256m -Xmx1536m -XX:+UseSerialGC" > ----------------------------------------------------------------- > 1 35 min 53 sec |35 min 35 sec > 2 38 min 49 sec |34 min 18 sec > 3 35 min 34 sec |29 min 38 sec > 4 34 min 38 sec |31 min 14 sec > 5 35 min 41 sec |35 min 11 sec > 6 36 min 41 sec |33 min 52 sec > 7 49 min 59 sec |31 min 35 sec > 8 37 min 0 sec |36 min 20 sec > 9 32 min 28 sec |31 min 48 sec > 10 38 min 25 sec |33 min 28 sec > 11 36 min 19 sec |38 min 24 sec > 12 25 min 0 sec |24 min 3 sec > ------------------------------------------------------------------ > Total 26187 sec |23726 sec > ------------------------------------------------------------------ > Acceleration |9.40% Also Evgeny Kincharov have done some tests with different options https://travis-ci.org/kenmy/flink/builds (see builds with "fix unittests time" description). He and I sure that using serial GC more suitable for tests than use default GCs of Java 7 and 8 (we have only one core per fork)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user rmetzger commented on the issue:

          https://github.com/apache/flink/pull/3571

          +1

          Show
          githubbot ASF GitHub Bot added a comment - Github user rmetzger commented on the issue: https://github.com/apache/flink/pull/3571 +1
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user greghogan commented on the issue:

          https://github.com/apache/flink/pull/3571

          Merging ...

          Show
          githubbot ASF GitHub Bot added a comment - Github user greghogan commented on the issue: https://github.com/apache/flink/pull/3571 Merging ...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3571

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3571
          Hide
          greghogan Greg Hogan added a comment -

          Implemented in 976e03c1ee0c1ac03e7755aef6a38ee0abc35b2f

          Show
          greghogan Greg Hogan added a comment - Implemented in 976e03c1ee0c1ac03e7755aef6a38ee0abc35b2f

            People

            • Assignee:
              dshkvyra Dmytro Shkvyra
              Reporter:
              dshkvyra Dmytro Shkvyra
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development