[SAMZA-2481] Improve memory usage in samza-core unit tests

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.5
• Assignee: Cameron Lee
• Reporter: Cameron Lee
• Created: 09/Mar/20 19:36
• Updated: 27/May/20 03:37
• Resolved: 24/Apr/20 16:48

Description

Gradle 5 reduced the default memory allocation for test executors. Before Gradle 5, our unit tests were given plenty of memory, so they ran fine. When upgrading to Gradle 5, we noticed some issues with running tests. Setting the memory (to 3GB heap) explicitly seems to have resolved the issues, but it seems odd that unit tests need so much. It would be good to investigate this possible inefficiency. It might also help build times, since heavy memory usage could mean a unnecessarily heavy unit test.

One potential example is TestZkMetadataStore. It tests large zookeeper values (10MB), and that seems to cause some issues. If heap usage is set to 1GB or 1.5GB, then that test takes a long time to execute (seems like it might be running out of memory). This can be reproduced by changing the heap usage for "test" for the samza-core module in build.gradle, and then running "./gradlew :samza-core:test". By setting the zookeeper value to 1MB (and using 1GB of heap), the tests will execute fine. It is odd that the test works on the scale of MB but seems to have memory requirements on the scale of GB.

Issue Links

• links to: GitHub Pull Request #1324