Test
Minor Description
There are issues in the past where the JobTracker has failures because of so called "memory-leaks" , or infinitely growing objects, unclean tmp/ folders from jobs, etc.
Thus, the proposal here is a lighteweight job which can be run rapidly, several 1000 times, which confirms that jobtracker state does not grow out of bounds or infinitely with respect to number of tasks/jobs run/submitted.
IMPLEMENTATION PROPOSAL:
Some simple starts would be to :
- run word count, or a the sleep job 100 or 1000 times or 10,000 times.
- create and delete the same file over and over again several thousand times to see if filesystem consistency is maintained
To start, I'd like to add all these tests in a single module , under test-executions/stress/. Then later we could shard it out in another way.
UPDATE:
As per comments below, just noting that although phrased in terms of "JobTracker", the spirit of this ticket is to be applicable in both mr1 and mr2, since in either case, the purpose is to test the impact that several 100/1000 mapreduce job runs has over time and confirm that tmp dirs, objects in memory, etc are all managed and lifecycled properly .
Attachments
Issue Links
- duplicates
-
BIGTOP-1198 itest should provide smoke tests with a way to simulate heavy load on the node
-
- Resolved
-
- is related to
-
BIGTOP-1207 Create a foundation layer to build longevity and performance tests for the stack
-
- Resolved
-