[YARN-11054] Alleviate LocalJobRunnerMetricName Conflicts

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.1
• Fix Version/s: 3.3.1
• Labels: pull-request-available
• Target Version/s: 3.5.0
• Assignee: Unassigned
• Reporter: Xingjun Hao
• Created: 28/Dec/21 03:24
• Updated: 04/Jan/24 08:29

Description

In some scenarios, Sqoop will use LocalJobRuner (YarnLocal mode) to run a lot of jobs,  assuming  2 million jobs have been run, LocalJobRunner MetricName generated by nextInt function is in the range of (0, 2147483647),
Then the probability of conflict is about 2000/2147483647 = 1/1000, which means that an average of 1 task will fail for every 1000 jobs run.
If LocalJobRunner MetricName is generated by nextLong() whose range is (0, 9223372036854775807), considering that Long's range is 1 billion times that of Int, the probability of a new MetricName conflicts is also reduced by one trillionth times.
the probability of conflict also goes to 1/500000000 from 1/1000. ( under the situation that there are about 200million jobs have been run)

Issue Links

• links to: GitHub Pull Request #3834