[YARN-6523] Optimize system credentials sent in node heartbeat responses - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 2.8.0, 2.7.3
Fix Version/s: 3.3.0
Component/s: RM
Labels:
None

Target Version/s:

3.3.0
Hadoop Flags:

Reviewed

Description

Currently as part of heartbeat response RM sets all application's tokens though all applications might not be active on the node. On top of it NodeHeartbeatResponsePBImpl converts tokens for each app into SystemCredentialsForAppsProto. Hence for each node and each heartbeat too many SystemCredentialsForAppsProto objects were getting created.
We hit a OOM while testing for 2000 concurrent apps on 500 nodes cluster with 8GB RAM configured for RM

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending

YARN-6523.015.patch
08/Jan/19 08:10
80 kB
Manikandan R
YARN-6523.014.patch
07/Jan/19 15:26
80 kB
Manikandan R
YARN-6523.013.patch
05/Jan/19 08:50
80 kB
Manikandan R
YARN-6523.012.patch
04/Jan/19 16:30
79 kB
Manikandan R
YARN-6523.011.patch
10/Dec/18 17:24
79 kB
Manikandan R
YARN-6523.010.patch
07/Dec/18 16:37
79 kB
Manikandan R
YARN-6523.009.patch
02/Dec/18 17:31
67 kB
Manikandan R
YARN-6523.008.patch
28/Nov/18 17:52
62 kB
Manikandan R
YARN-6523.007.patch
21/Nov/18 17:56
37 kB
Manikandan R
YARN-6523.006.patch
12/Nov/18 17:12
24 kB
Manikandan R
YARN-6523.005.patch
25/Oct/18 16:14
31 kB
Manikandan R
YARN-6523.004.patch
25/Oct/18 04:03
31 kB
Manikandan R
YARN-6523.003.patch
16/Feb/18 15:58
30 kB
Manikandan R
YARN-6523.002.patch
20/Nov/17 11:44
30 kB
Manikandan R
YARN-6523.001.patch
17/Nov/17 04:47
31 kB
Manikandan R

Activity

People

Assignee:: Manikandan R

Reporter:: Naganarasimha G R

Votes:: 0 Vote for this issue

Watchers:: 16 Start watching this issue

Dates

Created:: 25/Apr/17 02:13

Updated:: 26/Apr/21 04:22

Resolved:: 08/Jan/19 22:58