[HADOOP-18222] Prevent DelegationTokenSecretManagerMetrics from registering multiple times

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: common
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: Hector Sandoval Chaverri
• Reporter: Hector Sandoval Chaverri
• Created: 05/May/22 16:17
• Updated: 28/Jan/24 08:24
• Resolved: 10/May/22 21:56

Description

After committing HADOOP-18167, we received reports of the following error when ResourceManager is initialized:

Caused by: java.io.IOException: Problem starting http server
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1389)
        at org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:475)
        ... 4 more
Caused by: org.apache.hadoop.metrics2.MetricsException: Metrics source DelegationTokenSecretManagerMetrics already exists!
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
        at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
        at org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
        at org.apache.hadoop.metrics2.MetricsSystem.register(MetricsSystem.java:71)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager$DelegationTokenSecretManagerMetrics.create(AbstractDelegationTokenSecretManager.java:878)
        at org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.<init>(AbstractDelegationTokenSecretManager.java:152)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$DelegationTokenSecretManager.<init>(DelegationTokenManager.java:72)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.<init>(DelegationTokenManager.java:122)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:161)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.init(DelegationTokenAuthenticationHandler.java:130)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.initializeAuthHandler(DelegationTokenAuthenticationFilter.java:214)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
        at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationFilter.init(DelegationTokenAuthenticationFilter.java:180)
        at org.apache.hadoop.yarn.server.security.http.RMAuthenticationFilter.init(RMAuthenticationFilter.java:53)

This can happen if MetricsSystemImpl#init is called and multiple metrics are registered with the same name. A proposed solution is to declare the metrics in AbstractDelegationTokenSecretManager as singleton, which would prevent multiple instances DelegationTokenSecretManagerMetrics from being registered.

Issue Links

• Dependent: YARN-11122 Support getClusterNodes API in FederationClientInterceptor
• Dependent: YARN-11130 RouterClientRMService Has Unused import
• Dependent: YARN-11123 ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use
• is broken by: HADOOP-18167 Add metrics to track delegation token secret manager operations
• is fixed by: YARN-11138 TestRouterWebServicesREST Junit Test Error Fix
• links to: GitHub Pull Request #4266