[TWILL-254] Update to use ContainerId.fromString in Hadoop 2.6+

• Type: Bug
• Status: Resolved
• Priority: Trivial
• Resolution: Fixed
• Affects Version/s: 0.12.0
• Fix Version/s: 0.13.0, 0.12.1
• Component/s: yarn
• Assignee: Clay B.
• Reporter: Clay B.
• Created: 04/Feb/18 06:00
• Updated: 16/Mar/18 21:31
• Resolved: 06/Feb/18 01:02

Description

Hello, trying to use 0.12 or master, I get the following on a Hortonworks 2.6.1 (aka roughtly Hadoop 2.7.3) cluster. (One can avoid this by specifically building for Hadoop 2.6+ via mvn package -Dhadoop.version=2.6.0 but I was being a bit slow to realize what was up,) I have a patch to move us away from the deprecated (see YARN-1942) call in the stack trace anyhow and provide Hadoop 2.6 as a supported profile.

Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.twill.launcher.TwillLauncher.main(TwillLauncher.java:73)
Caused by: java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at com.google.common.base.Throwables.propagate(Throwables.java:160)
        at org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:61)
        at org.apache.twill.internal.appmaster.ApplicationMasterMain.doMain(ApplicationMasterMain.java:85)
        at org.apache.twill.internal.appmaster.ApplicationMasterMain.main(ApplicationMasterMain.java:71)
        ... 5 more
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.twill.internal.yarn.VersionDetectYarnAMClientFactory.create(VersionDetectYarnAMClientFactory.java:58)
        ... 7 more
Caused by: java.lang.IllegalArgumentException: Invalid ContainerId: container_e164_1516738752188_0024_01_000001
        at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:182)
        at org.apache.twill.internal.yarn.AbstractYarnAMClient.<init>(AbstractYarnAMClient.java:83)
        at org.apache.twill.internal.yarn.Hadoop21YarnAMClient.<init>(Hadoop21YarnAMClient.java:65)
        at org.apache.twill.internal.yarn.Hadoop22YarnAMClient.<init>(Hadoop22YarnAMClient.java:34)
        ... 12 more
Caused by: java.lang.NumberFormatException: For input string: "e164"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Long.parseLong(Long.java:589)
        at java.lang.Long.parseLong(Long.java:631)
        at org.apache.hadoop.yarn.util.ConverterUtils.toApplicationAttemptId(ConverterUtils.java:137)
        at org.apache.hadoop.yarn.util.ConverterUtils.toContainerId(ConverterUtils.java:177)
        ... 15 more

For reference, it looks like YARN-2182 appears in Hadoop 2.6 to have changed container ID formats and a new API call came out ContainerId.fromString. Meanwhile, the container ID format change was not reflected, of course, in older versions of Hadoop as clarified in YARN-3575.

Issue Links

• links to: GitHub Pull Request #65