[PARQUET-1923] parquet-tools 1.11.0: TestSimpleRecordConverter fails with ExceptionInInitializerError on openjdk 15

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.11.0
• Fix Version/s: 1.12.0

Environment

$ mvn --version
Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /usr/local/Cellar/maven/3.6.3_1/libexec
Java version: 15, vendor: N/A, runtime: /usr/local/Cellar/openjdk/15/libexec/openjdk.jdk/Contents/Home
Default locale: en_GB, platform encoding: UTF-8
OS name: "mac os x", version: "10.15.7", arch: "x86_64", family: "mac"

$ java -version
openjdk version "15" 2020-09-15
OpenJDK Runtime Environment (build 15+36)
OpenJDK 64-Bit Server VM (build 15+36, mixed mode, sharing)

$ sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.15.7
BuildVersion:	19H2

 

• Assignee: Alexander Bayandin
• Reporter: Alexander Bayandin
• Created: 13/Oct/20 10:59
• Updated: 27/Jan/21 14:33
• Resolved: 17/Oct/20 23:15

Description

mvn clean package -Plocal for parquet-tools 1.11.1 fails with a failing test testConverter(org.apache.parquet.tools.read.TestSimpleRecordConverter).

mvn clean -Dtest=TestSimpleRecordConverter "-Plocal" test:

--------------------------------------------------------------------------------------------------------------------------------------------------------------Test set: org.apache.parquet.tools.read.TestSimpleRecordConverter-------------------------------------------------------------------------------Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.338 sec <<< FAILURE!testConverter(org.apache.parquet.tools.read.TestSimpleRecordConverter)  Time elapsed: 0.268 sec  <<< ERROR!java.lang.ExceptionInInitializerError at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2823) at org.apache.hadoop.fs.FileSystem$Cache$Key.<init>(FileSystem.java:2818) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2684) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:357) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.parquet.hadoop.util.HadoopOutputFile.fromPath(HadoopOutputFile.java:58) at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:227) at org.apache.parquet.hadoop.ParquetWriter.<init>(ParquetWriter.java:192) at org.apache.parquet.tools.read.TestSimpleRecordConverter.createTestParquetFile(TestSimpleRecordConverter.java:114) at org.apache.parquet.tools.read.TestSimpleRecordConverter.setUp(TestSimpleRecordConverter.java:90) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24) at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4TestSet.execute(JUnit4TestSet.java:53) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:123) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:104) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray(ReflectionUtils.java:164) at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:110) at org.apache.maven.surefire.booter.SurefireStarter.invokeProvider(SurefireStarter.java:175) at org.apache.maven.surefire.booter.SurefireStarter.runSuitesInProcessWhenForked(SurefireStarter.java:107) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:68)Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2 at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3734) at java.base/java.lang.String.substring(String.java:1903) at org.apache.hadoop.util.Shell.<clinit>(Shell.java:52) ... 44 more

The error looks similar to SPARK-26134, so I suppose a fix should be the same.

Notes:

• I use parquet-tools 1.11.1 with https://github.com/apache/parquet-mr/commit/b6d07ae0744ba47aa9a8868ef2d7cbb232a60b22 patch
• I can't build current master at all: https://issues.apache.org/jira/browse/PARQUET-1896?focusedCommentId=17203146&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17203146