[FLINK-29092] HadoopPathBasedPartFileWriterTest.testWriteFile failed with AssertionError

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.16.0, 1.17.0, 1.15.2
• Fix Version/s: 1.17.0, 1.16.1, 1.15.4
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available test-stability
• Assignee: KurtDing
• Reporter: Xingbo Huang
• Created: 24/Aug/22 07:24
• Updated: 29/Nov/22 12:15
• Resolved: 29/Nov/22 12:15

Description

2022-08-24T02:00:01.1670618Z Aug 24 02:00:01 [ERROR] org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile  Time elapsed: 2.311 s  <<< FAILURE!
2022-08-24T02:00:01.1671250Z Aug 24 02:00:01 java.lang.AssertionError: 
2022-08-24T02:00:01.1671626Z Aug 24 02:00:01 
2022-08-24T02:00:01.1672001Z Aug 24 02:00:01 Expected size: 1 but was: 2 in:
2022-08-24T02:00:01.1673656Z Aug 24 02:00:01 [DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--02; isDirectory=true; modification_time=1661306400000; access_time=1661306400396; owner=; group=; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false},
2022-08-24T02:00:01.1676131Z Aug 24 02:00:01     DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--01; isDirectory=true; modification_time=1661306400000; access_time=1661306400326; owner=; group=; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}]
2022-08-24T02:00:01.1677339Z Aug 24 02:00:01 	at org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.validateResult(HadoopPathBasedPartFileWriterTest.java:107)
2022-08-24T02:00:01.1678274Z Aug 24 02:00:01 	at org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile(HadoopPathBasedPartFileWriterTest.java:97)
2022-08-24T02:00:01.1679017Z Aug 24 02:00:01 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2022-08-24T02:00:01.1679666Z Aug 24 02:00:01 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2022-08-24T02:00:01.1680369Z Aug 24 02:00:01 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2022-08-24T02:00:01.1681019Z Aug 24 02:00:01 	at java.lang.reflect.Method.invoke(Method.java:498)
2022-08-24T02:00:01.1681666Z Aug 24 02:00:01 	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
2022-08-24T02:00:01.1682385Z Aug 24 02:00:01 	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2022-08-24T02:00:01.1683094Z Aug 24 02:00:01 	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
2022-08-24T02:00:01.1683965Z Aug 24 02:00:01 	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2022-08-24T02:00:01.1684713Z Aug 24 02:00:01 	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
2022-08-24T02:00:01.1685956Z Aug 24 02:00:01 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
2022-08-24T02:00:01.1687185Z Aug 24 02:00:01 	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
2022-08-24T02:00:01.1688347Z Aug 24 02:00:01 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2022-08-24T02:00:01.1689324Z Aug 24 02:00:01 	at java.lang.Thread.run(Thread.java:748)
2022-08-24T02:00:01.1690062Z Aug 24 02:00:01 
2022-08-24T02:00:03.4727706Z Aug 24 02:00:03 Formatting using clusterid: testClusterID
2022-08-24T02:00:07.9860626Z Aug 24 02:00:07 [INFO] Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.712 s - in org.apache.flink.formats.hadoop.bulk.committer.HadoopRenameCommitterHDFSTest
2022-08-24T02:00:08.4139747Z Aug 24 02:00:08 [INFO] 
2022-08-24T02:00:08.4140678Z Aug 24 02:00:08 [INFO] Results:
2022-08-24T02:00:08.4141326Z Aug 24 02:00:08 [INFO] 
2022-08-24T02:00:08.4142008Z Aug 24 02:00:08 [ERROR] Failures: 
2022-08-24T02:00:08.4144242Z Aug 24 02:00:08 [ERROR]   HadoopPathBasedPartFileWriterTest.testWriteFile:97->validateResult:107 
2022-08-24T02:00:08.4145317Z Aug 24 02:00:08 Expected size: 1 but was: 2 in:
2022-08-24T02:00:08.4147711Z Aug 24 02:00:08 [DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--02; isDirectory=true; modification_time=1661306400000; access_time=1661306400396; owner=; group=; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false},
2022-08-24T02:00:08.4150885Z Aug 24 02:00:08     DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--01; isDirectory=true; modification_time=1661306400000; access_time=1661306400326; owner=; group=; permission=rwxrwxrwx; isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}] 

https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=40324&view=logs&j=7e3d33c3-a462-5ea8-98b8-27e1aafe4ceb&t=ef77f8d1-44c8-5ee2-f175-1c88f61de8c0

Issue Links

• is caused by: FLINK-17594 Support Hadoop path-based PartFileWriter with renaming committer
• is related to: FLINK-27185 Run the assertj conversion script to convert assertions in connectors and formats
• links to: GitHub Pull Request #20755
• links to: GitHub Pull Request #21413
• links to: GitHub Pull Request #21414