[HADOOP-17130] Configuration.getValByRegex() shouldn't update the results while fetching.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.3
• Fix Version/s: 3.2.2, 3.3.1
• Component/s: common
• Target Version/s: 3.1.4
• Assignee: Mukund Thakur
• Reporter: Mukund Thakur
• Created: 15/Jul/20 12:39
• Updated: 10/Jun/21 07:57
• Resolved: 17/Jul/20 09:42

Description

We have seen this stacktrace while using ABFS file system. After analysing the stack trace we can see that getValByRegex() is reading the properties and substituting the value in the same call. This may cause the ConcurrentModificationException. 

Caused by: java.util.concurrent.ExecutionException: java.util.ConcurrentModificationException at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1877) ... 18 more Caused by: java.util.ConcurrentModificationException at java.util.Hashtable$Enumerator.next(Hashtable.java:1387) at org.apache.hadoop.conf.Configuration.getValByRegex(Configuration.java:3855) at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.validateStorageAccountKeys(AbfsConfiguration.java:689) at org.apache.hadoop.fs.azurebfs.AbfsConfiguration.<init>(AbfsConfiguration.java:237) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.<init>(AzureBlobFileSystemStore.java:154) at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.initialize(AzureBlobFileSystem.java:113) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3396) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:158) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3456) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3424) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:518) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:361)
 

Issue Links

• links to: PR