[HADOOP-14996] wasb: ReadFully occasionally fails when using page blobs

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fs/azure
• Target Version/s: 3.5.0
• Assignee: Thomas Marqardt
• Reporter: Thomas Marqardt
• Created: 31/Oct/17 01:04
• Updated: 04/Jan/24 11:26

Description

Looks like there is a functional bug or concurrency bug in the PageBlobInputStream implemenation of ReadFully.

1) Use 1 mapper to copy results in success:
hadoop distcp -m 1 wasb://[e-mail]/hive_tables  wasb://[e-mail]/hdi_backup

2) Turn on DEBUG log by setting mapreduce.map.log.level=DEBUG in ambari. Then run with more than 1 mapper:

Saw debug log like this:

2017-10-27 06:18:53,545 DEBUG [main] org.apache.hadoop.fs.azure.NativeAzureFileSystem: Seek to position 136251. Bytes skipped 136210
2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.fs.azure.AzureNativeFileSystemStore: Closing page blob output stream.
2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.fs.azure.AzureNativeFileSystemStore: java.util.concurrent.ThreadPoolExecutor@73dce0e6[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 0]
2017-10-27 06:18:53,549 DEBUG [main] org.apache.hadoop.security.UserGroupInformation: PrivilegedActionException as:mssupport (auth:SIMPLE) cause:java.io.EOFException
2017-10-27 06:18:53,553 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at org.apache.hadoop.io.SequenceFile$Reader.sync(SequenceFile.java:2693)
at org.apache.hadoop.mapreduce.lib.input.SequenceFileRecordReader.initialize(SequenceFileRecordReader.java:58)
at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:548)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:786)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:170)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1866)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:164)