[BEAM-2790] Error while reading from Amazon S3 via Hadoop File System

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.0.0, 2.1.0
• Fix Version/s: 2.2.0
• Component/s: io-java-aws
• Assignee: Etienne Chauchot
• Reporter: Ismaël Mejía
• Created: 22/Aug/17 12:49
• Updated: 24/Jul/20 20:17
• Resolved: 15/Sep/17 15:09

Description

If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not ByteBufferReadable.

Exception in thread "main" java.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream
	at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146)
	at org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192)
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232)
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166)
	at org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198)
	at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481)
	at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476)
	at org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)

Issue Links

• relates to: HADOOP-14603 S3A input stream to support ByteBufferReadable
• links to: GitHub Pull Request #3744
• links to: GitHub Pull Request #3850