Details
-
Bug
-
Status: Resolved
-
P2
-
Resolution: Fixed
-
2.0.0, 2.1.0
-
None
Description
If you try to use hadoop-aws with Beam to read from AWS S3 it breaks because S3AInputStream (the implementation of Hadoop's FSDataInputStream) is not ByteBufferReadable.
Exception in thread "main" java.lang.UnsupportedOperationException: Byte-buffer read unsupported by input stream at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:146) at org.apache.beam.sdk.io.hdfs.HadoopFileSystem$HadoopSeekableByteChannel.read(HadoopFileSystem.java:192) at org.apache.beam.sdk.io.TextSource$TextBasedReader.tryToEnsureNumberOfBytesInBuffer(TextSource.java:232) at org.apache.beam.sdk.io.TextSource$TextBasedReader.findSeparatorBounds(TextSource.java:166) at org.apache.beam.sdk.io.TextSource$TextBasedReader.readNextRecord(TextSource.java:198) at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.advanceImpl(FileBasedSource.java:481) at org.apache.beam.sdk.io.FileBasedSource$FileBasedReader.startImpl(FileBasedSource.java:476) at org.apache.beam.sdk.io.OffsetBasedSource$OffsetBasedReader.start(OffsetBasedSource.java:261)
Attachments
Issue Links
- relates to
-
HADOOP-14603 S3A input stream to support ByteBufferReadable
- Open
- links to