[BEAM-9152] Hadoop Downloader Range not correct

• Type: Bug
• Status: Triage Needed
• Priority: P1
• Resolution: Unresolved
• Fix Version/s: Missing
• Component/s: io-py-tfrecord
• Assignee: Unassigned
• Reporter: Fangyuan Zhou
• Created: 20/Jan/20 06:58
• Updated: 04/Jun/22 15:28

Description

I found that the `HdfsDownloader.get_range(self, start, end)' get range [start, end] rather than [start, end). This will cause an error while reading hdfs file.

 

File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/tfrecordio.py", line 127, in read_record
buf = file_handle.read(buf_length_expected)
File "/data/anaconda3/envs/tfdv1.15/lib/python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
b[:len(data)] = data
ValueError: memoryview assignment: lvalue and rvalue have different structures

Issue Links

• is duplicated by: BEAM-14493 HdfsDownloader gets wrong range