[BEAM-14493] HdfsDownloader gets wrong range

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Affects Version/s: 2.31.0
• Fix Version/s: Not applicable
• Component/s: io-py-hadoop
• Assignee: Unassigned
• Reporter: Vincent Bernardi
• Created: 19/May/22 18:42
• Updated: 02/Jun/22 14:16
• Resolved: 26/May/22 18:01

Description

Trying to read avro data from HDFS from a python sidecar worker fails with:

File "python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
b[:len(data)] = data
ValueError: memoryview assignment: lvalue and rvalue have different structures

This is the same issue as https://issues.apache.org/jira/browse/BEAM-9152 which was marked as resolved without being resolved at all.

 As remarked by Jean-Christophe CARLES on https://issues.apache.org/jira/browse/BEAM-9152 , patching hadoopfilesystem.py by removing " + 1" in HdfsDownloader.get_range fixes it for us.

Issue Links

• duplicates: BEAM-9152 Hadoop Downloader Range not correct