Details

    • Bug
    • Status: Resolved
    • P2
    • Resolution: Fixed
    • 2.31.0
    • Not applicable
    • io-py-hadoop
    • None

    Description

      Trying to read avro data from HDFS from a python sidecar worker fails with:

      File "python3.7/site-packages/apache_beam/io/filesystemio.py", line 123, in readinto
      b[:len(data)] = data
      ValueError: memoryview assignment: lvalue and rvalue have different structures

      This is the same issue as https://issues.apache.org/jira/browse/BEAM-9152 which was marked as resolved without being resolved at all.

       As remarked by Jean-Christophe CARLES on https://issues.apache.org/jira/browse/BEAM-9152 , patching hadoopfilesystem.py by removing " + 1" in HdfsDownloader.get_range fixes it for us.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              Enzo90910 Vincent Bernardi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Slack

                  Issue deployment