[BEAM-12573] Bounded Source Reader DoFn PR #13154 causes some pipelines to fail with a PicklingError

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.33.0
• Component/s: io-py-common
• Assignee: Valentyn Tymofieiev
• Reporter: Valentyn Tymofieiev
• Created: 03/Jul/21 05:41
• Updated: 15/Dec/21 00:34
• Resolved: 15/Oct/21 20:50

Description

As reported in [1], when trying to run the coders.py cookbook example [2], it raises an
exception

```
PicklingError : _pickle.PicklingError: Can't pickle <class
'_main.JsonCoder'>: it's not the same object as __main_.JsonCoder
[while running 'read/Read/Map(<lambda at iobase.py:899>)']
```

To reproduce the issue, run:
```
python coders.py --input input.ndjson --output output.txt
```

Where the input file (input.ndjson) has the same values as coders_test.py:
```

{"host": ["Germany", 1], "guest": ["Italy", 0]} {"host": ["Germany", 1], "guest": ["Brasil", 3]} {"host": ["Brasil", 1], "guest": ["Italy", 0]}

```

I have tracked the change in behavior to[3] with git bisect. This needs further investigation to understand the issue.

[1] https://lists.apache.org/thread.html/r45340bbee91a6caf798fe62d24388f645f8792cc7506351fd66adec3%40%3Cdev.beam.apache.org%3E
[2] https://github.com/apache/beam/blob/35bac6a62f1dc548ee908cfeff7f73ffcac38e6f/sdks/python/apache_beam/examples/cookbook/coders.py
[3] https://github.com/apache/beam/pull/13154

Issue Links

• links to: GitHub Pull Request #15125