[BEAM-738] Update sourcetestutils to fail when BoundedSources maintain state

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: sdk-py-core
• Assignee: Chamikara Madhusanka Jayalath
• Reporter: Chamikara Madhusanka Jayalath
• Created: 10/Oct/16 21:12
• Updated: 16/May/20 13:54
• Resolved: 18/Oct/16 19:50

Description

Maintaining state in BoundedSource implementations is problematic and can lead to hard to debug errors. For example (1) pickling errors (2) errors due to a runner reusing a BoundedSource object with state.

We can try to prevent users from adding state to BoundedSource implementations in following two ways.

(1) Clearly mention in BoundedSource API that objects should not maintain transient state.
(2) Update sourcetesutils to catch source objects that maintain local state.

(2) can be done by adding a check that verifies that a source produces expected output in the presence of a re-entrant read.

i = s.read_records()
i.next()
i.next()
read the whole thing from s.read_records()
i.next() some more

Verify that 'i' produced correct output.