[BEAM-3401] Fully specify the length prefix coder

• Type: Improvement
• Status: Open
• Priority: P3
• Resolution: Unresolved
• Component/s: sdk-go, sdk-java-core, sdk-py-core
• Labels: Clarified portability
• Assignee: Unassigned
• Reporter: Luke Cwik
• Created: 02/Jan/18 23:34
• Updated: 03/Jun/22 20:14

Description

Choose a variable integer encoding that is efficient in encoding/decoding speed out of one of the many variants like LEB128 / PrefixVarint / SQLite varint

Also update the length prefix encoding to be parameterized by a block size so that the entire object doesn't have to be written in memory as per http://doc/1IGduUqmhWDi_69l9nG8kw73HZ5WI5wOps9Tshl5wpQA/edit#heading=h.j8zpiclux4id

Add cross language validation tests for encoding/decoding to https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/data/standard_coders.yaml

Document the format explicitly and choose a stable URN.