[BEAM-9502] SchemaCoder is not update compatible

• Type: Bug
• Status: Resolved
• Priority: P3
• Resolution: Duplicate
• Fix Version/s: Not applicable
• Component/s: runner-dataflow, sdk-java-core
• Labels: Clarified stale-assigned
• Assignee: Unassigned
• Reporter: Yaron Neuman
• Created: 13/Mar/20 17:53
• Updated: 17/May/21 19:32
• Resolved: 17/May/21 19:31

Description

See relevant dev@ discussion. Runners should consider schemas compatible if they have the same fields in the same order. They should also get the ability to re-order fields in equivalent schemas (same fields, possibly out of order) using the encoding_position field (BEAM-10277).

Original Description:

SchemaCoder assigns random UUID, causes Dataflow's compatibility check to fail

After fe4b7794, Schema.equals comparing only the UUIDs for faster comparison.
After 0b3b18c6 SchemaCoder forcing random UUID when schema.uuid is null.

thus, when trying to update (--update) a Dataflow job with row schemas in user-code, the compatibility check will fail because SchemaCoder produce another random UUID.

 

The user can set the UUID after creating the Schema, but not with Schema.Builder
and I'm afraid most users, that are not aware to the internal implementation, won't do that.

 

In my branch, I added .withUUID and .withRandomUUID to Schema.Builder

But I think a better solution will be to calculate the UUID based on the schema itself.

any thoughts?

reuvenlax

 

Issue Links

• duplicates: BEAM-12198 Support Dataflow update when schemas are used
• relates to: BEAM-10277 beam:coder:row:v1 implementations should respect encoding_position
• links to: GitHub Pull Request #11447
• links to: GitHub Pull Request #13049
• links to: GitHub Pull Request #13049