Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Unresolved
-
None
-
None
Description
By default, Kafka-connect sink using Java client should use direct markers by default. Errors are thrown if timeline-server-based markers are used. This could be because each sink task worker starts its own embedded timeline server, causing concurrent writes to the same marker file, leading to undefined behavior.
Found checksum error: b[0, 91]=706172746974696f6e5f342f31463946373736444442363643353038373631333144443446443036303332365f302d302d305f32303231313132393137343031393530362e706172717565742e6d61726b65722e415050454e440a (org.apache.hadoop.fs.FSInputChecker:309)org.apache.hadoop.fs.ChecksumException: Checksum error: file:/tmp/hoodie/hudi-test-topic/.hoodie/.temp/20211129174217738/MARKERS6 at 0 exp: -509813218 got: -1454124197
https://gist.github.com/yihua/12d02aec4174b657b2a8ac3cd7972a5a
Attachments
Issue Links
- links to