Description
Currently each SegmentNodeStoreBranch (and thus Root) instance has it's own SegmentWriter that it uses to construct new segments. This allows maximum write concurrency as there's no need for any synchronization at the SegmentWriter level.
The downside of this solution is that the resulting segments are often very small (< 100 bytes), which increases write overhead (in TarMK each segment needs at least 1kB of disk space) and reduces locality of reference (related content gets scattered across many segments). And concurrent write throughput is in any case constrained when more than one writer is updating the same journal.
Thus I think it would be good to instead have just one SegmentWriter per SegmentNodeStore and have all branches use that shared writer. This requires synchronization when writing individual records, and can thus lead to extra contention, but since records are fine-grained and all record writes happen in memory, the amount of contention should be fairly small for normal workloads (and for truly heavy concurrent writing, we'd in any case be using multiple processes across many cluster nodes, in which case this limit wouldn't apply).
Attachments
Issue Links
- relates to
-
OAK-1828 Improved SegmentWriter
-
- Closed
-