2.0 is the good time to redesign streaming API including protocol to make streaming more performant and reliable.
Design goals that come up in my mind:
- Protocol optimization
- Stream multiple files in parallel (
- Persistent connection (
- Cleaner API for error handling
- Integrate both IN/OUT streams into one session, so the components(bootstrap, move, bulkload, repair...) that use streaming can manage them easily.
- Better logging/tracing
- More metrics
- Progress reporting API for external client