Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
-
None
Description
SplitText is fairly CPU-intensive and quite slow. A simple flow that splits a 1.4 million line text file into 5k line chunks and then splits those 5k line chunks into 1 line chunks is only capable of pushing through about 10k lines per second. This equates to about 10 MB/sec. JVisualVM shows that the majority of the time is spent in the locateSplitPoint() method. Isolating this code and inspecting how it works, and using some micro-benchmarking, it appears that if we refactor the calls to InputStream.read() to instead read into a byte array, we can improve performance.
