Details
-
Improvement
-
Status: Resolved
-
Critical
-
Resolution: Fixed
-
None
-
None
Description
What's the problem ?
Read 1000M object, it cost about 470 seconds, i.e. 2.2M/s, which is too slow.
What's the reason ?
When read 1000M file, there are 50 GET requests, each GET request read 20M. When do GET, the stack is: IOUtils::copyLarge -> IOUtils::skipFully -> IOUtils::skip -> InputStream::read.
It means, the 50th GET request which should read 980M-1000M, but to skip 0-980M, it also InputStream::read 0-980M. So the 1st GET request read 0-20M, the 2nd GET request read 0-40M, the 3rd GET request read 0-60M, ..., the 50th GET request read 0-1000M. So the GET request from 1st-50th become slower and slower.
You can also refer it here why IOUtils implement skip by read rather than real skip, e.g. seek.
