Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-3396

uploading dependency jars too slow when StormSubmitter and Nimbus located in different IDC

    XMLWordPrintableJSON

    Details

      Description

           when storm client and server is locating in different IDC(one is in Beijing, while another in Shanghai), uploading dependency jars may take a very long long time(in my case, 31minutes!)...

          when I digged into this, I found that in DependencyUploader,  method "uploadDependencyToBlobStore" using JDK NIO's Files.copy to upload local jars to remote Blob server. In Files.copy(Path, OutputStream), the buffer size is 8k by default, given that latency between Beijing and Shanghai is about 20ms, a dependency fat jar of 360M finally cost me 'a lunch time' to finish uploading!!!

         

        Attachments

          Activity

            People

            • Assignee:
              StaticMian StaticMian
              Reporter:
              StaticMian StaticMian
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h