Hive
  1. Hive
  2. HIVE-4936

PTest increase performance by parallelizing rsync

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently the bottom neck in PTest2 when adding hosts is the rsync to the slaves. Today if we have 3 "threads" which will execute on a slave we rsync the source from the master to the slave 3 times. This causes the process to be CPU bound on the master.

        Issue Links

          Activity

          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open Resolved Resolved
          1d 20h 40m 1 Brock Noland 27/Jul/13 18:24
          Brock Noland made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Duplicate [ 3 ]
          Brock Noland made changes -
          Field Original Value New Value
          Link This issue is superceded by HIVE-4920 [ HIVE-4920 ]
          Hide
          Brock Noland added a comment -

          I implemented a fix for this by rsyncing master -> slave (thread 0) once and then executing rsync on the slave (thread 0) -> slave (thread 1 and 2) and achieved significant speedup. I will post a patch soon.

          Show
          Brock Noland added a comment - I implemented a fix for this by rsyncing master -> slave (thread 0) once and then executing rsync on the slave (thread 0) -> slave (thread 1 and 2) and achieved significant speedup. I will post a patch soon.
          Brock Noland created issue -

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development