Hive
  1. Hive
  2. HIVE-4936

PTest increase performance by parallelizing rsync

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Currently the bottom neck in PTest2 when adding hosts is the rsync to the slaves. Today if we have 3 "threads" which will execute on a slave we rsync the source from the master to the slave 3 times. This causes the process to be CPU bound on the master.

        Issue Links

          Activity

          Hide
          Brock Noland added a comment -

          I implemented a fix for this by rsyncing master -> slave (thread 0) once and then executing rsync on the slave (thread 0) -> slave (thread 1 and 2) and achieved significant speedup. I will post a patch soon.

          Show
          Brock Noland added a comment - I implemented a fix for this by rsyncing master -> slave (thread 0) once and then executing rsync on the slave (thread 0) -> slave (thread 1 and 2) and achieved significant speedup. I will post a patch soon.

            People

            • Assignee:
              Brock Noland
              Reporter:
              Brock Noland
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development