Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-19425 General usability improvements for Ptest
  3. HIVE-19571

Ability to run multiple pre-commit jobs on a ptest server

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Testing Infrastructure
    • Labels:
      None

      Description

      I've been taking a look at the Disk, Network, and CPU usage of the GCE instances we run ptest on, and it doesn't look like we are fully utilizing the machines. The resource usage is very up and down.

      During each ptest execution, there is a large chunk of time (~20 min) where its just the Jenkins job that is doing any work (checking out github repos, building code, figuring out test batches, etc.). During this time, the ptest nodes are mostly idle - the CPU and Disk I/O are almost zero.

      Even when ptest is running, I think some of resources are under-utilized. Network and disk resource spike at the beginning of the job, probably because ptest is distributing resources to each machine, each slave is downloading jars, etc. However, after that, when the actual tests run, there is almost 0 network activity (which makes sense since tests runs on a single node). For disk usage, there is activity, but not nearly as high as when the setup phase was occuring. CPU usage fluctuates between 40-80%.

        Attachments

        1. HIVE-17317.WIP.1.patch
          15 kB
          Sahil Takiar

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stakiar Sahil Takiar
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated: