Uploaded image for project: 'Tajo (Retired)'
  1. Tajo (Retired)
  2. TAJO-1397

Resource allocation should be fine grained.

    XMLWordPrintableJSON

Details

    Description

      See the comment:

      https://issues.apache.org/jira/browse/TAJO-540?focusedCommentId=14359478&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359478

      From the discussion in TAJO-540

      In general, query (or job) scheduler aims at the maximum resource utilization. For multi-tenancy, we also need to consider the fairness for multiple users (or queries). BTW, the maximum resource utilization and fairness are usually conflict to each other in many cases. To mitigate this problem, many scheduler seems to use preemption approach.

      In this point, our resource and scheduler system has the following problems:

      • A query exclusively uses allocated resources at the first time until the query is completed or failed.
      • There is no mechanism to deallocate resources during query processing.
      • Preempt is also not allowed.

      To achieve the multi tenancy, we should change our resource circulation. Especially, resource allocation must be fine grained instead of per query.
      So, I'll create a jira issue to change the resource circulation. We have to do this issue firstly in my opinion. If we achieve this, implementing multi-tenant scheduler would be much easier than now. It would be a good starting point of this issue.

      Attachments

        1. old_resource_circuit.png
          226 kB
          Hyunsik Choi
        2. ResoruceSequence.jpg
          37 kB
          Jinho Kim
        3. resource_circuit.png
          228 kB
          Hyunsik Choi
        4. TAJO-1397_2.patch
          758 kB
          Jinho Kim
        5. TAJO-1397_3.patch
          818 kB
          Jinho Kim
        6. TAJO-1397_4.patch
          818 kB
          Jinho Kim
        7. TAJO-1397.patch
          683 kB
          Jinho Kim

        Issue Links

          contains

          Bug - A problem which impairs or prevents the functions of the product. TAJO-1146 TajoWorkerResources are occasionally not released after the query completion relating to semaphore condition.

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. TAJO-1399 TajoResourceAllocator might hang on network error

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. TAJO-1622 UniformRangePartition occasionally causes IllegalStateException

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. TAJO-1536 Fix minor issues in QueryMaster

          • Trivial - Cosmetic problem like misspelt words or misaligned text.
          • Resolved

          Improvement - An improvement or enhancement to an existing feature or task. TAJO-1551 Reuse allocated resources in next execution block if possible

          • Major - Major loss of function.
          • Resolved
          is required by

          New Feature - A new feature of the product, which has yet to be developed. TAJO-540 (Umbrella) Implement Tajo Query Scheduler

          • Major - Major loss of function.
          • In Progress

          Activity

            People

              jhkim Jinho Kim
              hyunsik Hyunsik Choi
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: