Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1397

Resource allocation should be fine grained.

    Details

      Description

      See the comment:

      https://issues.apache.org/jira/browse/TAJO-540?focusedCommentId=14359478&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14359478

      From the discussion in TAJO-540

      In general, query (or job) scheduler aims at the maximum resource utilization. For multi-tenancy, we also need to consider the fairness for multiple users (or queries). BTW, the maximum resource utilization and fairness are usually conflict to each other in many cases. To mitigate this problem, many scheduler seems to use preemption approach.

      In this point, our resource and scheduler system has the following problems:

      • A query exclusively uses allocated resources at the first time until the query is completed or failed.
      • There is no mechanism to deallocate resources during query processing.
      • Preempt is also not allowed.

      To achieve the multi tenancy, we should change our resource circulation. Especially, resource allocation must be fine grained instead of per query.
      So, I'll create a jira issue to change the resource circulation. We have to do this issue firstly in my opinion. If we achieve this, implementing multi-tenant scheduler would be much easier than now. It would be a good starting point of this issue.

      1. old_resource_circuit.png
        226 kB
        Hyunsik Choi
      2. ResoruceSequence.jpg
        37 kB
        Jinho Kim
      3. resource_circuit.png
        228 kB
        Hyunsik Choi
      4. TAJO-1397_2.patch
        758 kB
        Jinho Kim
      5. TAJO-1397_3.patch
        818 kB
        Jinho Kim
      6. TAJO-1397_4.patch
        818 kB
        Jinho Kim
      7. TAJO-1397.patch
        683 kB
        Jinho Kim

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #755 (See https://builds.apache.org/job/Tajo-master-build/755/)
          close related issues by TAJO-1397 (jhkim: rev 4bfd693acb1a294c6d52fa28e7c61b77f0a2034f)

          • dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #755 (See https://builds.apache.org/job/Tajo-master-build/755/ ) close related issues by TAJO-1397 (jhkim: rev 4bfd693acb1a294c6d52fa28e7c61b77f0a2034f) dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-CODEGEN-build #395 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/395/)
          close related issues by TAJO-1397 (jhkim: rev 4bfd693acb1a294c6d52fa28e7c61b77f0a2034f)

          • dev-support/findbugs-exclude.xml
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #395 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/395/ ) close related issues by TAJO-1397 (jhkim: rev 4bfd693acb1a294c6d52fa28e7c61b77f0a2034f) dev-support/findbugs-exclude.xml
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #754 (See https://builds.apache.org/job/Tajo-master-build/754/)
          close related issues by TAJO-1397 (jhkim: rev a06b5e56490116e345afc737de46bf1a4ed256a9)

          • BUILDING
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #754 (See https://builds.apache.org/job/Tajo-master-build/754/ ) close related issues by TAJO-1397 (jhkim: rev a06b5e56490116e345afc737de46bf1a4ed256a9) BUILDING
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-CODEGEN-build #394 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/394/)
          close related issues by TAJO-1397 (jhkim: rev a06b5e56490116e345afc737de46bf1a4ed256a9)

          • BUILDING
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #394 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/394/ ) close related issues by TAJO-1397 (jhkim: rev a06b5e56490116e345afc737de46bf1a4ed256a9) BUILDING
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #753 (See https://builds.apache.org/job/Tajo-master-build/753/)
          TAJO-1397: Resource allocation should be fine grained. (jinho) (jhkim: rev 5a02873d5f0bd8fedd8527808bd3d4ecbe7d8af3)

          • tajo-core/src/main/proto/ResourceProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStartEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueState.java
          • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerLauncher.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceManager.java
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-core/src/main/resources/webapps/admin/cluster.jsp
          • tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskStartEvent.java
          • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
          • tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
          • tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeLivelinessMonitor.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryReader.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java
          • tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerEvent.java
          • tajo-core/src/main/proto/QueryCoordinatorProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java
          • tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java
          • tajo-common/src/main/java/org/apache/tajo/util/StringUtils.java
          • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequestImpl.java
          • tajo-core/src/main/resources/webapps/worker/taskdetail.jsp
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/builder/DistinctGroupbyBuilder.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEventType.java
          • tajo-project/pom.xml
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryCoordinatorService.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskCompletionEvent.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/AbstractTaskScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterManagerService.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestQueryState.java
          • tajo-core/src/main/resources/webapps/worker/queryplan.jsp
          • tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStartEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEventType.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeState.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QuerySchedulingInfo.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/enforce/Enforcer.java
          • tajo-core/src/main/java/org/apache/tajo/master/cluster/WorkerConnectionInfo.java
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java
          • tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeResourceManager.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockTaskManager.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskExecutor.java
          • tajo-core/src/main/resources/webapps/worker/querytasks.jsp
          • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerEvent.java
          • tajo-core/src/main/resources/webapps/admin/querydetail.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEventType.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerReconnectEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerLivelinessMonitor.java
          • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocationEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
          • tajo-core/src/main/java/org/apache/tajo/worker/FetchImpl.java
          • tajo-core/src/main/resources/webapps/worker/querydetail.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageContainerAllocationEvent.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SchedulingAlgorithms.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerState.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockWorkerContext.java
          • tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java
          • tajo-core/src/main/proto/ContainerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
          • tajo-core/src/main/proto/QueryMasterProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeReconnectEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEvent.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskContainer.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java
          • tajo-core/src/test/java/org/apache/tajo/util/metrics/TestSystemMetrics.java
          • tajo-core/src/main/resources/webapps/admin/querytasks.jsp
          • tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java
          • tajo-core/src/main/resources/webapps/admin/task.jsp
          • tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoAdmin.java
          • tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp
          • tajo-core/src/main/java/org/apache/tajo/worker/LegacyTaskImpl.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java
          • tajo-docs/src/main/sphinx/configuration/worker_configuration.rst
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/ClusterResource.java
          • tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageShuffleReportEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/QueryStopEvent.java
          • .travis.yml
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerStatusEvent.java
          • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageEventType.java
          • tajo-core/src/main/resources/webapps/worker/task.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestTaskManager.java
          • tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestSimpleScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/QMResourceAllocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/GrouppedContainerAllocatorEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockExecutionBlock.java
          • tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerIdPBImpl.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/src/main/java/org/apache/tajo/session/Session.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestTaskExecutor.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockErrorEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/TajoResourceScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEvent.java
          • tajo-core/src/main/resources/webapps/admin/query.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ScanExec.java
          • tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestFifoScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStopEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskManagerEvent.java
          • tajo-core/src/main/resources/webapps/worker/taskhistory.jsp
          • tajo-core/pom.xml
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptStatusUpdateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskExecutorEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskFatalErrorEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockTaskExecutor.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-core/src/main/java/org/apache/tajo/metrics/WorkerResourceMetricsGaugeSet.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java
          • tajo-core/src/main/resources/webapps/admin/index.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
          • tajo-core/src/main/resources/webapps/worker/tasks.jsp
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocatorEventType.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java
          • tajo-core/src/main/resources/webapps/worker/index.jsp
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/ws/rs/responses/WorkerResponse.java
          • tajo-core/src/main/resources/webapps/admin/query_executor.jsp
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStopEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueInfo.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/ResourceReserveSchedulerEvent.java
          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatusEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleFifoScheduler.java
          • tajo-client/src/main/proto/ClientProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #753 (See https://builds.apache.org/job/Tajo-master-build/753/ ) TAJO-1397 : Resource allocation should be fine grained. (jinho) (jhkim: rev 5a02873d5f0bd8fedd8527808bd3d4ecbe7d8af3) tajo-core/src/main/proto/ResourceProtos.proto tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStartEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueState.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerLauncher.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceManager.java tajo-plan/src/main/proto/Plan.proto tajo-core/src/main/resources/webapps/admin/cluster.jsp tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskStartEvent.java tajo-core/src/main/proto/ResourceTrackerProtocol.proto tajo-common/src/main/java/org/apache/tajo/util/TUtil.java tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeLivelinessMonitor.java tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/util/history/HistoryReader.java tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerEvent.java tajo-core/src/main/proto/QueryCoordinatorProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java tajo-common/src/main/java/org/apache/tajo/util/StringUtils.java tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequestImpl.java tajo-core/src/main/resources/webapps/worker/taskdetail.jsp tajo-core/src/main/java/org/apache/tajo/engine/planner/global/builder/DistinctGroupbyBuilder.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEventType.java tajo-project/pom.xml tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/src/main/java/org/apache/tajo/master/QueryCoordinatorService.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskCompletionEvent.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/querymaster/AbstractTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterManagerService.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestQueryState.java tajo-core/src/main/resources/webapps/worker/queryplan.jsp tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStartEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEventType.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeState.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/QuerySchedulingInfo.java tajo-core/src/main/java/org/apache/tajo/engine/planner/enforce/Enforcer.java tajo-core/src/main/java/org/apache/tajo/master/cluster/WorkerConnectionInfo.java tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeResourceManager.java tajo-core/src/test/java/org/apache/tajo/worker/MockTaskManager.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/src/main/java/org/apache/tajo/worker/TaskExecutor.java tajo-core/src/main/resources/webapps/worker/querytasks.jsp tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java tajo-core/src/main/java/org/apache/tajo/master/event/ContainerEvent.java tajo-core/src/main/resources/webapps/admin/querydetail.jsp tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEventType.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerReconnectEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerLivelinessMonitor.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocationEvent.java tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java tajo-core/src/main/java/org/apache/tajo/worker/FetchImpl.java tajo-core/src/main/resources/webapps/worker/querydetail.jsp tajo-core/src/main/java/org/apache/tajo/master/event/StageContainerAllocationEvent.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/SchedulingAlgorithms.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerState.java tajo-core/src/test/java/org/apache/tajo/worker/MockWorkerContext.java tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java tajo-core/src/main/proto/ContainerProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java tajo-core/src/main/proto/QueryMasterProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/rm/NodeReconnectEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEvent.java tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/java/org/apache/tajo/worker/TaskContainer.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleScheduler.java tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java CHANGES tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java tajo-core/src/test/java/org/apache/tajo/util/metrics/TestSystemMetrics.java tajo-core/src/main/resources/webapps/admin/querytasks.jsp tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java tajo-core/src/main/resources/webapps/admin/task.jsp tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoAdmin.java tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp tajo-core/src/main/java/org/apache/tajo/worker/LegacyTaskImpl.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java tajo-docs/src/main/sphinx/configuration/worker_configuration.rst tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/ClusterResource.java tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/master/event/StageShuffleReportEvent.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-core/src/main/java/org/apache/tajo/worker/event/QueryStopEvent.java .travis.yml tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerStatusEvent.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-core/src/main/java/org/apache/tajo/master/event/StageEventType.java tajo-core/src/main/resources/webapps/worker/task.jsp tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-core/src/test/java/org/apache/tajo/worker/TestTaskManager.java tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestSimpleScheduler.java tajo-core/src/main/java/org/apache/tajo/worker/event/QMResourceAllocateEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/GrouppedContainerAllocatorEvent.java tajo-core/src/test/java/org/apache/tajo/worker/MockExecutionBlock.java tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerIdPBImpl.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/src/main/java/org/apache/tajo/session/Session.java tajo-core/src/test/java/org/apache/tajo/worker/TestTaskExecutor.java tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockErrorEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/TajoResourceScheduler.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEvent.java tajo-core/src/main/resources/webapps/admin/query.jsp tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ScanExec.java tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestFifoScheduler.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStopEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskManagerEvent.java tajo-core/src/main/resources/webapps/worker/taskhistory.jsp tajo-core/pom.xml tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptStatusUpdateEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskExecutorEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskFatalErrorEvent.java tajo-core/src/test/java/org/apache/tajo/worker/MockTaskExecutor.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-core/src/main/java/org/apache/tajo/metrics/WorkerResourceMetricsGaugeSet.java tajo-core/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java tajo-core/src/main/resources/webapps/admin/index.jsp tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java tajo-core/src/main/resources/webapps/worker/tasks.jsp tajo-core/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java tajo-core/src/main/java/org/apache/tajo/worker/TaskManager.java tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocatorEventType.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java tajo-core/src/main/resources/webapps/worker/index.jsp tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/ws/rs/responses/WorkerResponse.java tajo-core/src/main/resources/webapps/admin/query_executor.jsp tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStopEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueInfo.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/ResourceReserveSchedulerEvent.java tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatusEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleFifoScheduler.java tajo-client/src/main/proto/ClientProtos.proto tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-CODEGEN-build #393 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/393/)
          TAJO-1397: Resource allocation should be fine grained. (jinho) (jhkim: rev 5a02873d5f0bd8fedd8527808bd3d4ecbe7d8af3)

          • tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java
          • tajo-core/src/main/proto/QueryCoordinatorProtocol.proto
          • tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • tajo-core/src/main/resources/webapps/admin/cluster.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/GrouppedContainerAllocatorEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/QMResourceAllocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleScheduler.java
          • tajo-core/src/main/proto/ContainerProtocol.proto
          • tajo-core/src/main/resources/webapps/admin/querytasks.jsp
          • tajo-common/src/main/java/org/apache/tajo/util/StringUtils.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/enforce/Enforcer.java
          • tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java
          • tajo-common/src/main/java/org/apache/tajo/util/TUtil.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestTaskExecutor.java
          • tajo-core/src/main/proto/QueryMasterProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/session/Session.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEvent.java
          • tajo-core/src/main/resources/webapps/worker/queryplan.jsp
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStopEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskManagerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskExecutorEvent.java
          • tajo-core/src/main/resources/webapps/admin/index.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockTaskManager.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStopEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockTaskExecutor.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskFatalErrorEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
          • tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerLivelinessMonitor.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerReconnectEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java
          • tajo-core/src/main/resources/webapps/admin/query.jsp
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java
          • tajo-core/src/test/java/org/apache/tajo/util/metrics/TestSystemMetrics.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskStartEvent.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequestImpl.java
          • tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEventType.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockExecutionBlock.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterManagerService.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java
          • .travis.yml
          • tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/ClusterResource.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QuerySchedulingInfo.java
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueInfo.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java
          • tajo-core/src/main/resources/webapps/admin/query_executor.jsp
          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptStatusUpdateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/TajoResourceScheduler.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockErrorEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SchedulingAlgorithms.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEvent.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestQueryState.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/worker/LegacyTaskImpl.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerIdPBImpl.java
          • tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestSimpleScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatusEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java
          • tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java
          • tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoAdmin.java
          • tajo-core/src/main/resources/webapps/worker/taskhistory.jsp
          • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-core/src/main/resources/webapps/worker/index.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageContainerAllocationEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEventType.java
          • tajo-core/src/main/resources/webapps/admin/task.jsp
          • tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java
          • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeReconnectEvent.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java
          • tajo-core/src/main/resources/webapps/worker/task.jsp
          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeState.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageEventType.java
          • tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStartEvent.java
          • tajo-core/src/main/proto/ResourceProtos.proto
          • tajo-core/src/main/resources/webapps/worker/querydetail.jsp
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/StageShuffleReportEvent.java
          • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/metrics/WorkerResourceMetricsGaugeSet.java
          • tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java
          • tajo-core/src/main/resources/webapps/worker/tasks.jsp
          • tajo-core/src/main/resources/webapps/worker/taskdetail.jsp
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ScanExec.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStartEvent.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/QueryStopEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskCompletionEvent.java
          • tajo-core/pom.xml
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
          • tajo-core/src/main/java/org/apache/tajo/util/history/HistoryReader.java
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocatorEventType.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEventType.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java
          • tajo-project/pom.xml
          • tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java
          • tajo-core/src/main/resources/webapps/admin/querydetail.jsp
          • tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskContainer.java
          • tajo-core/src/main/java/org/apache/tajo/ws/rs/responses/WorkerResponse.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockWorkerContext.java
          • tajo-docs/src/main/sphinx/configuration/worker_configuration.rst
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/src/main/java/org/apache/tajo/worker/FetchImpl.java
          • tajo-core/src/main/java/org/apache/tajo/master/QueryCoordinatorService.java
          • tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
          • tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp
          • tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerLauncher.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/ResourceReserveSchedulerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/NodeLivelinessMonitor.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerStatusEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestTaskManager.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
          • tajo-client/src/main/proto/ClientProtos.proto
          • tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocationEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerState.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/AbstractTaskScheduler.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java
          • tajo-core/src/main/java/org/apache/tajo/master/cluster/WorkerConnectionInfo.java
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskExecutor.java
          • tajo-core/src/main/resources/webapps/worker/querytasks.jsp
          • tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/builder/DistinctGroupbyBuilder.java
          • tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestFifoScheduler.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueState.java
          • tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java
          • tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleFifoScheduler.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #393 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/393/ ) TAJO-1397 : Resource allocation should be fine grained. (jinho) (jhkim: rev 5a02873d5f0bd8fedd8527808bd3d4ecbe7d8af3) tajo-core/src/main/java/org/apache/tajo/worker/TaskHistory.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java tajo-core/src/main/proto/QueryCoordinatorProtocol.proto tajo-core/src/test/java/org/apache/tajo/worker/TestHistory.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoRMContext.java tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/resources/webapps/admin/cluster.jsp tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainer.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestKillQuery.java tajo-core/src/main/java/org/apache/tajo/master/event/GrouppedContainerAllocatorEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/QMResourceAllocateEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleScheduler.java tajo-core/src/main/proto/ContainerProtocol.proto tajo-core/src/main/resources/webapps/admin/querytasks.jsp tajo-common/src/main/java/org/apache/tajo/util/StringUtils.java tajo-core/src/main/java/org/apache/tajo/engine/planner/enforce/Enforcer.java tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/AsyncRpcServer.java tajo-core/src/main/java/org/apache/tajo/master/event/ContainerEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerResourceManager.java tajo-common/src/main/java/org/apache/tajo/util/TUtil.java tajo-core/src/test/java/org/apache/tajo/worker/TestTaskExecutor.java tajo-core/src/main/proto/QueryMasterProtocol.proto tajo-core/src/main/java/org/apache/tajo/session/Session.java tajo-core/src/main/java/org/apache/tajo/master/event/LocalTaskEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEvent.java tajo-core/src/main/resources/webapps/worker/queryplan.jsp tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStopEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskManagerEvent.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskExecutorEvent.java tajo-core/src/main/resources/webapps/admin/index.jsp tajo-core/src/main/java/org/apache/tajo/master/LaunchTaskRunnersEvent.java tajo-core/src/main/java/org/apache/tajo/querymaster/Query.java tajo-core/src/main/java/org/apache/tajo/querymaster/Stage.java tajo-core/src/test/java/org/apache/tajo/worker/MockTaskManager.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStopEvent.java tajo-core/src/test/java/org/apache/tajo/worker/MockTaskExecutor.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskFatalErrorEvent.java tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java tajo-core/src/test/java/org/apache/tajo/master/rm/TestTajoResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerLivelinessMonitor.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/master/event/TaskRequestEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerReconnectEvent.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerGroupEvent.java tajo-core/src/main/resources/webapps/admin/query.jsp tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java tajo-core/src/test/java/org/apache/tajo/util/metrics/TestSystemMetrics.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerId.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskStartEvent.java tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequestImpl.java tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeEventType.java tajo-core/src/test/java/org/apache/tajo/worker/MockExecutionBlock.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestOuterJoinQuery.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerEvent.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterManagerService.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java .travis.yml tajo-core/src/main/java/org/apache/tajo/ws/rs/resources/ClusterResource.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/QuerySchedulingInfo.java CHANGES tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueInfo.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java tajo-core/src/main/java/org/apache/tajo/master/TajoContainerProxy.java tajo-core/src/main/resources/webapps/admin/query_executor.jsp tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-core/src/main/java/org/apache/tajo/worker/TaskManager.java tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptStatusUpdateEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/TajoResourceScheduler.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockErrorEvent.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/SchedulingAlgorithms.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMaster.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEvent.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestQueryState.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/util/history/HistoryWriter.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeResourceManager.java tajo-core/src/main/java/org/apache/tajo/worker/LegacyTaskImpl.java tajo-core/src/main/java/org/apache/tajo/master/event/QueryStartEvent.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainerIdPBImpl.java tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestSimpleScheduler.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatusEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerHistory.java tajo-core/src/main/java/org/apache/tajo/engine/parser/SQLAnalyzer.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoWorkerContainerId.java tajo-cli/src/main/java/org/apache/tajo/cli/tools/TajoAdmin.java tajo-core/src/main/resources/webapps/worker/taskhistory.jsp tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/src/main/resources/webapps/worker/index.jsp tajo-core/src/main/java/org/apache/tajo/master/event/StageContainerAllocationEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerEventType.java tajo-core/src/main/resources/webapps/admin/task.jsp tajo-core/src/test/java/org/apache/tajo/worker/TestFetcher.java tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerResourceManager.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeReconnectEvent.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java tajo-core/src/main/resources/webapps/worker/task.jsp tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeState.java tajo-core/src/main/java/org/apache/tajo/master/event/StageEventType.java tajo-core/src/main/java/org/apache/tajo/worker/ResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/worker/event/ExecutionBlockStartEvent.java tajo-core/src/main/proto/ResourceProtos.proto tajo-core/src/main/resources/webapps/worker/querydetail.jsp tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorkerManagerService.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoContainer.java tajo-core/src/main/java/org/apache/tajo/master/event/StageShuffleReportEvent.java tajo-core/src/main/proto/ResourceTrackerProtocol.proto tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java tajo-core/src/main/java/org/apache/tajo/metrics/WorkerResourceMetricsGaugeSet.java tajo-core/src/main/java/org/apache/tajo/util/JSPUtil.java tajo-core/src/main/resources/webapps/worker/tasks.jsp tajo-core/src/main/resources/webapps/worker/taskdetail.jsp tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ScanExec.java tajo-core/src/main/java/org/apache/tajo/worker/TajoResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptEventType.java tajo-core/src/main/java/org/apache/tajo/worker/event/TaskRunnerStartEvent.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java tajo-core/src/main/java/org/apache/tajo/worker/event/QueryStopEvent.java tajo-core/src/main/java/org/apache/tajo/master/event/TaskCompletionEvent.java tajo-core/pom.xml tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java tajo-core/src/main/java/org/apache/tajo/util/history/HistoryReader.java tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocatorEventType.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEventType.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/CatalogServer.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinOnPartitionedTables.java tajo-project/pom.xml tajo-core/src/main/java/org/apache/tajo/master/ContainerProxy.java tajo-core/src/main/resources/webapps/admin/querydetail.jsp tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptAssignedEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TaskContainer.java tajo-core/src/main/java/org/apache/tajo/ws/rs/responses/WorkerResponse.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceManager.java tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java tajo-core/src/test/java/org/apache/tajo/worker/MockWorkerContext.java tajo-docs/src/main/sphinx/configuration/worker_configuration.rst tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/querymaster/QueryMasterTask.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/src/main/java/org/apache/tajo/worker/FetchImpl.java tajo-core/src/main/java/org/apache/tajo/master/QueryCoordinatorService.java tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java tajo-core/src/main/resources/webapps/worker/taskcontainers.jsp tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java tajo-core/src/main/java/org/apache/tajo/master/TaskRunnerLauncher.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/ResourceReserveSchedulerEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/NodeLivelinessMonitor.java tajo-core/src/main/java/org/apache/tajo/querymaster/TaskAttempt.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerStatusEvent.java tajo-core/src/test/java/org/apache/tajo/worker/TestTaskManager.java tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java tajo-client/src/main/proto/ClientProtos.proto tajo-core/src/main/java/org/apache/tajo/master/event/ContainerAllocationEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/WorkerState.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java tajo-core/src/main/java/org/apache/tajo/worker/AbstractResourceAllocator.java tajo-core/src/main/java/org/apache/tajo/querymaster/AbstractTaskScheduler.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunnerManager.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java tajo-plan/src/main/java/org/apache/tajo/plan/util/PlannerUtil.java tajo-core/src/main/java/org/apache/tajo/worker/TaskRunner.java tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java tajo-core/src/main/java/org/apache/tajo/master/cluster/WorkerConnectionInfo.java tajo-plan/src/main/proto/Plan.proto tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/event/SchedulerEvent.java tajo-core/src/main/java/org/apache/tajo/worker/TaskExecutor.java tajo-core/src/main/resources/webapps/worker/querytasks.jsp tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/builder/DistinctGroupbyBuilder.java tajo-core/src/test/java/org/apache/tajo/master/scheduler/TestFifoScheduler.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/QueueState.java tajo-core/src/main/java/org/apache/tajo/master/container/TajoConverterUtils.java tajo-core/src/main/java/org/apache/tajo/master/scheduler/SimpleFifoScheduler.java
          Hide
          jhkim Jinho Kim added a comment -

          committed it to master branch
          Thanks!

          Show
          jhkim Jinho Kim added a comment - committed it to master branch Thanks!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/608

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/608
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-122786979

          +1 ship it!

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-122786979 +1 ship it!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-122786068

          LGTM. Here is my +1.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-122786068 LGTM. Here is my +1.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-122619034

          The latest patch looks good to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-122619034 The latest patch looks good to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-122522460

          @hyunsik
          Thank you for your review
          I've update the patch that reflects your comments

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-122522460 @hyunsik Thank you for your review I've update the patch that reflects your comments
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34889423

          — Diff: .travis.yml —
          @@ -1,4 +1,5 @@

          1. Licensed to the Apache Software Foundation (ASF) under one or more
            +
              • End diff –

          It seems to be a mistake enter.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34889423 — Diff: .travis.yml — @@ -1,4 +1,5 @@ Licensed to the Apache Software Foundation (ASF) under one or more + End diff – It seems to be a mistake enter.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34887805

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java —
          @@ -0,0 +1,293 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.master.rm;
          +
          +import io.netty.util.internal.PlatformDependent;
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.hadoop.yarn.event.EventHandler;
          +import org.apache.hadoop.yarn.state.*;
          +import org.apache.tajo.master.cluster.WorkerConnectionInfo;
          +import org.apache.tajo.resource.NodeResource;
          +import org.apache.tajo.resource.NodeResources;
          +import org.apache.tajo.util.TUtil;
          +
          +import java.util.EnumSet;
          +import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
          +import java.util.concurrent.atomic.AtomicLongFieldUpdater;
          +
          +/**
          + * It contains resource and various information for a node.
          + */
          +public class NodeStatus implements EventHandler<NodeEvent>, Comparable<NodeStatus> {
          + /** class logger */
          + private static final Log LOG = LogFactory.getLog(NodeStatus.class);
          +
          + /** context of

          {@link TajoResourceManager}

          */
          + private final TajoRMContext rmContext;
          +
          + /** last heartbeat time */
          + private volatile long lastHeartbeatTime;
          +
          + private volatile int numRunningTasks;
          +
          + private volatile int numRunningQueryMaster;
          +
          + private static AtomicLongFieldUpdater HEARTBEAT_TIME_UPDATER;
          + private static AtomicIntegerFieldUpdater RUNNING_TASK_UPDATER;
          + private static AtomicIntegerFieldUpdater RUNNING_QM_UPDATER;
          +
          + static {
          + HEARTBEAT_TIME_UPDATER = PlatformDependent.newAtomicLongFieldUpdater(NodeStatus.class, "lastHeartbeatTime");
          + if (HEARTBEAT_TIME_UPDATER == null)

          { + HEARTBEAT_TIME_UPDATER = AtomicLongFieldUpdater.newUpdater(NodeStatus.class, "lastHeartbeatTime"); + RUNNING_TASK_UPDATER = AtomicIntegerFieldUpdater.newUpdater(NodeStatus.class, "numRunningTasks"); + RUNNING_QM_UPDATER = AtomicIntegerFieldUpdater.newUpdater(NodeStatus.class, "numRunningQueryMaster"); + }

          else

          { + RUNNING_TASK_UPDATER = PlatformDependent.newAtomicIntegerFieldUpdater(NodeStatus.class, "numRunningTasks"); + RUNNING_QM_UPDATER = PlatformDependent.newAtomicIntegerFieldUpdater(NodeStatus.class, "numRunningQueryMaster"); + }

          + }
          +
          + /** Available resources on the node. */
          + private final NodeResource availableResource;
          +
          + /** Total resources on the node. */
          + private final NodeResource totalResourceCapability;
          +
          + /** Node connection information */
          + private WorkerConnectionInfo connectionInfo;
          +
          + private static final ReconnectNodeTransition RECONNECT_NODE_TRANSITION = new ReconnectNodeTransition();
          + private static final StatusUpdateTransition STATUS_UPDATE_TRANSITION = new StatusUpdateTransition();
          +
          + private static final StateMachineFactory<NodeStatus,
          + NodeState,
          + NodeEventType,
          + NodeEvent> stateMachineFactory
          + = new StateMachineFactory<NodeStatus,
          + NodeState,
          + NodeEventType,
          + NodeEvent>(NodeState.NEW)
          +
          + // Transition from NEW
          + .addTransition(NodeState.NEW, NodeState.RUNNING,
          + NodeEventType.STARTED,
          + new AddNodeTransition())
          +
          + // Transition from RUNNING
          + .addTransition(NodeState.RUNNING, EnumSet.of(NodeState.RUNNING, NodeState.UNHEALTHY),
          + NodeEventType.STATE_UPDATE,
          + STATUS_UPDATE_TRANSITION)
          + .addTransition(NodeState.RUNNING, NodeState.LOST,
          + NodeEventType.EXPIRE,
          + new DeactivateNodeTransition(NodeState.LOST))
          + .addTransition(NodeState.RUNNING, NodeState.RUNNING,
          + NodeEventType.RECONNECTED,
          + RECONNECT_NODE_TRANSITION)
          +
          + // Transitions from UNHEALTHY state
          + .addTransition(NodeState.UNHEALTHY, EnumSet.of(NodeState.RUNNING, NodeState.UNHEALTHY),
          + NodeEventType.STATE_UPDATE,
          + STATUS_UPDATE_TRANSITION)
          + .addTransition(NodeState.UNHEALTHY, NodeState.LOST,
          + NodeEventType.EXPIRE,
          + new DeactivateNodeTransition(NodeState.LOST))
          + .addTransition(NodeState.UNHEALTHY, NodeState.UNHEALTHY,
          + NodeEventType.RECONNECTED,
          + RECONNECT_NODE_TRANSITION);
          +
          + private final StateMachine<NodeState, NodeEventType, NodeEvent> stateMachine =
          + stateMachineFactory.make(this, NodeState.NEW);
          +
          + public NodeStatus(TajoRMContext rmContext, NodeResource totalResourceCapability, WorkerConnectionInfo connectionInfo)

          { + this.rmContext = rmContext; + + this.connectionInfo = connectionInfo; + this.lastHeartbeatTime = System.currentTimeMillis(); + this.totalResourceCapability = totalResourceCapability; + this.availableResource = NodeResources.clone(totalResourceCapability); + }

          +
          + public int getWorkerId()

          { + return connectionInfo.getId(); + }

          +
          + public WorkerConnectionInfo getConnectionInfo()

          { + return connectionInfo; + }

          +
          + public void setLastHeartbeatTime(long lastheartbeatReportTime)

          { + HEARTBEAT_TIME_UPDATER.lazySet(this, lastheartbeatReportTime); + }

          +
          + public void setNumRunningQueryMaster(int numRunningQueryMaster)

          { + RUNNING_QM_UPDATER.lazySet(this, numRunningQueryMaster); + }

          +
          + public int getNumRunningQueryMaster()

          { + return numRunningQueryMaster; + }

          +
          + public void setNumRunningTasks(int numRunningTasks) {
          + RUNNING_TASK_UPDATER.lazySet(this, numRunningTasks);
          — End diff –

          ``RUNNING_TASK_UPDATE`` contains the number of running tasks. But, the getter method ``getNumRunningTasks()`` uses just the member variable ``numrunningTasks``. It seems to be invalid.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34887805 — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/NodeStatus.java — @@ -0,0 +1,293 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.master.rm; + +import io.netty.util.internal.PlatformDependent; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.hadoop.yarn.event.EventHandler; +import org.apache.hadoop.yarn.state.*; +import org.apache.tajo.master.cluster.WorkerConnectionInfo; +import org.apache.tajo.resource.NodeResource; +import org.apache.tajo.resource.NodeResources; +import org.apache.tajo.util.TUtil; + +import java.util.EnumSet; +import java.util.concurrent.atomic.AtomicIntegerFieldUpdater; +import java.util.concurrent.atomic.AtomicLongFieldUpdater; + +/** + * It contains resource and various information for a node. + */ +public class NodeStatus implements EventHandler<NodeEvent>, Comparable<NodeStatus> { + /** class logger */ + private static final Log LOG = LogFactory.getLog(NodeStatus.class); + + /** context of {@link TajoResourceManager} */ + private final TajoRMContext rmContext; + + /** last heartbeat time */ + private volatile long lastHeartbeatTime; + + private volatile int numRunningTasks; + + private volatile int numRunningQueryMaster; + + private static AtomicLongFieldUpdater HEARTBEAT_TIME_UPDATER; + private static AtomicIntegerFieldUpdater RUNNING_TASK_UPDATER; + private static AtomicIntegerFieldUpdater RUNNING_QM_UPDATER; + + static { + HEARTBEAT_TIME_UPDATER = PlatformDependent.newAtomicLongFieldUpdater(NodeStatus.class, "lastHeartbeatTime"); + if (HEARTBEAT_TIME_UPDATER == null) { + HEARTBEAT_TIME_UPDATER = AtomicLongFieldUpdater.newUpdater(NodeStatus.class, "lastHeartbeatTime"); + RUNNING_TASK_UPDATER = AtomicIntegerFieldUpdater.newUpdater(NodeStatus.class, "numRunningTasks"); + RUNNING_QM_UPDATER = AtomicIntegerFieldUpdater.newUpdater(NodeStatus.class, "numRunningQueryMaster"); + } else { + RUNNING_TASK_UPDATER = PlatformDependent.newAtomicIntegerFieldUpdater(NodeStatus.class, "numRunningTasks"); + RUNNING_QM_UPDATER = PlatformDependent.newAtomicIntegerFieldUpdater(NodeStatus.class, "numRunningQueryMaster"); + } + } + + /** Available resources on the node. */ + private final NodeResource availableResource; + + /** Total resources on the node. */ + private final NodeResource totalResourceCapability; + + /** Node connection information */ + private WorkerConnectionInfo connectionInfo; + + private static final ReconnectNodeTransition RECONNECT_NODE_TRANSITION = new ReconnectNodeTransition(); + private static final StatusUpdateTransition STATUS_UPDATE_TRANSITION = new StatusUpdateTransition(); + + private static final StateMachineFactory<NodeStatus, + NodeState, + NodeEventType, + NodeEvent> stateMachineFactory + = new StateMachineFactory<NodeStatus, + NodeState, + NodeEventType, + NodeEvent>(NodeState.NEW) + + // Transition from NEW + .addTransition(NodeState.NEW, NodeState.RUNNING, + NodeEventType.STARTED, + new AddNodeTransition()) + + // Transition from RUNNING + .addTransition(NodeState.RUNNING, EnumSet.of(NodeState.RUNNING, NodeState.UNHEALTHY), + NodeEventType.STATE_UPDATE, + STATUS_UPDATE_TRANSITION) + .addTransition(NodeState.RUNNING, NodeState.LOST, + NodeEventType.EXPIRE, + new DeactivateNodeTransition(NodeState.LOST)) + .addTransition(NodeState.RUNNING, NodeState.RUNNING, + NodeEventType.RECONNECTED, + RECONNECT_NODE_TRANSITION) + + // Transitions from UNHEALTHY state + .addTransition(NodeState.UNHEALTHY, EnumSet.of(NodeState.RUNNING, NodeState.UNHEALTHY), + NodeEventType.STATE_UPDATE, + STATUS_UPDATE_TRANSITION) + .addTransition(NodeState.UNHEALTHY, NodeState.LOST, + NodeEventType.EXPIRE, + new DeactivateNodeTransition(NodeState.LOST)) + .addTransition(NodeState.UNHEALTHY, NodeState.UNHEALTHY, + NodeEventType.RECONNECTED, + RECONNECT_NODE_TRANSITION); + + private final StateMachine<NodeState, NodeEventType, NodeEvent> stateMachine = + stateMachineFactory.make(this, NodeState.NEW); + + public NodeStatus(TajoRMContext rmContext, NodeResource totalResourceCapability, WorkerConnectionInfo connectionInfo) { + this.rmContext = rmContext; + + this.connectionInfo = connectionInfo; + this.lastHeartbeatTime = System.currentTimeMillis(); + this.totalResourceCapability = totalResourceCapability; + this.availableResource = NodeResources.clone(totalResourceCapability); + } + + public int getWorkerId() { + return connectionInfo.getId(); + } + + public WorkerConnectionInfo getConnectionInfo() { + return connectionInfo; + } + + public void setLastHeartbeatTime(long lastheartbeatReportTime) { + HEARTBEAT_TIME_UPDATER.lazySet(this, lastheartbeatReportTime); + } + + public void setNumRunningQueryMaster(int numRunningQueryMaster) { + RUNNING_QM_UPDATER.lazySet(this, numRunningQueryMaster); + } + + public int getNumRunningQueryMaster() { + return numRunningQueryMaster; + } + + public void setNumRunningTasks(int numRunningTasks) { + RUNNING_TASK_UPDATER.lazySet(this, numRunningTasks); — End diff – ``RUNNING_TASK_UPDATE`` contains the number of running tasks. But, the getter method ``getNumRunningTasks()`` uses just the member variable ``numrunningTasks``. It seems to be invalid.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34887204

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java —
          @@ -44,30 +43,19 @@ public TaskAttemptScheduleContext getContext() {
          }

          public static class TaskAttemptScheduleContext {

          • private TajoContainerId containerId;
            private String host;
          • private RpcCallback<TajoWorkerProtocol.TaskRequestProto> callback;
            + private RpcCallback<TaskRequestProto> callback;

          public TaskAttemptScheduleContext() {

          }

          • public TaskAttemptScheduleContext(TajoContainerId containerId,
          • String host,
          • RpcCallback<TajoWorkerProtocol.TaskRequestProto> callback) {
          • this.containerId = containerId;
            + public TaskAttemptScheduleContext(String host,
              • End diff –

          Unused constructor

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34887204 — Diff: tajo-core/src/main/java/org/apache/tajo/master/event/TaskAttemptToSchedulerEvent.java — @@ -44,30 +43,19 @@ public TaskAttemptScheduleContext getContext() { } public static class TaskAttemptScheduleContext { private TajoContainerId containerId; private String host; private RpcCallback<TajoWorkerProtocol.TaskRequestProto> callback; + private RpcCallback<TaskRequestProto> callback; public TaskAttemptScheduleContext() { } public TaskAttemptScheduleContext(TajoContainerId containerId, String host, RpcCallback<TajoWorkerProtocol.TaskRequestProto> callback) { this.containerId = containerId; + public TaskAttemptScheduleContext(String host, End diff – Unused constructor
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34887052

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java —
          @@ -21,32 +21,31 @@
          */
          package org.apache.tajo.engine.query;

          +import org.apache.tajo.ResourceProtos.TaskRequestProto;
          import org.apache.tajo.TaskAttemptId;
          import org.apache.tajo.catalog.proto.CatalogProtos;
          import org.apache.tajo.common.ProtoObject;
          import org.apache.tajo.conf.TajoConf;
          import org.apache.tajo.engine.planner.enforce.Enforcer;
          import org.apache.tajo.engine.planner.global.DataChannel;
          -import org.apache.tajo.ipc.TajoWorkerProtocol;
          import org.apache.tajo.plan.serder.PlanProto;
          import org.apache.tajo.worker.FetchImpl;

          import java.util.List;

          -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> {
          +public interface TaskRequest extends ProtoObject<TaskRequestProto> {

          • public TaskAttemptId getId();
          • public List<CatalogProtos.FragmentProto> getFragments();
          • public String getOutputTableId();
          • public boolean isClusteredOutput();
          • public PlanProto.LogicalNodeTree getPlan();
          • public boolean isInterQuery();
          • public void setInterQuery();
          • public void addFetch(String name, FetchImpl fetch);
          • public List<FetchImpl> getFetches();
          • public boolean shouldDie();
          • public void setShouldDie();
          • public QueryContext getQueryContext(TajoConf conf);
          • public DataChannel getDataChannel();
          • public Enforcer getEnforcer();
            + String getQueryMasterHostAndPort();
            + TaskAttemptId getId();
            + List<CatalogProtos.FragmentProto> getFragments();
            + String getOutputTableId();
            + boolean isClusteredOutput();
            + PlanProto.LogicalNodeTree getPlan();
            + boolean isInterQuery();
              • End diff –

          unused method

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34887052 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java — @@ -21,32 +21,31 @@ */ package org.apache.tajo.engine.query; +import org.apache.tajo.ResourceProtos.TaskRequestProto; import org.apache.tajo.TaskAttemptId; import org.apache.tajo.catalog.proto.CatalogProtos; import org.apache.tajo.common.ProtoObject; import org.apache.tajo.conf.TajoConf; import org.apache.tajo.engine.planner.enforce.Enforcer; import org.apache.tajo.engine.planner.global.DataChannel; -import org.apache.tajo.ipc.TajoWorkerProtocol; import org.apache.tajo.plan.serder.PlanProto; import org.apache.tajo.worker.FetchImpl; import java.util.List; -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> { +public interface TaskRequest extends ProtoObject<TaskRequestProto> { public TaskAttemptId getId(); public List<CatalogProtos.FragmentProto> getFragments(); public String getOutputTableId(); public boolean isClusteredOutput(); public PlanProto.LogicalNodeTree getPlan(); public boolean isInterQuery(); public void setInterQuery(); public void addFetch(String name, FetchImpl fetch); public List<FetchImpl> getFetches(); public boolean shouldDie(); public void setShouldDie(); public QueryContext getQueryContext(TajoConf conf); public DataChannel getDataChannel(); public Enforcer getEnforcer(); + String getQueryMasterHostAndPort(); + TaskAttemptId getId(); + List<CatalogProtos.FragmentProto> getFragments(); + String getOutputTableId(); + boolean isClusteredOutput(); + PlanProto.LogicalNodeTree getPlan(); + boolean isInterQuery(); End diff – unused method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34887031

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java —
          @@ -21,32 +21,31 @@
          */
          package org.apache.tajo.engine.query;

          +import org.apache.tajo.ResourceProtos.TaskRequestProto;
          import org.apache.tajo.TaskAttemptId;
          import org.apache.tajo.catalog.proto.CatalogProtos;
          import org.apache.tajo.common.ProtoObject;
          import org.apache.tajo.conf.TajoConf;
          import org.apache.tajo.engine.planner.enforce.Enforcer;
          import org.apache.tajo.engine.planner.global.DataChannel;
          -import org.apache.tajo.ipc.TajoWorkerProtocol;
          import org.apache.tajo.plan.serder.PlanProto;
          import org.apache.tajo.worker.FetchImpl;

          import java.util.List;

          -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> {
          +public interface TaskRequest extends ProtoObject<TaskRequestProto> {

          • public TaskAttemptId getId();
          • public List<CatalogProtos.FragmentProto> getFragments();
          • public String getOutputTableId();
          • public boolean isClusteredOutput();
          • public PlanProto.LogicalNodeTree getPlan();
          • public boolean isInterQuery();
          • public void setInterQuery();
          • public void addFetch(String name, FetchImpl fetch);
          • public List<FetchImpl> getFetches();
          • public boolean shouldDie();
          • public void setShouldDie();
          • public QueryContext getQueryContext(TajoConf conf);
          • public DataChannel getDataChannel();
          • public Enforcer getEnforcer();
            + String getQueryMasterHostAndPort();
            + TaskAttemptId getId();
            + List<CatalogProtos.FragmentProto> getFragments();
            + String getOutputTableId();
            + boolean isClusteredOutput();
              • End diff –

          unused method

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34887031 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java — @@ -21,32 +21,31 @@ */ package org.apache.tajo.engine.query; +import org.apache.tajo.ResourceProtos.TaskRequestProto; import org.apache.tajo.TaskAttemptId; import org.apache.tajo.catalog.proto.CatalogProtos; import org.apache.tajo.common.ProtoObject; import org.apache.tajo.conf.TajoConf; import org.apache.tajo.engine.planner.enforce.Enforcer; import org.apache.tajo.engine.planner.global.DataChannel; -import org.apache.tajo.ipc.TajoWorkerProtocol; import org.apache.tajo.plan.serder.PlanProto; import org.apache.tajo.worker.FetchImpl; import java.util.List; -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> { +public interface TaskRequest extends ProtoObject<TaskRequestProto> { public TaskAttemptId getId(); public List<CatalogProtos.FragmentProto> getFragments(); public String getOutputTableId(); public boolean isClusteredOutput(); public PlanProto.LogicalNodeTree getPlan(); public boolean isInterQuery(); public void setInterQuery(); public void addFetch(String name, FetchImpl fetch); public List<FetchImpl> getFetches(); public boolean shouldDie(); public void setShouldDie(); public QueryContext getQueryContext(TajoConf conf); public DataChannel getDataChannel(); public Enforcer getEnforcer(); + String getQueryMasterHostAndPort(); + TaskAttemptId getId(); + List<CatalogProtos.FragmentProto> getFragments(); + String getOutputTableId(); + boolean isClusteredOutput(); End diff – unused method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34887013

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java —
          @@ -21,32 +21,31 @@
          */
          package org.apache.tajo.engine.query;

          +import org.apache.tajo.ResourceProtos.TaskRequestProto;
          import org.apache.tajo.TaskAttemptId;
          import org.apache.tajo.catalog.proto.CatalogProtos;
          import org.apache.tajo.common.ProtoObject;
          import org.apache.tajo.conf.TajoConf;
          import org.apache.tajo.engine.planner.enforce.Enforcer;
          import org.apache.tajo.engine.planner.global.DataChannel;
          -import org.apache.tajo.ipc.TajoWorkerProtocol;
          import org.apache.tajo.plan.serder.PlanProto;
          import org.apache.tajo.worker.FetchImpl;

          import java.util.List;

          -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> {
          +public interface TaskRequest extends ProtoObject<TaskRequestProto> {

          • public TaskAttemptId getId();
          • public List<CatalogProtos.FragmentProto> getFragments();
          • public String getOutputTableId();
          • public boolean isClusteredOutput();
          • public PlanProto.LogicalNodeTree getPlan();
          • public boolean isInterQuery();
          • public void setInterQuery();
          • public void addFetch(String name, FetchImpl fetch);
          • public List<FetchImpl> getFetches();
          • public boolean shouldDie();
          • public void setShouldDie();
          • public QueryContext getQueryContext(TajoConf conf);
          • public DataChannel getDataChannel();
          • public Enforcer getEnforcer();
            + String getQueryMasterHostAndPort();
            + TaskAttemptId getId();
            + List<CatalogProtos.FragmentProto> getFragments();
            + String getOutputTableId();
              • End diff –

          unused method

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34887013 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java — @@ -21,32 +21,31 @@ */ package org.apache.tajo.engine.query; +import org.apache.tajo.ResourceProtos.TaskRequestProto; import org.apache.tajo.TaskAttemptId; import org.apache.tajo.catalog.proto.CatalogProtos; import org.apache.tajo.common.ProtoObject; import org.apache.tajo.conf.TajoConf; import org.apache.tajo.engine.planner.enforce.Enforcer; import org.apache.tajo.engine.planner.global.DataChannel; -import org.apache.tajo.ipc.TajoWorkerProtocol; import org.apache.tajo.plan.serder.PlanProto; import org.apache.tajo.worker.FetchImpl; import java.util.List; -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> { +public interface TaskRequest extends ProtoObject<TaskRequestProto> { public TaskAttemptId getId(); public List<CatalogProtos.FragmentProto> getFragments(); public String getOutputTableId(); public boolean isClusteredOutput(); public PlanProto.LogicalNodeTree getPlan(); public boolean isInterQuery(); public void setInterQuery(); public void addFetch(String name, FetchImpl fetch); public List<FetchImpl> getFetches(); public boolean shouldDie(); public void setShouldDie(); public QueryContext getQueryContext(TajoConf conf); public DataChannel getDataChannel(); public Enforcer getEnforcer(); + String getQueryMasterHostAndPort(); + TaskAttemptId getId(); + List<CatalogProtos.FragmentProto> getFragments(); + String getOutputTableId(); End diff – unused method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34886998

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java —
          @@ -21,32 +21,31 @@
          */
          package org.apache.tajo.engine.query;

          +import org.apache.tajo.ResourceProtos.TaskRequestProto;
          import org.apache.tajo.TaskAttemptId;
          import org.apache.tajo.catalog.proto.CatalogProtos;
          import org.apache.tajo.common.ProtoObject;
          import org.apache.tajo.conf.TajoConf;
          import org.apache.tajo.engine.planner.enforce.Enforcer;
          import org.apache.tajo.engine.planner.global.DataChannel;
          -import org.apache.tajo.ipc.TajoWorkerProtocol;
          import org.apache.tajo.plan.serder.PlanProto;
          import org.apache.tajo.worker.FetchImpl;

          import java.util.List;

          -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> {
          +public interface TaskRequest extends ProtoObject<TaskRequestProto> {

          • public TaskAttemptId getId();
          • public List<CatalogProtos.FragmentProto> getFragments();
          • public String getOutputTableId();
          • public boolean isClusteredOutput();
          • public PlanProto.LogicalNodeTree getPlan();
          • public boolean isInterQuery();
          • public void setInterQuery();
          • public void addFetch(String name, FetchImpl fetch);
          • public List<FetchImpl> getFetches();
          • public boolean shouldDie();
          • public void setShouldDie();
          • public QueryContext getQueryContext(TajoConf conf);
          • public DataChannel getDataChannel();
          • public Enforcer getEnforcer();
            + String getQueryMasterHostAndPort();
              • End diff –

          unused method

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34886998 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/query/TaskRequest.java — @@ -21,32 +21,31 @@ */ package org.apache.tajo.engine.query; +import org.apache.tajo.ResourceProtos.TaskRequestProto; import org.apache.tajo.TaskAttemptId; import org.apache.tajo.catalog.proto.CatalogProtos; import org.apache.tajo.common.ProtoObject; import org.apache.tajo.conf.TajoConf; import org.apache.tajo.engine.planner.enforce.Enforcer; import org.apache.tajo.engine.planner.global.DataChannel; -import org.apache.tajo.ipc.TajoWorkerProtocol; import org.apache.tajo.plan.serder.PlanProto; import org.apache.tajo.worker.FetchImpl; import java.util.List; -public interface TaskRequest extends ProtoObject<TajoWorkerProtocol.TaskRequestProto> { +public interface TaskRequest extends ProtoObject<TaskRequestProto> { public TaskAttemptId getId(); public List<CatalogProtos.FragmentProto> getFragments(); public String getOutputTableId(); public boolean isClusteredOutput(); public PlanProto.LogicalNodeTree getPlan(); public boolean isInterQuery(); public void setInterQuery(); public void addFetch(String name, FetchImpl fetch); public List<FetchImpl> getFetches(); public boolean shouldDie(); public void setShouldDie(); public QueryContext getQueryContext(TajoConf conf); public DataChannel getDataChannel(); public Enforcer getEnforcer(); + String getQueryMasterHostAndPort(); End diff – unused method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34886637

          — Diff: tajo-common/src/main/java/org/apache/tajo/util/TUtil.java —
          @@ -19,6 +19,7 @@
          package org.apache.tajo.util;

          import com.google.common.base.Objects;
          +import org.apache.tajo.conf.TajoConf;
          — End diff –

          This is unused.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34886637 — Diff: tajo-common/src/main/java/org/apache/tajo/util/TUtil.java — @@ -19,6 +19,7 @@ package org.apache.tajo.util; import com.google.common.base.Objects; +import org.apache.tajo.conf.TajoConf; — End diff – This is unused.
          Hide
          jhkim Jinho Kim added a comment -

          fix jdk6 compilation failure

          Show
          jhkim Jinho Kim added a comment - fix jdk6 compilation failure
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-122219316

          I've rename *Worker* to *NodeStatus*

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-122219316 I've rename * Worker * to * NodeStatus *
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34868642

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java —
          @@ -34,17 +39,37 @@
          /** class logger */
          private static final Log LOG = LogFactory.getLog(Worker.class);
          — End diff –

          The worker class has a status internally. How about "NodeStatus" ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34868642 — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java — @@ -34,17 +39,37 @@ /** class logger */ private static final Log LOG = LogFactory.getLog(Worker.class); — End diff – The worker class has a status internally. How about "NodeStatus" ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34868312

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java —
          @@ -89,20 +99,29 @@ public void init(Configuration conf) {
          @Override
          public void start() {
          LOG.info("Start TaskScheduler");
          + isLeaf = stage.getMasterPlan().isLeaf(stage.getBlock());
          +
          + if (!isLeaf) {
          — End diff –

          It is my big mistake. Thank you!!

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34868312 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java — @@ -89,20 +99,29 @@ public void init(Configuration conf) { @Override public void start() { LOG.info("Start TaskScheduler"); + isLeaf = stage.getMasterPlan().isLeaf(stage.getBlock()); + + if (!isLeaf) { — End diff – It is my big mistake. Thank you!!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34868269

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java —
          @@ -251,63 +250,70 @@ public void handle(TaskSchedulerEvent event) {
          }
          }

          • @Override
          • public void handleTaskRequestEvent(TaskRequestEvent event) {
            -
          • taskRequests.handle(event);
          • int hosts = scheduledRequests.leafTaskHostMapping.size();
            + private Set<Integer> getWorkerIds(Collection<String> hosts){
            + Set<Integer> workerIds = Sets.newHashSet();
            + if(hosts.isEmpty()) return workerIds;
          • // if available cluster resource are large then tasks, the scheduler thread are working immediately.
          • if(remainingScheduledObjectNum() > 0 &&
          • (remainingScheduledObjectNum() <= hosts || hosts <= taskRequests.size())){
          • synchronized (schedulingThread){
          • schedulingThread.notifyAll();
            + for (WorkerConnectionInfo worker : stage.getContext().getWorkerMap().values())
            Unknown macro: { + if(hosts.contains(worker.getHost())){ + workerIds.add(worker.getId()); } }

            + return workerIds;
            }

          • @Override
          • public int remainingScheduledObjectNum() { - return scheduledObjectNum; - }
          • private class TaskRequests implements EventHandler<TaskRequestEvent> {
          • private final LinkedBlockingQueue<TaskRequestEvent> taskRequestQueue =
          • new LinkedBlockingQueue<TaskRequestEvent>();
            + protected LinkedList<TaskRequestEvent> createTaskRequest() throws Exception {
            + LinkedList<TaskRequestEvent> taskRequestEvents = new LinkedList<TaskRequestEvent>();
          • @Override
          • public void handle(TaskRequestEvent event) {
          • if(LOG.isDebugEnabled()) { - LOG.debug("TaskRequest: " + event.getContainerId() + "," + event.getExecutionBlockId()); - }

            + int requestContainerNum = Math.max(remainingScheduledObjectNum(), 1);

              • End diff –

          Nice finding! It is my mistake. I will remove the Math.max

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34868269 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java — @@ -251,63 +250,70 @@ public void handle(TaskSchedulerEvent event) { } } @Override public void handleTaskRequestEvent(TaskRequestEvent event) { - taskRequests.handle(event); int hosts = scheduledRequests.leafTaskHostMapping.size(); + private Set<Integer> getWorkerIds(Collection<String> hosts){ + Set<Integer> workerIds = Sets.newHashSet(); + if(hosts.isEmpty()) return workerIds; // if available cluster resource are large then tasks, the scheduler thread are working immediately. if(remainingScheduledObjectNum() > 0 && (remainingScheduledObjectNum() <= hosts || hosts <= taskRequests.size())){ synchronized (schedulingThread){ schedulingThread.notifyAll(); + for (WorkerConnectionInfo worker : stage.getContext().getWorkerMap().values()) Unknown macro: { + if(hosts.contains(worker.getHost())){ + workerIds.add(worker.getId()); } } + return workerIds; } @Override public int remainingScheduledObjectNum() { - return scheduledObjectNum; - } private class TaskRequests implements EventHandler<TaskRequestEvent> { private final LinkedBlockingQueue<TaskRequestEvent> taskRequestQueue = new LinkedBlockingQueue<TaskRequestEvent>(); + protected LinkedList<TaskRequestEvent> createTaskRequest() throws Exception { + LinkedList<TaskRequestEvent> taskRequestEvents = new LinkedList<TaskRequestEvent>(); @Override public void handle(TaskRequestEvent event) { if(LOG.isDebugEnabled()) { - LOG.debug("TaskRequest: " + event.getContainerId() + "," + event.getExecutionBlockId()); - } + int requestContainerNum = Math.max(remainingScheduledObjectNum(), 1); End diff – Nice finding! It is my mistake. I will remove the Math.max
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864885

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java —
          @@ -34,17 +39,37 @@
          /** class logger */
          private static final Log LOG = LogFactory.getLog(Worker.class);
          — End diff –

          The name of this class looks less intuitive. How about changing it to a more intuitive name such as LiveWorkerInfo? If you have a good idea, please talk to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864885 — Diff: tajo-core/src/main/java/org/apache/tajo/master/rm/Worker.java — @@ -34,17 +39,37 @@ /** class logger */ private static final Log LOG = LogFactory.getLog(Worker.class); — End diff – The name of this class looks less intuitive. How about changing it to a more intuitive name such as LiveWorkerInfo? If you have a good idea, please talk to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864573

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java —
          @@ -251,63 +250,70 @@ public void handle(TaskSchedulerEvent event) {
          }
          }

          • @Override
          • public void handleTaskRequestEvent(TaskRequestEvent event) {
            -
          • taskRequests.handle(event);
          • int hosts = scheduledRequests.leafTaskHostMapping.size();
            + private Set<Integer> getWorkerIds(Collection<String> hosts){
            + Set<Integer> workerIds = Sets.newHashSet();
            + if(hosts.isEmpty()) return workerIds;
          • // if available cluster resource are large then tasks, the scheduler thread are working immediately.
          • if(remainingScheduledObjectNum() > 0 &&
          • (remainingScheduledObjectNum() <= hosts || hosts <= taskRequests.size())){
          • synchronized (schedulingThread){
          • schedulingThread.notifyAll();
            + for (WorkerConnectionInfo worker : stage.getContext().getWorkerMap().values())
            Unknown macro: { + if(hosts.contains(worker.getHost())){ + workerIds.add(worker.getId()); } }

            + return workerIds;
            }

          • @Override
          • public int remainingScheduledObjectNum() { - return scheduledObjectNum; - }
          • private class TaskRequests implements EventHandler<TaskRequestEvent> {
          • private final LinkedBlockingQueue<TaskRequestEvent> taskRequestQueue =
          • new LinkedBlockingQueue<TaskRequestEvent>();
            + protected LinkedList<TaskRequestEvent> createTaskRequest() throws Exception {
            + LinkedList<TaskRequestEvent> taskRequestEvents = new LinkedList<TaskRequestEvent>();
          • @Override
          • public void handle(TaskRequestEvent event) {
          • if(LOG.isDebugEnabled()) { - LOG.debug("TaskRequest: " + event.getContainerId() + "," + event.getExecutionBlockId()); - }

            + int requestContainerNum = Math.max(remainingScheduledObjectNum(), 1);

              • End diff –

          Would you give a explanation on the above line?
          Even though ```remainingScheduledObjectNum()``` is 0, requestContainerNum is 1. I wonder why at least one container is requested even though there is no remaining scheduled objects.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864573 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java — @@ -251,63 +250,70 @@ public void handle(TaskSchedulerEvent event) { } } @Override public void handleTaskRequestEvent(TaskRequestEvent event) { - taskRequests.handle(event); int hosts = scheduledRequests.leafTaskHostMapping.size(); + private Set<Integer> getWorkerIds(Collection<String> hosts){ + Set<Integer> workerIds = Sets.newHashSet(); + if(hosts.isEmpty()) return workerIds; // if available cluster resource are large then tasks, the scheduler thread are working immediately. if(remainingScheduledObjectNum() > 0 && (remainingScheduledObjectNum() <= hosts || hosts <= taskRequests.size())){ synchronized (schedulingThread){ schedulingThread.notifyAll(); + for (WorkerConnectionInfo worker : stage.getContext().getWorkerMap().values()) Unknown macro: { + if(hosts.contains(worker.getHost())){ + workerIds.add(worker.getId()); } } + return workerIds; } @Override public int remainingScheduledObjectNum() { - return scheduledObjectNum; - } private class TaskRequests implements EventHandler<TaskRequestEvent> { private final LinkedBlockingQueue<TaskRequestEvent> taskRequestQueue = new LinkedBlockingQueue<TaskRequestEvent>(); + protected LinkedList<TaskRequestEvent> createTaskRequest() throws Exception { + LinkedList<TaskRequestEvent> taskRequestEvents = new LinkedList<TaskRequestEvent>(); @Override public void handle(TaskRequestEvent event) { if(LOG.isDebugEnabled()) { - LOG.debug("TaskRequest: " + event.getContainerId() + "," + event.getExecutionBlockId()); - } + int requestContainerNum = Math.max(remainingScheduledObjectNum(), 1); End diff – Would you give a explanation on the above line? Even though ```remainingScheduledObjectNum()``` is 0, requestContainerNum is 1. I wonder why at least one container is requested even though there is no remaining scheduled objects.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864506

          — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java —
          @@ -89,20 +99,29 @@ public void init(Configuration conf) {
          @Override
          public void start() {
          LOG.info("Start TaskScheduler");
          + isLeaf = stage.getMasterPlan().isLeaf(stage.getBlock());
          +
          + if (!isLeaf) {
          — End diff –

          Please fix this condition.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864506 — Diff: tajo-core/src/main/java/org/apache/tajo/querymaster/DefaultTaskScheduler.java — @@ -89,20 +99,29 @@ public void init(Configuration conf) { @Override public void start() { LOG.info("Start TaskScheduler"); + isLeaf = stage.getMasterPlan().isLeaf(stage.getBlock()); + + if (!isLeaf) { — End diff – Please fix this condition.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864494

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java —
          @@ -0,0 +1,70 @@
          +/**
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo.master.scheduler;
          +
          +import org.apache.hadoop.service.AbstractService;
          +import org.apache.tajo.QueryId;
          +import org.apache.tajo.resource.NodeResource;
          +import org.apache.tajo.resource.NodeResources;
          +import org.apache.tajo.resource.ResourceCalculator;
          +
          +public abstract class AbstractQueryScheduler extends AbstractService implements TajoResourceScheduler {
          — End diff –

          I know that there are detailed comments at TajoResourceScheduler interface.
          So, please add a simple comment such as ```please refer to TajoResourceScheduler for detailed information```.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864494 — Diff: tajo-core/src/main/java/org/apache/tajo/master/scheduler/AbstractQueryScheduler.java — @@ -0,0 +1,70 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo.master.scheduler; + +import org.apache.hadoop.service.AbstractService; +import org.apache.tajo.QueryId; +import org.apache.tajo.resource.NodeResource; +import org.apache.tajo.resource.NodeResources; +import org.apache.tajo.resource.ResourceCalculator; + +public abstract class AbstractQueryScheduler extends AbstractService implements TajoResourceScheduler { — End diff – I know that there are detailed comments at TajoResourceScheduler interface. So, please add a simple comment such as ```please refer to TajoResourceScheduler for detailed information```.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864411

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java —
          @@ -178,35 +174,29 @@ public QueryInfo scheduleQuery(Session session, QueryContext queryContext, Strin
          QueryInProgress queryInProgress = new QueryInProgress(masterContext, session, queryContext, queryId, sql,
          jsonExpr, plan);

          • synchronized (submittedQueries) { - queryInProgress.getQueryInfo().setQueryMaster(""); - submittedQueries.put(queryInProgress.getQueryId(), queryInProgress); - }

            + queryInProgress.getQueryInfo().setQueryMaster("");

              • End diff –

          How about defining a constant to represent an empty query master host?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864411 — Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java — @@ -178,35 +174,29 @@ public QueryInfo scheduleQuery(Session session, QueryContext queryContext, Strin QueryInProgress queryInProgress = new QueryInProgress(masterContext, session, queryContext, queryId, sql, jsonExpr, plan); synchronized (submittedQueries) { - queryInProgress.getQueryInfo().setQueryMaster(""); - submittedQueries.put(queryInProgress.getQueryId(), queryInProgress); - } + queryInProgress.getQueryInfo().setQueryMaster(""); End diff – How about defining a constant to represent an empty query master host?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/608#discussion_r34864385

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java —
          @@ -53,9 +56,9 @@

          private LogicalRootNode plan;

          • private AtomicBoolean querySubmitted = new AtomicBoolean(false);
            + private volatile boolean querySubmitted = false;
          • private AtomicBoolean stopped = new AtomicBoolean(false);
            + private AtomicBoolean isStopped = new AtomicBoolean(false);
              • End diff –

          This variable also seems to be able to be defined as a volatile boolean instead of AtomicBoolean.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on a diff in the pull request: https://github.com/apache/tajo/pull/608#discussion_r34864385 — Diff: tajo-core/src/main/java/org/apache/tajo/master/QueryInProgress.java — @@ -53,9 +56,9 @@ private LogicalRootNode plan; private AtomicBoolean querySubmitted = new AtomicBoolean(false); + private volatile boolean querySubmitted = false; private AtomicBoolean stopped = new AtomicBoolean(false); + private AtomicBoolean isStopped = new AtomicBoolean(false); End diff – This variable also seems to be able to be defined as a volatile boolean instead of AtomicBoolean.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-121877409

          @jihoonson
          I've update the patch but startQueryJob is not condition method

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-121877409 @jihoonson I've update the patch but startQueryJob is not condition method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-121798206

          @jihoonson
          Thank you for your review.
          PARALLEL_QUERY_LIMIT is maximum running QM. I will change to MAXIMUM_RUNNING_QM_RATE
          I'm going to update the patch that reflect your comments

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-121798206 @jihoonson Thank you for your review. PARALLEL_QUERY_LIMIT is maximum running QM. I will change to MAXIMUM_RUNNING_QM_RATE I'm going to update the patch that reflect your comments
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-121633024

          @jinossy, really nice work!
          I'm still reviewing and not finished yet. Here, I left some comments first, and will finish soon.

          • StringUtils.java
          • The comment of isPureAscii() is incomplete.
          • QueryCoordinatorProtocol.proto
          • As you know, @hyunsik is working on refactoring our error propagation system. According to the guide at https://cwiki.apache.org/confluence/display/TAJO/RPC+Protocol+Design+Guide, please separate QueryCoordinatorProtocolService from this file.
          • Also, please add some comments for the methods of QueryCoordinatorProtocolService.
          • QueryInProgress.java
          • Please change the comments of allocateToQueryMaster() into the formal java doc format.
          • Please add some comments for submitToQueryMaster().
          • QueryManager.java
          • The name of startQueryJob() looks not intuitive. How about isQueryExecutable()?
          • TajoResourceManager.java
          • Please remove the unused parameter, ```systemConf```, from the constructor.
          • QuerySchedulingInfo.java
          • Please add some comments for the class and its variables.
          • SimpleScheduler.java
          • Please add some comments for the class.
          • Would you please explain what PARALLEL_QUERY_LIMIT means?
          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-121633024 @jinossy, really nice work! I'm still reviewing and not finished yet. Here, I left some comments first, and will finish soon. StringUtils.java The comment of isPureAscii() is incomplete. QueryCoordinatorProtocol.proto As you know, @hyunsik is working on refactoring our error propagation system. According to the guide at https://cwiki.apache.org/confluence/display/TAJO/RPC+Protocol+Design+Guide , please separate QueryCoordinatorProtocolService from this file. Also, please add some comments for the methods of QueryCoordinatorProtocolService. QueryInProgress.java Please change the comments of allocateToQueryMaster() into the formal java doc format. Please add some comments for submitToQueryMaster(). QueryManager.java The name of startQueryJob() looks not intuitive. How about isQueryExecutable()? TajoResourceManager.java Please remove the unused parameter, ```systemConf```, from the constructor. QuerySchedulingInfo.java Please add some comments for the class and its variables. SimpleScheduler.java Please add some comments for the class. Would you please explain what PARALLEL_QUERY_LIMIT means?
          Hide
          jhkim Jinho Kim added a comment -

          These issues does not cause in this patch

          Show
          jhkim Jinho Kim added a comment - These issues does not cause in this patch
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-121178442

          This patch is ready, please review this PR

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-121178442 This patch is ready, please review this PR
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/608#issuecomment-113912553

          This PR is not ready to review. please don't start review.
          I just test the Travis-CI

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/608#issuecomment-113912553 This PR is not ready to review. please don't start review. I just test the Travis-CI
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jinossy opened a pull request:

          https://github.com/apache/tajo/pull/608

          TAJO-1397

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jinossy/tajo TAJO-1397

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/608.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #608


          commit 6dd0c0a90b1c2aaa56e3f144ecc57c996715c481
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-01T08:58:08Z

          merge from TAJO-1615

          commit 91319ee3eed1c950551071eca10a6587f635a48e
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-08T07:57:23Z

          TAJO-1397

          commit aba1188da4452c3c641e48411d6dd4a24998d93f
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-08T07:57:39Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397

          commit 1bfa430fd65006b79bfcdc382b0b7aeb17bd9686
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-16T11:13:30Z

          TAJO-1397

          commit 13e71b99ea0136fe5ad7f2489af0883664ca6547
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-16T11:13:46Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397

          commit 133e3f2c04a435a6da9ae823be0fa2dd803db377
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-17T15:50:18Z

          TAJO-1397

          commit b347e04aa5372bfb34f0e283aead42f9ff904112
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-18T02:16:12Z

          TAJO-1397

          commit b397ff15eedf982de704fb38ec1abe9a0f69df69
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-18T07:16:28Z

          TAJO-1397

          commit 4fab8787591302dace885417f4487976d586642d
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-18T08:01:07Z

          remove unused codes

          commit b82916043dccb26dd4dcb20f5ae70c1c202d5327
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-18T08:38:46Z

          add debug message

          commit e5318b8030bbf1821aacbe0ddae45d94bc403afa
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-18T09:23:07Z

          add changable heartBeatInterval

          commit 7ec50d034f9cd1bbc2fc7cce3eda35324681fee8
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-19T03:40:06Z

          TAJO-1397

          commit a8424871c2fb9a6f7af700f198ebe2f332852a80
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-19T04:15:14Z

          fix wrong QM address

          commit 50b78e6d4e4f6f9464f149cf3452d51115d9c22e
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-19T04:28:30Z

          TAJO-1397

          commit a520494bd141da189074e1ea170efaadd3bbd3a0
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-19T07:33:35Z

          TAJO-1397

          commit 3983008808e3351ba723941095643feae458eb49
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-21T14:42:53Z

          remove StartExecutionBlock event

          commit 749cfa2bf742bd913d4faf57b5ba7a673ebabb5b
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-21T14:44:03Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397

          commit 07b7b91f8c3cb8614dcf4ecc6e19d6e790a220ec
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-21T14:47:44Z

          remove unused class

          commit ee0c34e59df250e5b88f91b2d8c2d6b231318bde
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-06-21T15:07:40Z

          add internal error handling in DefaultTaskScheduler


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/608 TAJO-1397 You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1397 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/608.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #608 commit 6dd0c0a90b1c2aaa56e3f144ecc57c996715c481 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-01T08:58:08Z merge from TAJO-1615 commit 91319ee3eed1c950551071eca10a6587f635a48e Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-08T07:57:23Z TAJO-1397 commit aba1188da4452c3c641e48411d6dd4a24998d93f Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-08T07:57:39Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397 commit 1bfa430fd65006b79bfcdc382b0b7aeb17bd9686 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-16T11:13:30Z TAJO-1397 commit 13e71b99ea0136fe5ad7f2489af0883664ca6547 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-16T11:13:46Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397 commit 133e3f2c04a435a6da9ae823be0fa2dd803db377 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-17T15:50:18Z TAJO-1397 commit b347e04aa5372bfb34f0e283aead42f9ff904112 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-18T02:16:12Z TAJO-1397 commit b397ff15eedf982de704fb38ec1abe9a0f69df69 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-18T07:16:28Z TAJO-1397 commit 4fab8787591302dace885417f4487976d586642d Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-18T08:01:07Z remove unused codes commit b82916043dccb26dd4dcb20f5ae70c1c202d5327 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-18T08:38:46Z add debug message commit e5318b8030bbf1821aacbe0ddae45d94bc403afa Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-18T09:23:07Z add changable heartBeatInterval commit 7ec50d034f9cd1bbc2fc7cce3eda35324681fee8 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-19T03:40:06Z TAJO-1397 commit a8424871c2fb9a6f7af700f198ebe2f332852a80 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-19T04:15:14Z fix wrong QM address commit 50b78e6d4e4f6f9464f149cf3452d51115d9c22e Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-19T04:28:30Z TAJO-1397 commit a520494bd141da189074e1ea170efaadd3bbd3a0 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-19T07:33:35Z TAJO-1397 commit 3983008808e3351ba723941095643feae458eb49 Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-21T14:42:53Z remove StartExecutionBlock event commit 749cfa2bf742bd913d4faf57b5ba7a673ebabb5b Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-21T14:44:03Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1397 commit 07b7b91f8c3cb8614dcf4ecc6e19d6e790a220ec Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-21T14:47:44Z remove unused class commit ee0c34e59df250e5b88f91b2d8c2d6b231318bde Author: Jinho Kim <jhkim@apache.org> Date: 2015-06-21T15:07:40Z add internal error handling in DefaultTaskScheduler
          Hide
          jhkim Jinho Kim added a comment -

          OK, I will start initial work

          Show
          jhkim Jinho Kim added a comment - OK, I will start initial work
          Hide
          hyunsik Hyunsik Choi added a comment -

          Please go ahead. You can also feel free to assign itself if you want. I don't have any free slot right now.

          Show
          hyunsik Hyunsik Choi added a comment - Please go ahead. You can also feel free to assign itself if you want. I don't have any free slot right now.
          Hide
          jhkim Jinho Kim added a comment - - edited

          If you don't start this issue yet, I will create a sub task for this.

          Show
          jhkim Jinho Kim added a comment - - edited If you don't start this issue yet, I will create a sub task for this.
          Hide
          jhkim Jinho Kim added a comment -

          Hyunsik Choi
          I've attach the sequence diagram from your proposal.
          I think If a node dynamically handle period of heartbeat, delay will not be a problem.

          Show
          jhkim Jinho Kim added a comment - Hyunsik Choi I've attach the sequence diagram from your proposal. I think If a node dynamically handle period of heartbeat, delay will not be a problem.
          Hide
          jhkim Jinho Kim added a comment -

          Hyunsik Choi
          Your proposal looks great to large cluster.
          I have a question. If a tasks is fast(50~ 1000 ms) finished, minimum scheduling costs will be heartbeat delay of node.
          Have you an idea?

          Show
          jhkim Jinho Kim added a comment - Hyunsik Choi Your proposal looks great to large cluster. I have a question. If a tasks is fast(50~ 1000 ms) finished, minimum scheduling costs will be heartbeat delay of node. Have you an idea?
          Hide
          hyunsik Hyunsik Choi added a comment -

          Here is my proposed resource circulation. See resource_circuit.png file.

          • Node sends a heartbeats with its resource and status periodically.
          • ResourceTracker maintains a global view of node resources.

          When a query master is launched, the query master requests necessary resource to ResourceTracker. This is repeatedly performed while a query is executing. The request will includes the following information:

          • query id
          • user
          • A set of resource requests, each of which will include
          • priority
          • the number of containers and resource capacity for containers
          • desired host names
          • type of task (leaf or intermediate)
          • ...

          When the query master receives the resource requests, it schedules tasks to nodes. In some cases where a node cannot accept the task schedule, the node can reject the request. It can be occur when the available resources that ResourceTracker and actual nodes keep are different in some cases.

          In the proposed resource circulation, the resource allocation will be performed in a task level. So, it will give even more flexibility to adjust the resource capacity per user or query during query processing.

          Show
          hyunsik Hyunsik Choi added a comment - Here is my proposed resource circulation. See resource_circuit.png file. Node sends a heartbeats with its resource and status periodically. ResourceTracker maintains a global view of node resources. When a query master is launched, the query master requests necessary resource to ResourceTracker. This is repeatedly performed while a query is executing. The request will includes the following information: query id user A set of resource requests, each of which will include priority the number of containers and resource capacity for containers desired host names type of task (leaf or intermediate) ... When the query master receives the resource requests, it schedules tasks to nodes. In some cases where a node cannot accept the task schedule, the node can reject the request. It can be occur when the available resources that ResourceTracker and actual nodes keep are different in some cases. In the proposed resource circulation, the resource allocation will be performed in a task level. So, it will give even more flexibility to adjust the resource capacity per user or query during query processing.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I'll post the figure of my proposed circulation and its description soon.

          Show
          hyunsik Hyunsik Choi added a comment - I'll post the figure of my proposed circulation and its description soon.
          Hide
          hyunsik Hyunsik Choi added a comment -

          This figure represents the current resource circulation.

          Show
          hyunsik Hyunsik Choi added a comment - This figure represents the current resource circulation.

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development