Details

      Description

      NodeResourceManager can allocate the request. if node is no more available resource, It will be reject the request.
      When a resource is deallocated, NodeStatusUpdater sends resource status repeatedly to ResourceTracker.

      NodeResourceManager

      • Allocate batch requests
      • Cancel part of batch

      NodeStatusUpdater

      • Send ping
      • Send available resource status
      • Send node status
      1. TAJO-1599_3.patch
        78 kB
        Jinho Kim
      2. TAJO-1599_2.patch
        77 kB
        Jinho Kim
      3. TAJO-1599.patch
        77 kB
        Jinho Kim

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/577#issuecomment-105546419

          I was add the new key of disk configuration because if I change the configuration, it affects the current resource.
          When I remove the legacy configuration, I will update the document.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/577#issuecomment-105546419 I was add the new key of disk configuration because if I change the configuration, it affects the current resource. When I remove the legacy configuration, I will update the document.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/577#issuecomment-105503486

          This patch requires updating documentation because it affects the resource configuration.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/577#issuecomment-105503486 This patch requires updating documentation because it affects the resource configuration.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #717 (See https://builds.apache.org/job/Tajo-master-build/717/)
          TAJO-1599: Implement NodeResourceManager and Status updater. (jinho) (jhkim: rev 25bd5cb44a03ee425b02e2bc2553f7d0f8affff5)

          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceManagerEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
          • CHANGES
          • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java
          • tajo-common/src/main/proto/tajo_protos.proto
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResources.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
          • tajo-core/src/main/java/org/apache/tajo/resource/ResourceCalculator.java
          • tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/DiskUtil.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java
          • tajo-core/src/test/java/org/apache/tajo/resource/TestResources.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeStatusEvent.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #717 (See https://builds.apache.org/job/Tajo-master-build/717/ ) TAJO-1599 : Implement NodeResourceManager and Status updater. (jinho) (jhkim: rev 25bd5cb44a03ee425b02e2bc2553f7d0f8affff5) tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceManagerEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java CHANGES tajo-core/src/main/proto/ResourceTrackerProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java tajo-common/src/main/proto/tajo_protos.proto tajo-core/src/main/java/org/apache/tajo/resource/NodeResources.java tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java tajo-core/src/main/java/org/apache/tajo/resource/ResourceCalculator.java tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/DiskUtil.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java tajo-core/src/test/java/org/apache/tajo/resource/TestResources.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/worker/event/NodeStatusEvent.java
          Hide
          hudson Hudson added a comment -

          ABORTED: Integrated in Tajo-master-CODEGEN-build #357 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/357/)
          TAJO-1599: Implement NodeResourceManager and Status updater. (jinho) (jhkim: rev 25bd5cb44a03ee425b02e2bc2553f7d0f8affff5)

          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java
          • tajo-core/src/main/proto/ResourceTrackerProtocol.proto
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceManagerEvent.java
          • tajo-core/src/test/java/org/apache/tajo/resource/TestResources.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java
          • tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java
          • CHANGES
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java
          • tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/DiskUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java
          • tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java
          • tajo-core/src/main/java/org/apache/tajo/resource/ResourceCalculator.java
          • tajo-common/src/main/proto/tajo_protos.proto
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java
          • tajo-core/src/main/java/org/apache/tajo/resource/NodeResources.java
          • tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java
          • tajo-core/src/main/java/org/apache/tajo/worker/event/NodeStatusEvent.java
          • tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
          Show
          hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-CODEGEN-build #357 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/357/ ) TAJO-1599 : Implement NodeResourceManager and Status updater. (jinho) (jhkim: rev 25bd5cb44a03ee425b02e2bc2553f7d0f8affff5) tajo-core/src/test/java/org/apache/tajo/worker/TestNodeResourceManager.java tajo-core/src/main/proto/ResourceTrackerProtocol.proto tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/main/java/org/apache/tajo/worker/NodeResourceManager.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceManagerEvent.java tajo-core/src/test/java/org/apache/tajo/resource/TestResources.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/resource/DefaultResourceCalculator.java tajo-core/src/main/java/org/apache/tajo/worker/NodeStatusUpdater.java tajo-core/src/test/java/org/apache/tajo/worker/TestNodeStatusUpdater.java CHANGES tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceAllocateEvent.java tajo-storage/tajo-storage-common/src/main/java/org/apache/tajo/storage/DiskUtil.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeResourceDeallocateEvent.java tajo-core/src/test/java/org/apache/tajo/worker/MockNodeStatusUpdater.java tajo-core/src/main/java/org/apache/tajo/resource/ResourceCalculator.java tajo-common/src/main/proto/tajo_protos.proto tajo-core/src/main/java/org/apache/tajo/resource/NodeResource.java tajo-core/src/main/java/org/apache/tajo/resource/NodeResources.java tajo-core/src/main/java/org/apache/tajo/worker/WorkerHeartbeatService.java tajo-core/src/main/java/org/apache/tajo/worker/event/NodeStatusEvent.java tajo-core/src/main/java/org/apache/tajo/master/rm/TajoResourceTracker.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy closed the pull request at:

          https://github.com/apache/tajo/pull/577

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy closed the pull request at: https://github.com/apache/tajo/pull/577
          Hide
          jhkim Jinho Kim added a comment -

          committed it to master

          Show
          jhkim Jinho Kim added a comment - committed it to master
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/577#issuecomment-105395728

          Thank you for the review!
          I'll commit soon

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/577#issuecomment-105395728 Thank you for the review! I'll commit soon
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/577#issuecomment-105347498

          +1 LGTM!

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/577#issuecomment-105347498 +1 LGTM!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/577#issuecomment-105342169

          Hi @jinossy, it's really nice work.
          I've almost read your patch. I'll finish my review soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/577#issuecomment-105342169 Hi @jinossy, it's really nice work. I've almost read your patch. I'll finish my review soon.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12734061/TAJO-1599_3.patch
          against master revision release-0.9.0-rc0-304-g5264156.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-storage/tajo-storage-common.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/795//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/795//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/795//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12734061/TAJO-1599_3.patch against master revision release-0.9.0-rc0-304-g5264156. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-storage/tajo-storage-common. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/795//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/795//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/795//console This message is automatically generated.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12733692/TAJO-1599_2.patch
          against master revision release-0.9.0-rc0-302-g4b1b779.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 4 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-storage/tajo-storage-common.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/794//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/794//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/794//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12733692/TAJO-1599_2.patch against master revision release-0.9.0-rc0-302-g4b1b779. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-common tajo-core tajo-storage/tajo-storage-common. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/794//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/794//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/794//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jinossy opened a pull request:

          https://github.com/apache/tajo/pull/577

          TAJO-1599: Implement NodeResourceManager and Status updater

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jinossy/tajo TAJO-1599

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/577.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #577


          commit 0d6ca33ec2d5b6f5be8bd3d78385b620548a7efb
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-05-15T03:03:59Z

          TAJO-1599

          commit 211f000bb8c14af81b5d77be9ae355b6d0b56cb1
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-05-18T07:15:14Z

          TAJO-1599

          commit 9631601b875f01e72fa4a3e74ec7c2a157d8a34f
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-05-18T07:15:30Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1599

          commit c0257fdc33009f26e31cbd8a1043f3bd6f7600ce
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-05-18T16:42:49Z

          add test cases

          commit fafd6487a56980355b3b596d392f299aad2d434f
          Author: Jinho Kim <jhkim@apache.org>
          Date: 2015-05-18T16:43:54Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1599


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/577 TAJO-1599 : Implement NodeResourceManager and Status updater You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1599 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/577.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #577 commit 0d6ca33ec2d5b6f5be8bd3d78385b620548a7efb Author: Jinho Kim <jhkim@apache.org> Date: 2015-05-15T03:03:59Z TAJO-1599 commit 211f000bb8c14af81b5d77be9ae355b6d0b56cb1 Author: Jinho Kim <jhkim@apache.org> Date: 2015-05-18T07:15:14Z TAJO-1599 commit 9631601b875f01e72fa4a3e74ec7c2a157d8a34f Author: Jinho Kim <jhkim@apache.org> Date: 2015-05-18T07:15:30Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1599 commit c0257fdc33009f26e31cbd8a1043f3bd6f7600ce Author: Jinho Kim <jhkim@apache.org> Date: 2015-05-18T16:42:49Z add test cases commit fafd6487a56980355b3b596d392f299aad2d434f Author: Jinho Kim <jhkim@apache.org> Date: 2015-05-18T16:43:54Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1599

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development