Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1135

Implement queryable virtual table for cluster information

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.11.0
    • Fix Version/s: 0.11.0
    • Component/s: TajoMaster
    • Labels:
      None

      Description

      Currently, Tajo does not provide queryable cluster information. We just can display cluster nodes through 'bin/tajo cluster' inline command.

      I propose that TajoMaster should maintain a queryable virtual table where each row represents each cluster node and columns should contain hostname, resource, last ping, and so on. The virtual table should be processed by API and SQL.

      Users can discovery cluster information in ad-hoc manner. Also, internal Tajo system can make good use of the virtual table for scheduling resources or displaying cluster information in WEB UI. I also expect that this feature also simplify the many parts related to cluster information.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ykrips opened a pull request:

          https://github.com/apache/tajo/pull/366

          TAJO-1135: Implement queryable virtual table for cluster information

          This virtual table will be shown like following.
          ```
          information_schema> select * from workers;
          host, port, type, status, total_cpu, used_mem, total_mem, free_heap, max_heap, used_diskslots, total_diskslots, running_tasks, heartbeat
          -------------------------------
          hostnameA, 28093, QueryMaster, RUNNING, , , , 130004408, 932118528, , , 0, 2015-01-29 17:09:18.859999
          hostnameA, 28091, Worker, RUNNING, 1, 0, 1024, 130004408, 932118528, 0.0, 1.0, 0, 2015-01-29 17:09:18.859999
          (2 rows, 0.025 sec, 0 B selected)
          ```

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ykrips/tajo TAJO-1135

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/366.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #366


          commit 0217440de6bfeac45a175663dec23bd732b22904
          Author: Jihun Kang <jihun@apache.org>
          Date: 2015-01-29T08:16:18Z

          TAJO-1135: Implement queryable virtual table for cluster information


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ykrips opened a pull request: https://github.com/apache/tajo/pull/366 TAJO-1135 : Implement queryable virtual table for cluster information This virtual table will be shown like following. ``` information_schema> select * from workers; host, port, type, status, total_cpu, used_mem, total_mem, free_heap, max_heap, used_diskslots, total_diskslots, running_tasks, heartbeat ------------------------------- hostnameA, 28093, QueryMaster, RUNNING, , , , 130004408, 932118528, , , 0, 2015-01-29 17:09:18.859999 hostnameA, 28091, Worker, RUNNING, 1, 0, 1024, 130004408, 932118528, 0.0, 1.0, 0, 2015-01-29 17:09:18.859999 (2 rows, 0.025 sec, 0 B selected) ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/ykrips/tajo TAJO-1135 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/366.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #366 commit 0217440de6bfeac45a175663dec23bd732b22904 Author: Jihun Kang <jihun@apache.org> Date: 2015-01-29T08:16:18Z TAJO-1135 : Implement queryable virtual table for cluster information
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-73225187

          Hi @ykrips, thanks for your work.
          In overall, the patch looks good to me.
          Only one thing I want to suggest is the table name.
          Since the newly added table contains the information of query masters as well as workers, I think that ```cluster``` is the more proper name than ```workers```.
          What do you think about it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-73225187 Hi @ykrips, thanks for your work. In overall, the patch looks good to me. Only one thing I want to suggest is the table name. Since the newly added table contains the information of query masters as well as workers, I think that ```cluster``` is the more proper name than ```workers```. What do you think about it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ykrips commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-73232809

          Good idea. ```cluster``` would be better. fix up and post the commit soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ykrips commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-73232809 Good idea. ```cluster``` would be better. fix up and post the commit soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-75350776

          I'm sorry for the late review. I'll review it soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-75350776 I'm sorry for the late review. I'll review it soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-76565070

          Sorry for late review.
          The latest patch looks good to me.
          Here is my +1, but it will be good to wait for Hyunsik's review.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-76565070 Sorry for late review. The latest patch looks good to me. Here is my +1, but it will be good to wait for Hyunsik's review.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ykrips commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-76587528

          No problems. I'll commit it after hyunsik finishes the review.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ykrips commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-76587528 No problems. I'll commit it after hyunsik finishes the review.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-76587997

          I'll finish the review by tomorrow.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-76587997 I'll finish the review by tomorrow.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ykrips commented on the pull request:

          https://github.com/apache/tajo/pull/366#issuecomment-77476639

          That would be good. This column name will be changed in a next commit.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ykrips commented on the pull request: https://github.com/apache/tajo/pull/366#issuecomment-77476639 That would be good. This column name will be changed in a next commit.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12702988/TAJO-1135.patch
          against master revision release-0.9.0-rc0-185-g1617fa9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-core.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/600//testReport/
          Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/600//findbugsResult
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/600//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12702988/TAJO-1135.patch against master revision release-0.9.0-rc0-185-g1617fa9. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to cause Findbugs (version 2.0.3) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-server tajo-core. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/600//testReport/ Findbugs results: https://builds.apache.org/job/PreCommit-TAJO-Build/600//findbugsResult Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/600//console This message is automatically generated.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1 ship it.

          The patch looks nice to me.

          Show
          hyunsik Hyunsik Choi added a comment - +1 ship it. The patch looks nice to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/366

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/366
          Hide
          ykrips Jihun Kang added a comment -

          Hyunsik Choi, this patch committed to master branch.

          Show
          ykrips Jihun Kang added a comment - Hyunsik Choi , this patch committed to master branch.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #237 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/237/)
          TAJO-1135: Implement queryable virtual table for cluster information (jihun: rev 6e519bcf39a526b64e62d9957404bfd8a3888486)

          • tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java
          • CHANGES
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #237 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/237/ ) TAJO-1135 : Implement queryable virtual table for cluster information (jihun: rev 6e519bcf39a526b64e62d9957404bfd8a3888486) tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java CHANGES tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #599 (See https://builds.apache.org/job/Tajo-master-build/599/)
          TAJO-1135: Implement queryable virtual table for cluster information (jihun: rev 6e519bcf39a526b64e62d9957404bfd8a3888486)

          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #599 (See https://builds.apache.org/job/Tajo-master-build/599/ ) TAJO-1135 : Implement queryable virtual table for cluster information (jihun: rev 6e519bcf39a526b64e62d9957404bfd8a3888486) tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/InfoSchemaMetadataDictionary.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/dictionary/ClusterTableDescriptor.java CHANGES tajo-core/src/test/java/org/apache/tajo/master/TestNonForwardQueryResultSystemScanner.java tajo-core/src/main/java/org/apache/tajo/master/exec/NonForwardQueryResultSystemScanner.java

            People

            • Assignee:
              ykrips Jihun Kang
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development