Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-928

Session variables should override query configs in TajoConf.

    Details

      Description

      Currently, we should use tajo-site in order to change the configurations related queries, such as optimization option or some parameters. It is never practical because we need to restart a Tajo cluster in order to change the config.

      The main purpose of this issue is to refactor the system of session variable and some part to recognize query configs to accept the session variables. Also, when there are duplicated configs in session and TajoConf, session variables should override the existing config of TajoConf.

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #330 (See https://builds.apache.org/job/Tajo-master-build/330/)
          TAJO-928: Session variables should override query configs in TajoConf. (hyunsik: rev ddfc3f33039ee15ab0a2d3fe5890b9acb40aec3d)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVerifier.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java
          • tajo-core/src/main/java/org/apache/tajo/master/session/Session.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/cli/TestTajoCli.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestNLJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalOptimizer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-common/src/main/java/org/apache/tajo/ConfigKey.java
          • tajo-core/src/test/resources/results/TestTajoCli/testHelpSessionVars.result
          • tajo-core/src/test/resources/results/TestTajoCli/testStopWhenErrorDeprecated.result
          • tajo-common/src/main/java/org/apache/tajo/datum/NullDatum.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/StorageUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalOptimizer.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterNLJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
          • tajo-core/src/test/java/org/apache/tajo/LocalTajoTestingUtility.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/src/test/java/org/apache/tajo/master/querymaster/TestQueryUnitStatusUpdate.java
          • tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java
          • tajo-client/src/main/java/org/apache/tajo/cli/DefaultTajoCliOutputFormatter.java
          • tajo-client/src/main/java/org/apache/tajo/cli/SetCommand.java
          • tajo-common/src/main/java/org/apache/tajo/datum/Datum.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
          • tajo-client/src/main/java/org/apache/tajo/cli/TajoShellCommand.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/avro/AvroUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableMeta.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/v2/RCFileScanner.java
          • tajo-client/src/main/java/org/apache/tajo/cli/HelpCommand.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/Appender.java
          • tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java
          • tajo-common/src/main/java/org/apache/tajo/SessionVars.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterHashJoinExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestKeyValueSet.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/QueryUnitRequestImpl.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/global/TestBroadcastJoinPlan.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java
          • tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java
          • tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
          • tajo-common/src/test/java/org/apache/tajo/datum/TestArithmeticOperator.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java
          • tajo-client/src/main/java/org/apache/tajo/cli/TajoCliOutputFormatter.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanPreprocessor.java
          • tajo-core/src/main/resources/webapps/worker/querydetail.jsp
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java
          • tajo-core/src/test/resources/results/TestTajoCli/testSelectResultWithNullTrueDeprecated.result
          • tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java
          • tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java
          • tajo-core/src/test/java/org/apache/tajo/cli/TestDefaultCliOutputFormatter.java
          • tajo-core/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java
          • tajo-core/src/main/java/org/apache/tajo/master/session/SessionLivelinessMonitor.java
          • tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java
          • tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java
          • tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java
          • CHANGES
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-common/src/main/java/org/apache/tajo/OverridableConf.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #330 (See https://builds.apache.org/job/Tajo-master-build/330/ ) TAJO-928 : Session variables should override query configs in TajoConf. (hyunsik: rev ddfc3f33039ee15ab0a2d3fe5890b9acb40aec3d) tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanVerifier.java tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlanner.java tajo-core/src/main/java/org/apache/tajo/master/session/Session.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterHashJoinExec.java tajo-core/src/test/java/org/apache/tajo/cli/TestTajoCli.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestNLJoinExec.java tajo-core/src/test/java/org/apache/tajo/TpchTestBase.java tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileScanner.java tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalOptimizer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-common/src/main/java/org/apache/tajo/ConfigKey.java tajo-core/src/test/resources/results/TestTajoCli/testHelpSessionVars.result tajo-core/src/test/resources/results/TestTajoCli/testStopWhenErrorDeprecated.result tajo-common/src/main/java/org/apache/tajo/datum/NullDatum.java tajo-storage/src/main/java/org/apache/tajo/storage/StorageUtil.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestExternalSortExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBNLJoinExec.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-storage/src/main/java/org/apache/tajo/storage/rcfile/RCFile.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterMergeJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashSemiJoinExec.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterTask.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestNullValues.java tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalOptimizer.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestLeftOuterNLJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core/src/test/java/org/apache/tajo/LocalTajoTestingUtility.java tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlan.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/test/java/org/apache/tajo/master/querymaster/TestQueryUnitStatusUpdate.java tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java tajo-core/src/test/java/org/apache/tajo/master/TestGlobalPlanner.java tajo-client/src/main/java/org/apache/tajo/cli/DefaultTajoCliOutputFormatter.java tajo-client/src/main/java/org/apache/tajo/cli/SetCommand.java tajo-common/src/main/java/org/apache/tajo/datum/Datum.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-client/src/main/java/org/apache/tajo/cli/TajoShellCommand.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-storage/src/main/java/org/apache/tajo/storage/avro/AvroUtil.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-storage/src/main/java/org/apache/tajo/storage/sequencefile/SequenceFileAppender.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/TableMeta.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinQuery.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashAntiJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestFullOuterMergeJoinExec.java tajo-storage/src/main/java/org/apache/tajo/storage/v2/RCFileScanner.java tajo-client/src/main/java/org/apache/tajo/cli/HelpCommand.java tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanner.java tajo-storage/src/main/java/org/apache/tajo/storage/Appender.java tajo-core/src/test/java/org/apache/tajo/jdbc/TestResultSet.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestMergeJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestSortExec.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMaster.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Query.java tajo-common/src/main/java/org/apache/tajo/SessionVars.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/test/java/org/apache/tajo/catalog/store/TestHCatalogStore.java tajo-core/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestRightOuterHashJoinExec.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestKeyValueSet.java tajo-core/src/main/java/org/apache/tajo/engine/query/QueryUnitRequestImpl.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestJoinBroadcast.java tajo-core/src/test/java/org/apache/tajo/engine/planner/global/TestBroadcastJoinPlan.java tajo-core/src/test/java/org/apache/tajo/engine/planner/TestPlannerUtil.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/src/test/java/org/apache/tajo/engine/planner/TestLogicalPlan.java tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java tajo-storage/src/main/java/org/apache/tajo/storage/CSVFile.java tajo-core/src/main/java/org/apache/tajo/master/TajoMasterClientService.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestProgressExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-common/src/test/java/org/apache/tajo/datum/TestArithmeticOperator.java tajo-core/src/test/java/org/apache/tajo/master/TestExecutionBlockCursor.java tajo-client/src/main/java/org/apache/tajo/cli/TajoCliOutputFormatter.java tajo-core/src/main/java/org/apache/tajo/engine/planner/LogicalPlanPreprocessor.java tajo-core/src/main/resources/webapps/worker/querydetail.jsp tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestHashJoinExec.java tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java tajo-core/src/test/resources/results/TestTajoCli/testSelectResultWithNullTrueDeprecated.result tajo-core/src/main/java/org/apache/tajo/master/querymaster/QueryMasterManagerService.java tajo-core/src/test/java/org/apache/tajo/QueryTestCaseBase.java tajo-core/src/test/java/org/apache/tajo/cli/TestDefaultCliOutputFormatter.java tajo-core/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java tajo-core/src/main/java/org/apache/tajo/master/session/SessionLivelinessMonitor.java tajo-catalog/tajo-catalog-drivers/tajo-hcatalog/src/main/java/org/apache/tajo/catalog/store/HCatalogStore.java tajo-core/src/test/java/org/apache/tajo/engine/planner/physical/TestBSTIndexExec.java tajo-core/src/test/java/org/apache/tajo/client/TestTajoClient.java CHANGES tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-common/src/main/java/org/apache/tajo/OverridableConf.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/98

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/98
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed it to master branch.

          Show
          hyunsik Hyunsik Choi added a comment - committed it to master branch.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51738322

          +1

          Thank you for your great patch, it will be very useful to tajo.
          It looks good overall and I tested it on my testing cluster.
          For reference, It works expectedly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51738322 +1 Thank you for your great patch, it will be very useful to tajo. It looks good overall and I tested it on my testing cluster. For reference, It works expectedly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51734846

          I've rebased the patch against the latest revision. Thank you for your review.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51734846 I've rebased the patch against the latest revision. Thank you for your review.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51734603

          Sorry for my late review.

          Could you rebase against the master branch?
          I failed to merge it as follows:

              CONFLICT (content): Merge conflict in tajo-core/src/test/java/org/apache/tajo/cli/TestTajoCli.java
              Auto-merging tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java
              Automatic merge failed; fix conflicts and then commit the result.
              
          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51734603 Sorry for my late review. Could you rebase against the master branch? I failed to merge it as follows: CONFLICT (content): Merge conflict in tajo-core/src/test/java/org/apache/tajo/cli/TestTajoCli.java Auto-merging tajo-client/src/main/java/org/apache/tajo/cli/TajoCli.java Automatic merge failed; fix conflicts and then commit the result.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51555782

          If the failure occurs occasionally, we need to handle it in another jira issue.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51555782 If the failure occurs occasionally, we need to handle it in another jira issue.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51555728

          I haven't experienced the failure. The failure seems to occur occasionally due to unknown reason. In addition, I strongly believe that this change affects the test.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51555728 I haven't experienced the failure. The failure seems to occur occasionally due to unknown reason. In addition, I strongly believe that this change affects the test.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51554635

          Hi @hyunsik

          Unfortunately, 'mvn clean install' failed as follows:

              Results :
              
              Failed tests:   testForwardedQuery(org.apache.tajo.scheduler.TestFifoScheduler): expected:<QUERY_RUNNING> but was:<QUERY_NOT_ASSIGNED>
              
              Tests run: 1096, Failures: 1, Errors: 0, Skipped: 0
              

          Could you check it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51554635 Hi @hyunsik Unfortunately, 'mvn clean install' failed as follows: Results : Failed tests: testForwardedQuery(org.apache.tajo.scheduler.TestFifoScheduler): expected: <QUERY_RUNNING> but was: <QUERY_NOT_ASSIGNED> Tests run: 1096, Failures: 1, Errors: 0, Skipped: 0 Could you check it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51436313

          @blrunner Thanks in advance!

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51436313 @blrunner Thanks in advance!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51436220

          Thank you for your detailed description and sorry but I'm hard to review it now.
          Could I review it tonight?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51436220 Thank you for your detailed description and sorry but I'm hard to review it now. Could I review it tonight?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-51433096

          I've described some changes.

          The current way is to set some keys in TajoConf.ConfVars. I deprecated the way, but the way still works for a while. If you use the old way, you can see the deprecation warning.
          ```
          Try ? for help.
          default> \set tajo.dist-query.join.broadcast.threshold-bytes 50000
          Warning: deprecated to directly use config key in TajoConf.ConfVars. Please execute '\help set'.
          ```

          But, the conf key automatically is replaced by new key corresponding to the conf key.

          ```
          default> \set
          'BROADCAST_TABLE_SIZE_LIMIT'='50000'
          'CURRENT_DATABASE'='default'
          'SESSION_ID'='1a104a50-3f8d-4bac-a8a2-5a8efc936298'
          'USERNAME'='hyunsik'
          'SESSION_LAST_ACCESS_TIME'='1407388966573'
          ```

          Also, I've added the help command and its alias. So you can use ```? [command name]``` or ```\help [command name]```. I've extended some method for help to ```TajoShellCommand``` interface. The following is the help of set command.
          ```
          default> \help set

          Available Session Variables:

          \set SESSION_EXPIRY_TIME [int value] - session expiry time (secs)
          \set CLI_COLUMNS [int value] - Sets the width for the wrapped format
          \set CLI_FORMATTER_CLASS [text value] - Sets the output format class to display results
          \set CLI_NULL_CHAR [text value] - Sets the string to be printed in place of a null value.
          \set CLI_PAGE_ROWS [int value] - Sets the number of rows for paging
          \set CLI_PAGING_ENABLED [true or false] - Enable paging of result display
          \set CLI_DISPLAY_ERROR_TRACE [true or false] - Enable display of error trace
          \set ON_ERROR_STOP [true or false] - tsql will exist if an error occurs.
          \set TZ [text value] - Sets timezone
          \set DATE_ORDER [text value] - date order (default is YMD)
          \set LANG [text value] - Language
          \set LC_ALL [text value] - String sort order
          \set LC_COLLATE [text value] - String sort order
          \set LC_CTYPE [text value] - Character classification (What is a letter? Its upper-case equivalent?)
          \set LC_MESSAGES [text value] - Language of messages
          \set LC_MONETARY [text value] - Formatting of currency amounts
          \set LC_NUMERIC [text value] - Formatting of numbers
          \set LC_TIME [text value] - Formatting of dates and times
          \set BROADCAST_TABLE_SIZE_LIMIT [long value] - limited size (bytes) of broadcast table
          \set JOIN_TASK_INPUT_SIZE [int value] - join task input size (mb)
          \set SORT_TASK_INPUT_SIZE [int value] - sort task input size (mb)
          \set GROUPBY_TASK_INPUT_SIZE [int value] - group by task input size (mb)
          \set JOIN_PER_SHUFFLE_SIZE [int value] - shuffle output size for join (mb)
          \set GROUPBY_PER_SHUFFLE_SIZE [int value] - shuffle output size for sort (mb)
          \set TABLE_PARTITION_PER_SHUFFLE_SIZE [int value] - shuffle output size for partition table write (mb)
          \set EXTSORT_BUFFER_SIZE [long value] - sort buffer size for external sort (mb)
          \set HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash join (mb)
          \set INNER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash inner join (mb)
          \set OUTER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash outer join (mb)
          \set HASH_GROUPBY_SIZE_LIMIT [long value] - limited size for hash groupby (mb)
          \set MAX_OUTPUT_FILE_SIZE [int value] - Maximum per-output file size (mb). 0 means infinite.
          \set NULL_CHAR [text value] - null char of text file output
          \set ARITHABORT [true or false] - If true, a running query will be terminated when an overflow or divide-by-zero occurs.
          \set DEBUG_ENABLED [true or false] - (debug only) debug mode enabled
          ```

          Also, I've updated querydetail.jsp file to show session variables applied to the query.

          I believe that this patch is ready to be committed. Please review this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-51433096 I've described some changes. The current way is to set some keys in TajoConf.ConfVars. I deprecated the way, but the way still works for a while. If you use the old way, you can see the deprecation warning. ``` Try ? for help. default> \set tajo.dist-query.join.broadcast.threshold-bytes 50000 Warning: deprecated to directly use config key in TajoConf.ConfVars. Please execute '\help set'. ``` But, the conf key automatically is replaced by new key corresponding to the conf key. ``` default> \set 'BROADCAST_TABLE_SIZE_LIMIT'='50000' 'CURRENT_DATABASE'='default' 'SESSION_ID'='1a104a50-3f8d-4bac-a8a2-5a8efc936298' 'USERNAME'='hyunsik' 'SESSION_LAST_ACCESS_TIME'='1407388966573' ``` Also, I've added the help command and its alias. So you can use ```? [command name] ``` or ```\help [command name] ```. I've extended some method for help to ```TajoShellCommand``` interface. The following is the help of set command. ``` default> \help set Available Session Variables: \set SESSION_EXPIRY_TIME [int value] - session expiry time (secs) \set CLI_COLUMNS [int value] - Sets the width for the wrapped format \set CLI_FORMATTER_CLASS [text value] - Sets the output format class to display results \set CLI_NULL_CHAR [text value] - Sets the string to be printed in place of a null value. \set CLI_PAGE_ROWS [int value] - Sets the number of rows for paging \set CLI_PAGING_ENABLED [true or false] - Enable paging of result display \set CLI_DISPLAY_ERROR_TRACE [true or false] - Enable display of error trace \set ON_ERROR_STOP [true or false] - tsql will exist if an error occurs. \set TZ [text value] - Sets timezone \set DATE_ORDER [text value] - date order (default is YMD) \set LANG [text value] - Language \set LC_ALL [text value] - String sort order \set LC_COLLATE [text value] - String sort order \set LC_CTYPE [text value] - Character classification (What is a letter? Its upper-case equivalent?) \set LC_MESSAGES [text value] - Language of messages \set LC_MONETARY [text value] - Formatting of currency amounts \set LC_NUMERIC [text value] - Formatting of numbers \set LC_TIME [text value] - Formatting of dates and times \set BROADCAST_TABLE_SIZE_LIMIT [long value] - limited size (bytes) of broadcast table \set JOIN_TASK_INPUT_SIZE [int value] - join task input size (mb) \set SORT_TASK_INPUT_SIZE [int value] - sort task input size (mb) \set GROUPBY_TASK_INPUT_SIZE [int value] - group by task input size (mb) \set JOIN_PER_SHUFFLE_SIZE [int value] - shuffle output size for join (mb) \set GROUPBY_PER_SHUFFLE_SIZE [int value] - shuffle output size for sort (mb) \set TABLE_PARTITION_PER_SHUFFLE_SIZE [int value] - shuffle output size for partition table write (mb) \set EXTSORT_BUFFER_SIZE [long value] - sort buffer size for external sort (mb) \set HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash join (mb) \set INNER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash inner join (mb) \set OUTER_HASH_JOIN_SIZE_LIMIT [long value] - limited size for hash outer join (mb) \set HASH_GROUPBY_SIZE_LIMIT [long value] - limited size for hash groupby (mb) \set MAX_OUTPUT_FILE_SIZE [int value] - Maximum per-output file size (mb). 0 means infinite. \set NULL_CHAR [text value] - null char of text file output \set ARITHABORT [true or false] - If true, a running query will be terminated when an overflow or divide-by-zero occurs. \set DEBUG_ENABLED [true or false] - (debug only) debug mode enabled ``` Also, I've updated querydetail.jsp file to show session variables applied to the query. I believe that this patch is ready to be committed. Please review this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/98#discussion_r15918947

          — Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java —
          @@ -0,0 +1,239 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo;
          +
          +import com.google.common.base.Preconditions;
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.tajo.conf.TajoConf;
          +import org.apache.tajo.util.KeyValueSet;
          +
          +import java.lang.ref.WeakReference;
          +import java.util.Map;
          +import java.util.WeakHashMap;
          +
          +import static org.apache.tajo.ConfigKey.ConfigType;
          +import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto;
          +
          +/**
          + * QueryContext provides a consolidated config system for a query instant.
          + *
          + * In Tajo, there are three configurable layers:
          + * <ul>
          + * <li>
          + * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site,
          + * catalog-default and TajoConf.</ul>
          + * <ul>Session variables - they are instantly configured by users.
          + * Each client session has it own set of session variables.</ul>
          + * <ul>Query config - it is internally used for meta information of a query instance.</ul>
          + * </li>
          + * </ul>
          + *
          + * System configs and session variables can set the same config in the same time. System configs are usually used to set
          + * default configs, and session variables is user-specified configs. So, session variables can override system configs.
          + *
          + * QueryContent provides a query with a uniform way to access various configs without considering their priorities.
          + */
          +public class OverridableConf extends KeyValueSet {
          + private static final Log LOG = LogFactory.getLog(OverridableConf.class);
          + private ConfigType [] configTypes;
          +
          + private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>>
          + CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>();
          +
          + private TajoConf conf;
          +
          + public OverridableConf(final TajoConf conf, ConfigType...configTypes)

          { + this.conf = conf; + this.configTypes = configTypes; + }

          +
          + public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes)

          { + super(proto); + this.conf = conf; + this.configTypes = configTypes; + }

          +
          + public void setConf(TajoConf conf)

          { + this.conf = conf; + }

          +
          + public TajoConf getConf()

          { + return conf; + }

          +
          + public void setBool(ConfigKey key, boolean val)

          { + setBool(key.keyname(), val); + }

          +
          + public boolean getBool(ConfigKey key, Boolean defaultVal) {
          + assertRegisteredEnum(key);
          +
          + switch (key.type())

          { + case QUERY: + return getBool(key.keyname()); + case SESSION: + return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getBoolVar((TajoConf.ConfVars) key); + default: + return getBool(key.keyname(), defaultVal); + }

          + }
          +
          + public boolean getBool(ConfigKey key)

          { + return getBool(key, null); + }

          +
          + public int getInt(ConfigKey key, Integer defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return getInt(key.keyname()); + }

          else {
          + switch (key.type())

          { + case SESSION: + return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getIntVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public int getInt(ConfigKey key)

          { + return getInt(key, null); + }

          +
          + public long getLong(ConfigKey key, Long defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return getLong(key.keyname()); + }

          else {
          + switch (key.type())

          { + case SESSION: + return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getLongVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public long getLong(ConfigKey key)

          { + return getLong(key, null); + }
          +
          + public float getFloat(ConfigKey key, Float defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getFloat(key.keyname()); + } else {
          + switch (key.type()) { + case SESSION: + return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getFloatVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }
          + }
          + }
          +
          + public float getFloat(ConfigKey key) { + return getLong(key, null); + }

          +
          + public void put(ConfigKey key, String val)

          { + set(key.keyname(), val); + }

          +
          + private void assertRegisteredEnum(ConfigKey key) {
          + boolean registered = false;
          +
          + for (ConfigType c : configTypes)

          { + registered = key.type() == c; + }

          +
          + registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM;
          +
          + Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " +
          + getClass().getSimpleName());
          + }
          +
          + public String get(ConfigKey key, String defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return get(key.keyname(), defaultVal); + }

          else {
          + switch (key.type())

          { + case SESSION: + return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public String get(ConfigKey key)

          { + return get(key, null); + }

          +
          + public Class<?> getClass(ConfigKey key) {
          +// if (containsKey(key)) {
          — End diff –

          I removed the commented out lines.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/98#discussion_r15918947 — Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java — @@ -0,0 +1,239 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo; + +import com.google.common.base.Preconditions; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.tajo.conf.TajoConf; +import org.apache.tajo.util.KeyValueSet; + +import java.lang.ref.WeakReference; +import java.util.Map; +import java.util.WeakHashMap; + +import static org.apache.tajo.ConfigKey.ConfigType; +import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto; + +/** + * QueryContext provides a consolidated config system for a query instant. + * + * In Tajo, there are three configurable layers: + * <ul> + * <li> + * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site, + * catalog-default and TajoConf.</ul> + * <ul>Session variables - they are instantly configured by users. + * Each client session has it own set of session variables.</ul> + * <ul>Query config - it is internally used for meta information of a query instance.</ul> + * </li> + * </ul> + * + * System configs and session variables can set the same config in the same time. System configs are usually used to set + * default configs, and session variables is user-specified configs. So, session variables can override system configs. + * + * QueryContent provides a query with a uniform way to access various configs without considering their priorities. + */ +public class OverridableConf extends KeyValueSet { + private static final Log LOG = LogFactory.getLog(OverridableConf.class); + private ConfigType [] configTypes; + + private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>> + CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>(); + + private TajoConf conf; + + public OverridableConf(final TajoConf conf, ConfigType...configTypes) { + this.conf = conf; + this.configTypes = configTypes; + } + + public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes) { + super(proto); + this.conf = conf; + this.configTypes = configTypes; + } + + public void setConf(TajoConf conf) { + this.conf = conf; + } + + public TajoConf getConf() { + return conf; + } + + public void setBool(ConfigKey key, boolean val) { + setBool(key.keyname(), val); + } + + public boolean getBool(ConfigKey key, Boolean defaultVal) { + assertRegisteredEnum(key); + + switch (key.type()) { + case QUERY: + return getBool(key.keyname()); + case SESSION: + return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getBoolVar((TajoConf.ConfVars) key); + default: + return getBool(key.keyname(), defaultVal); + } + } + + public boolean getBool(ConfigKey key) { + return getBool(key, null); + } + + public int getInt(ConfigKey key, Integer defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getInt(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getIntVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public int getInt(ConfigKey key) { + return getInt(key, null); + } + + public long getLong(ConfigKey key, Long defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getLong(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getLongVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public long getLong(ConfigKey key) { + return getLong(key, null); + } + + public float getFloat(ConfigKey key, Float defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getFloat(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getFloatVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public float getFloat(ConfigKey key) { + return getLong(key, null); + } + + public void put(ConfigKey key, String val) { + set(key.keyname(), val); + } + + private void assertRegisteredEnum(ConfigKey key) { + boolean registered = false; + + for (ConfigType c : configTypes) { + registered = key.type() == c; + } + + registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM; + + Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " + + getClass().getSimpleName()); + } + + public String get(ConfigKey key, String defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return get(key.keyname(), defaultVal); + } else { + switch (key.type()) { + case SESSION: + return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public String get(ConfigKey key) { + return get(key, null); + } + + public Class<?> getClass(ConfigKey key) { +// if (containsKey(key)) { — End diff – I removed the commented out lines.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user blrunner commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/98#discussion_r15792495

          — Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java —
          @@ -0,0 +1,239 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.tajo;
          +
          +import com.google.common.base.Preconditions;
          +import org.apache.commons.logging.Log;
          +import org.apache.commons.logging.LogFactory;
          +import org.apache.tajo.conf.TajoConf;
          +import org.apache.tajo.util.KeyValueSet;
          +
          +import java.lang.ref.WeakReference;
          +import java.util.Map;
          +import java.util.WeakHashMap;
          +
          +import static org.apache.tajo.ConfigKey.ConfigType;
          +import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto;
          +
          +/**
          + * QueryContext provides a consolidated config system for a query instant.
          + *
          + * In Tajo, there are three configurable layers:
          + * <ul>
          + * <li>
          + * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site,
          + * catalog-default and TajoConf.</ul>
          + * <ul>Session variables - they are instantly configured by users.
          + * Each client session has it own set of session variables.</ul>
          + * <ul>Query config - it is internally used for meta information of a query instance.</ul>
          + * </li>
          + * </ul>
          + *
          + * System configs and session variables can set the same config in the same time. System configs are usually used to set
          + * default configs, and session variables is user-specified configs. So, session variables can override system configs.
          + *
          + * QueryContent provides a query with a uniform way to access various configs without considering their priorities.
          + */
          +public class OverridableConf extends KeyValueSet {
          + private static final Log LOG = LogFactory.getLog(OverridableConf.class);
          + private ConfigType [] configTypes;
          +
          + private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>>
          + CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>();
          +
          + private TajoConf conf;
          +
          + public OverridableConf(final TajoConf conf, ConfigType...configTypes)

          { + this.conf = conf; + this.configTypes = configTypes; + }

          +
          + public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes)

          { + super(proto); + this.conf = conf; + this.configTypes = configTypes; + }

          +
          + public void setConf(TajoConf conf)

          { + this.conf = conf; + }

          +
          + public TajoConf getConf()

          { + return conf; + }

          +
          + public void setBool(ConfigKey key, boolean val)

          { + setBool(key.keyname(), val); + }

          +
          + public boolean getBool(ConfigKey key, Boolean defaultVal) {
          + assertRegisteredEnum(key);
          +
          + switch (key.type())

          { + case QUERY: + return getBool(key.keyname()); + case SESSION: + return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getBoolVar((TajoConf.ConfVars) key); + default: + return getBool(key.keyname(), defaultVal); + }

          + }
          +
          + public boolean getBool(ConfigKey key)

          { + return getBool(key, null); + }

          +
          + public int getInt(ConfigKey key, Integer defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return getInt(key.keyname()); + }

          else {
          + switch (key.type())

          { + case SESSION: + return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getIntVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public int getInt(ConfigKey key)

          { + return getInt(key, null); + }

          +
          + public long getLong(ConfigKey key, Long defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return getLong(key.keyname()); + }

          else {
          + switch (key.type())

          { + case SESSION: + return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getLongVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public long getLong(ConfigKey key)

          { + return getLong(key, null); + }
          +
          + public float getFloat(ConfigKey key, Float defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getFloat(key.keyname()); + } else {
          + switch (key.type()) { + case SESSION: + return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getFloatVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }
          + }
          + }
          +
          + public float getFloat(ConfigKey key) { + return getLong(key, null); + }

          +
          + public void put(ConfigKey key, String val)

          { + set(key.keyname(), val); + }

          +
          + private void assertRegisteredEnum(ConfigKey key) {
          + boolean registered = false;
          +
          + for (ConfigType c : configTypes)

          { + registered = key.type() == c; + }

          +
          + registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM;
          +
          + Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " +
          + getClass().getSimpleName());
          + }
          +
          + public String get(ConfigKey key, String defaultVal) {
          + assertRegisteredEnum(key);
          +
          + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM)

          { + return get(key.keyname(), defaultVal); + }

          else {
          + switch (key.type())

          { + case SESSION: + return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + }

          + }
          + }
          +
          + public String get(ConfigKey key)

          { + return get(key, null); + }

          +
          + public Class<?> getClass(ConfigKey key) {
          +// if (containsKey(key)) {
          — End diff –

          Is this a comment?

          Show
          githubbot ASF GitHub Bot added a comment - Github user blrunner commented on a diff in the pull request: https://github.com/apache/tajo/pull/98#discussion_r15792495 — Diff: tajo-common/src/main/java/org/apache/tajo/OverridableConf.java — @@ -0,0 +1,239 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.tajo; + +import com.google.common.base.Preconditions; +import org.apache.commons.logging.Log; +import org.apache.commons.logging.LogFactory; +import org.apache.tajo.conf.TajoConf; +import org.apache.tajo.util.KeyValueSet; + +import java.lang.ref.WeakReference; +import java.util.Map; +import java.util.WeakHashMap; + +import static org.apache.tajo.ConfigKey.ConfigType; +import static org.apache.tajo.rpc.protocolrecords.PrimitiveProtos.KeyValueSetProto; + +/** + * QueryContext provides a consolidated config system for a query instant. + * + * In Tajo, there are three configurable layers: + * <ul> + * <li> + * <ul>System Config - it comes from Hadoop's Configuration class. by tajo-site, catalog-site, + * catalog-default and TajoConf.</ul> + * <ul>Session variables - they are instantly configured by users. + * Each client session has it own set of session variables.</ul> + * <ul>Query config - it is internally used for meta information of a query instance.</ul> + * </li> + * </ul> + * + * System configs and session variables can set the same config in the same time. System configs are usually used to set + * default configs, and session variables is user-specified configs. So, session variables can override system configs. + * + * QueryContent provides a query with a uniform way to access various configs without considering their priorities. + */ +public class OverridableConf extends KeyValueSet { + private static final Log LOG = LogFactory.getLog(OverridableConf.class); + private ConfigType [] configTypes; + + private static final Map<ClassLoader, Map<String, WeakReference<Class<?>>>> + CACHE_CLASSES = new WeakHashMap<ClassLoader, Map<String, WeakReference<Class<?>>>>(); + + private TajoConf conf; + + public OverridableConf(final TajoConf conf, ConfigType...configTypes) { + this.conf = conf; + this.configTypes = configTypes; + } + + public OverridableConf(final TajoConf conf, KeyValueSetProto proto, ConfigType...configTypes) { + super(proto); + this.conf = conf; + this.configTypes = configTypes; + } + + public void setConf(TajoConf conf) { + this.conf = conf; + } + + public TajoConf getConf() { + return conf; + } + + public void setBool(ConfigKey key, boolean val) { + setBool(key.keyname(), val); + } + + public boolean getBool(ConfigKey key, Boolean defaultVal) { + assertRegisteredEnum(key); + + switch (key.type()) { + case QUERY: + return getBool(key.keyname()); + case SESSION: + return getBool(key.keyname(), conf.getBoolVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getBoolVar((TajoConf.ConfVars) key); + default: + return getBool(key.keyname(), defaultVal); + } + } + + public boolean getBool(ConfigKey key) { + return getBool(key, null); + } + + public int getInt(ConfigKey key, Integer defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getInt(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getInt(key.keyname(), conf.getIntVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getIntVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public int getInt(ConfigKey key) { + return getInt(key, null); + } + + public long getLong(ConfigKey key, Long defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getLong(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getLong(key.keyname(), conf.getLongVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getLongVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public long getLong(ConfigKey key) { + return getLong(key, null); + } + + public float getFloat(ConfigKey key, Float defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return getFloat(key.keyname()); + } else { + switch (key.type()) { + case SESSION: + return getFloat(key.keyname(), conf.getFloatVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getFloatVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public float getFloat(ConfigKey key) { + return getLong(key, null); + } + + public void put(ConfigKey key, String val) { + set(key.keyname(), val); + } + + private void assertRegisteredEnum(ConfigKey key) { + boolean registered = false; + + for (ConfigType c : configTypes) { + registered = key.type() == c; + } + + registered |= key.type() == ConfigType.SESSION || key.type() != ConfigType.SYSTEM; + + Preconditions.checkArgument(registered, key.keyname() + " (" + key.type() + ") is not allowed in " + + getClass().getSimpleName()); + } + + public String get(ConfigKey key, String defaultVal) { + assertRegisteredEnum(key); + + if (key.type() != ConfigType.SESSION && key.type() != ConfigType.SYSTEM) { + return get(key.keyname(), defaultVal); + } else { + switch (key.type()) { + case SESSION: + return get(key.keyname(), conf.getVar(((SessionVars) key).getConfVars())); + case SYSTEM: + return conf.getVar((TajoConf.ConfVars) key); + default: + throw new IllegalStateException("key does not belong to Session and System config sets"); + } + } + } + + public String get(ConfigKey key) { + return get(key, null); + } + + public Class<?> getClass(ConfigKey key) { +// if (containsKey(key)) { — End diff – Is this a comment?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-50998335

          It is ready to be reviewed. Please review the patch. I'll also add some documentation soon.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-50998335 It is ready to be reviewed. Please review the patch. I'll also add some documentation soon.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/98#issuecomment-50657070

          This is still ongoing work. The remain works are as follows:

          • Some session variables only used in cli will be removed
          • Add more comments and description
          • Add new meta comment for tsql display configuration
          • Add documentation about new session variable system
          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/98#issuecomment-50657070 This is still ongoing work. The remain works are as follows: Some session variables only used in cli will be removed Add more comments and description Add new meta comment for tsql display configuration Add documentation about new session variable system
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user hyunsik opened a pull request:

          https://github.com/apache/tajo/pull/98

          TAJO-928: Session variables should override query configs in TajoConf.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/hyunsik/tajo TAJO-928

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/98.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #98


          commit a3b78642abb6c160b147eae2f29a10e362c14cac
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-08T08:47:42Z

          Improve session variables to affect the query config.

          commit 0a0035d9b259a1a05ba790b7a778a745251d27bd
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-08T12:54:32Z

          Fixed.

          commit 3fb54a6dde89d2d8e972253c1eccd17f334180d4
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-09T02:23:28Z

          Completed output file rotating.

          commit 8028f5f876af2050bb602e277026e76ca802619a
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-15T03:57:29Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING

          Conflicts:
          tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
          tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
          tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java

          commit 50f6af418b42704ba14a4c7a084372f80c7ce1ec
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-15T06:25:09Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING

          commit 4d0abc0dfbf6c5898bce6bd0e1ecd4c995108571
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-15T11:13:55Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING

          Conflicts:
          tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashBasedColPartitionStoreExec.java
          tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortBasedColPartitionStoreExec.java

          commit dd79f666d81875bf6a547478b76fc55b60f37d09
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-15T12:31:11Z

          Added estimatedwrittensize.

          commit da231ca89e5cf3638ea16faad281f8296854a9dd
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-17T03:03:37Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING

          commit c006382a3b16973872d753c9a0e0150da1c0f687
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-17T03:10:20Z

          Reflect session variables to GlobalPlanner, Repartitioner, and PhysicalPlannerImpl.

          commit bccd8528c84e09bc9861648bb427fb7706f8652d
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-25T01:20:33Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING

          commit 244318ccf8154c4b4e45dc24fdab23f85513e496
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-25T01:27:08Z

          Reset output rotating part.

          commit 244b2614f0986248cec43bd0012bce8c489ea95f
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-25T06:28:35Z

          Add SessionVars and improve QueryContext.

          commit ed05c03448e9b49fa1280266d6b4fc38e473f314
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-25T08:06:35Z

          Improved QueryContext.

          commit c8e53b4d140b7f6d656f10ded1767272b62a526e
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-29T07:29:05Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-928

          Conflicts:
          tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java

          commit 2becff67dafba948e37463ef840c2d96ce8c56a1
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-29T10:16:01Z

          Refactored QueryContext to take conf always.

          commit 59861210fe4f16b50b076fd0b0456d6f7bd8a145
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-29T10:35:55Z

          Refactored TaskAttemptContext to not take TajoConf.

          commit ea952a4a3068f9ccff8cf2455300ddd7968da0fd
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2014-07-30T18:09:19Z

          Fixed all unit tests.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/98 TAJO-928 : Session variables should override query configs in TajoConf. You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-928 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/98.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #98 commit a3b78642abb6c160b147eae2f29a10e362c14cac Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-08T08:47:42Z Improve session variables to affect the query config. commit 0a0035d9b259a1a05ba790b7a778a745251d27bd Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-08T12:54:32Z Fixed. commit 3fb54a6dde89d2d8e972253c1eccd17f334180d4 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-09T02:23:28Z Completed output file rotating. commit 8028f5f876af2050bb602e277026e76ca802619a Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-15T03:57:29Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING Conflicts: tajo-core/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java commit 50f6af418b42704ba14a4c7a084372f80c7ce1ec Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-15T06:25:09Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING commit 4d0abc0dfbf6c5898bce6bd0e1ecd4c995108571 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-15T11:13:55Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING Conflicts: tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashBasedColPartitionStoreExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SortBasedColPartitionStoreExec.java commit dd79f666d81875bf6a547478b76fc55b60f37d09 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-15T12:31:11Z Added estimatedwrittensize. commit da231ca89e5cf3638ea16faad281f8296854a9dd Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-17T03:03:37Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING commit c006382a3b16973872d753c9a0e0150da1c0f687 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-17T03:10:20Z Reflect session variables to GlobalPlanner, Repartitioner, and PhysicalPlannerImpl. commit bccd8528c84e09bc9861648bb427fb7706f8652d Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-25T01:20:33Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into OUTPUT_ROTATING commit 244318ccf8154c4b4e45dc24fdab23f85513e496 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-25T01:27:08Z Reset output rotating part. commit 244b2614f0986248cec43bd0012bce8c489ea95f Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-25T06:28:35Z Add SessionVars and improve QueryContext. commit ed05c03448e9b49fa1280266d6b4fc38e473f314 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-25T08:06:35Z Improved QueryContext. commit c8e53b4d140b7f6d656f10ded1767272b62a526e Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-29T07:29:05Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-928 Conflicts: tajo-core/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java commit 2becff67dafba948e37463ef840c2d96ce8c56a1 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-29T10:16:01Z Refactored QueryContext to take conf always. commit 59861210fe4f16b50b076fd0b0456d6f7bd8a145 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-29T10:35:55Z Refactored TaskAttemptContext to not take TajoConf. commit ea952a4a3068f9ccff8cf2455300ddd7968da0fd Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-07-30T18:09:19Z Fixed all unit tests.

            People

            • Assignee:
              hyunsik Hyunsik Choi
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development