Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Function/UDF
    • Labels:
      None

      Description

      Python has abundant users and third-party libraries. This language is widely used in data analytic area. So, it would be great if Tajo supports Python UDF.

      1. TAJO-1344_2.patch
        79 kB
        Jihoon Son
      2. TAJO-1344_3.patch
        85 kB
        Jihoon Son
      3. TAJO-1344_4.patch
        88 kB
        Jihoon Son
      4. TAJO-1344_5.patch
        261 kB
        Jihoon Son
      5. TAJO-1344_6.patch
        262 kB
        Jihoon Son
      6. TAJO-1344_7.patch
        262 kB
        Jihoon Son
      7. TAJO-1344_8.patch
        262 kB
        Jihoon Son
      8. TAJO-1344.patch
        78 kB
        Jihoon Son

        Issue Links

          Activity

          Hide
          jihoonson Jihoon Son added a comment -

          Committed to master.

          Show
          jihoonson Jihoon Son added a comment - Committed to master.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-build #673 (See https://builds.apache.org/job/Tajo-master-build/673/)
          TAJO-1344: Python UDF support. (jihoon) (jihoonson: rev a7453853975dc8b5f4162ae16a1669aeb7ed995c)

          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/BufferPool.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineDeserializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalNode.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/LogicalPlanEqualityTester.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/PythonInvocationDesc.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectPythonFuncs.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/FieldEval.java
          • tajo-common/src/main/java/org/apache/tajo/datum/BlobDatum.java
          • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestJsonSerDe/testVariousType.json
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/InputHandler.java
          • tajo-common/src/main/java/org/apache/tajo/util/datetime/DateTimeFormat.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerDe.java
          • tajo-common/src/main/java/org/apache/tajo/QueryVars.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc2.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/OutputHandler.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/rewriter/rules/GlobalPlanEqualityTester.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/Projector.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/LegacyScalarFunctionInvoke.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalTreeUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufInputChannel.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvoke.java
          • tajo-core/src/main/resources/python/tajo_util.py
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizationRule.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
          • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-docs/src/main/sphinx/functions.rst
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/function/TestPythonFunctions.java
          • tajo-core/src/test/java/org/apache/tajo/engine/function/TestMathFunctions.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/CastEval.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/FunctionEval.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/TajoScriptEngine.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSplitProcessor.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc2.sql
          • tajo-core/src/main/resources/python/controller.py
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeSerializer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/PythonFunctionInvoke.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvokeContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSerializerDeserializer.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectWithPredicateOnPythonFunc.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/PatternMatchPredicateEval.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
          • tajo-common/src/main/java/org/apache/tajo/storage/Tuple.java
          • tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestFunctionDesc.java
          • tajo-core/src/test/resources/python/_init_.py
          • tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java
          • tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/StreamingUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-core/src/test/resources/python/test_funcs.pyc
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufLineReader.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/EvalCodeGenContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextFieldSerializerDeserializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/LegacyFunctionBindingEmitter.java
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/VariablesPreBuilder.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testNestedPythonFunction.result
          • tajo-common/src/main/java/org/apache/tajo/json/DatumAdapter.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyFirstAggregationExec.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/WindowAggExec.java
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance1.json
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeDeserializer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineDeserializer.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-storage/tajo-storage-common/src/test/java/org/apache/tajo/storage/TestLazyTuple.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/BetweenPredicateEval.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionInvocation.java
          • tajo-core/src/test/resources/python/test_funcs.py
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonJoinExec.java
          • tajo-core/src/test/resources/queries/TestSelectQuery/testNestedPythonFunction.sql
          • tajo-core/src/test/resources/queries/TestSelectQuery/testSelectPythonFuncs.sql
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc.sql
          • tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/GeneralFunctionEval.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbySecondAggregationExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineParsingError.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyThirdAggregationExec.java
          • tajo-docs/src/main/sphinx/functions/json_func.rst
          • tajo-core/src/test/resources/python/test_funcs2.py
          • tajo-core/src/test/resources/queries/TestSelectQuery/testSelectWithPredicateOnPythonFunc.sql
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerDe.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/ExprAnnotator.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/LineSplitProcessor.java
          • tajo-core/src/main/resources/python/_init_.py
          • tajo-common/src/main/java/org/apache/tajo/datum/AnyDatum.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #673 (See https://builds.apache.org/job/Tajo-master-build/673/ ) TAJO-1344 : Python UDF support. (jihoon) (jihoonson: rev a7453853975dc8b5f4162ae16a1669aeb7ed995c) tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/BufferPool.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineDeserializer.java tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalNode.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/LogicalPlanEqualityTester.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/PythonInvocationDesc.java tajo-core/src/test/resources/results/TestSelectQuery/testSelectPythonFuncs.result tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/FieldEval.java tajo-common/src/main/java/org/apache/tajo/datum/BlobDatum.java tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestJsonSerDe/testVariousType.json tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/InputHandler.java tajo-common/src/main/java/org/apache/tajo/util/datetime/DateTimeFormat.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerDe.java tajo-common/src/main/java/org/apache/tajo/QueryVars.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc2.result tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/OutputHandler.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerializer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/rewriter/rules/GlobalPlanEqualityTester.java tajo-core/src/main/java/org/apache/tajo/engine/planner/Projector.java tajo-plan/src/main/java/org/apache/tajo/plan/function/LegacyScalarFunctionInvoke.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalContext.java tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalTreeUtil.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufInputChannel.java tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvoke.java tajo-core/src/main/resources/python/tajo_util.py tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizationRule.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-docs/src/main/sphinx/functions.rst tajo-plan/src/main/proto/Plan.proto tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java tajo-core/src/test/java/org/apache/tajo/engine/function/TestPythonFunctions.java tajo-core/src/test/java/org/apache/tajo/engine/function/TestMathFunctions.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/CastEval.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/FunctionEval.java tajo-plan/src/main/java/org/apache/tajo/plan/function/python/TajoScriptEngine.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSplitProcessor.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc2.sql tajo-core/src/main/resources/python/controller.py tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeSerializer.java tajo-plan/src/main/java/org/apache/tajo/plan/function/PythonFunctionInvoke.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvokeContext.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSerializerDeserializer.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/src/test/resources/results/TestSelectQuery/testSelectWithPredicateOnPythonFunc.result tajo-plan/src/main/java/org/apache/tajo/plan/expr/PatternMatchPredicateEval.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java tajo-common/src/main/java/org/apache/tajo/storage/Tuple.java tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestFunctionDesc.java tajo-core/src/test/resources/python/_ init _.py tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/StreamingUtil.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-core/src/test/resources/python/test_funcs.pyc tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufLineReader.java tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java tajo-core/src/main/java/org/apache/tajo/engine/codegen/EvalCodeGenContext.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextFieldSerializerDeserializer.java tajo-core/src/main/java/org/apache/tajo/engine/codegen/LegacyFunctionBindingEmitter.java tajo-core/src/main/java/org/apache/tajo/engine/codegen/VariablesPreBuilder.java tajo-core/src/test/resources/results/TestSelectQuery/testNestedPythonFunction.result tajo-common/src/main/java/org/apache/tajo/json/DatumAdapter.java tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerializer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyFirstAggregationExec.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/WindowAggExec.java CHANGES tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance1.json tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeDeserializer.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineDeserializer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-storage/tajo-storage-common/src/test/java/org/apache/tajo/storage/TestLazyTuple.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/BetweenPredicateEval.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionInvocation.java tajo-core/src/test/resources/python/test_funcs.py tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionUtil.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonJoinExec.java tajo-core/src/test/resources/queries/TestSelectQuery/testNestedPythonFunction.sql tajo-core/src/test/resources/queries/TestSelectQuery/testSelectPythonFuncs.sql tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc.sql tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/GeneralFunctionEval.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbySecondAggregationExec.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineParsingError.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyThirdAggregationExec.java tajo-docs/src/main/sphinx/functions/json_func.rst tajo-core/src/test/resources/python/test_funcs2.py tajo-core/src/test/resources/queries/TestSelectQuery/testSelectWithPredicateOnPythonFunc.sql tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc.result tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerDe.java tajo-plan/src/main/java/org/apache/tajo/plan/ExprAnnotator.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/LineSplitProcessor.java tajo-core/src/main/resources/python/_ init _.py tajo-common/src/main/java/org/apache/tajo/datum/AnyDatum.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #311 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/311/)
          TAJO-1344: Python UDF support. (jihoon) (jihoonson: rev a7453853975dc8b5f4162ae16a1669aeb7ed995c)

          • tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java
          • tajo-core/src/main/resources/python/tajo_util.py
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/LogicalPlanEqualityTester.java
          • tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineDeserializer.java
          • tajo-common/src/main/java/org/apache/tajo/storage/Tuple.java
          • tajo-core/src/test/resources/python/_init_.py
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineDeserializer.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/CastEval.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizer.java
          • tajo-core/src/test/resources/python/test_funcs2.py
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/FunctionEval.java
          • tajo-core/src/test/resources/queries/TestSelectQuery/testSelectWithPredicateOnPythonFunc.sql
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerDe.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufInputChannel.java
          • tajo-core/src/test/resources/python/test_funcs.pyc
          • tajo-plan/src/main/proto/Plan.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java
          • tajo-core/src/test/java/org/apache/tajo/engine/function/TestMathFunctions.java
          • tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeDeserializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/global/rewriter/rules/GlobalPlanEqualityTester.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/WindowAggExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/ExprAnnotator.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java
          • tajo-core/src/test/resources/queries/TestSelectQuery/testNestedPythonFunction.sql
          • tajo-core/src/main/resources/python/controller.py
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/LegacyFunctionBindingEmitter.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/PatternMatchPredicateEval.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizationRule.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/OutputHandler.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSplitProcessor.java
          • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc2.sql
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionInvocation.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java
          • tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerializer.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testNestedPythonFunction.result
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/EvalCodeGenContext.java
          • tajo-core/src/main/java/org/apache/tajo/worker/Task.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/PythonFunctionInvoke.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc.result
          • tajo-common/src/main/java/org/apache/tajo/json/DatumAdapter.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbySecondAggregationExec.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java
          • tajo-core/src/test/resources/python/test_funcs.py
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/LineSplitProcessor.java
          • tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc2.result
          • tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java
          • tajo-core/src/test/resources/queries/TestSelectQuery/testSelectPythonFuncs.sql
          • tajo-common/src/main/java/org/apache/tajo/datum/AnyDatum.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSerializerDeserializer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufLineReader.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java
          • tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java
          • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestJsonSerDe/testVariousType.json
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java
          • tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java
          • tajo-core/src/main/java/org/apache/tajo/engine/codegen/VariablesPreBuilder.java
          • tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestFunctionDesc.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/function/TestPythonFunctions.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerDe.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvokeContext.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/TajoScriptEngine.java
          • tajo-common/src/main/java/org/apache/tajo/datum/BlobDatum.java
          • tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeSerializer.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/LegacyScalarFunctionInvoke.java
          • tajo-core/src/main/resources/python/_init_.py
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java
          • tajo-docs/src/main/sphinx/functions/json_func.rst
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/InputHandler.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextFieldSerializerDeserializer.java
          • tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyThirdAggregationExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalTreeUtil.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/Projector.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/BetweenPredicateEval.java
          • tajo-common/src/main/java/org/apache/tajo/util/datetime/DateTimeFormat.java
          • tajo-storage/tajo-storage-common/src/test/java/org/apache/tajo/storage/TestLazyTuple.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalNode.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvoke.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonJoinExec.java
          • tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerializer.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/PythonInvocationDesc.java
          • tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance1.json
          • tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java
          • tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc.sql
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineParsingError.java
          • tajo-common/src/main/java/org/apache/tajo/QueryVars.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectWithPredicateOnPythonFunc.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyFirstAggregationExec.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
          • tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionUtil.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/FieldEval.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/StreamingUtil.java
          • tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java
          • tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java
          • tajo-plan/src/main/java/org/apache/tajo/plan/expr/GeneralFunctionEval.java
          • tajo-docs/src/main/sphinx/functions.rst
          • tajo-core/src/test/resources/results/TestSelectQuery/testSelectPythonFuncs.result
          • tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/BufferPool.java
          • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #311 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/311/ ) TAJO-1344 : Python UDF support. (jihoon) (jihoonson: rev a7453853975dc8b5f4162ae16a1669aeb7ed995c) tajo-core/src/main/java/org/apache/tajo/master/TajoMaster.java tajo-plan/src/main/java/org/apache/tajo/plan/LogicalPlanner.java tajo-core/src/main/resources/python/tajo_util.py tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SeqScanExec.java tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/LogicalPlanEqualityTester.java tajo-common/src/main/java/org/apache/tajo/util/KeyValueSet.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineDeserializer.java tajo-common/src/main/java/org/apache/tajo/storage/Tuple.java tajo-core/src/test/resources/python/_ init _.py tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineDeserializer.java tajo-core/src/main/java/org/apache/tajo/worker/TajoWorker.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/CastEval.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizer.java tajo-core/src/test/resources/python/test_funcs2.py tajo-plan/src/main/java/org/apache/tajo/plan/expr/FunctionEval.java tajo-core/src/test/resources/queries/TestSelectQuery/testSelectWithPredicateOnPythonFunc.sql tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerDe.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufInputChannel.java tajo-core/src/test/resources/python/test_funcs.pyc tajo-plan/src/main/proto/Plan.proto tajo-plan/src/main/java/org/apache/tajo/plan/rewrite/rules/PartitionedTableRewriter.java tajo-core/src/test/java/org/apache/tajo/engine/function/TestMathFunctions.java tajo-core/src/main/java/org/apache/tajo/engine/query/QueryContext.java tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeDeserializer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/global/rewriter/rules/GlobalPlanEqualityTester.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/WindowAggExec.java tajo-plan/src/main/java/org/apache/tajo/plan/ExprAnnotator.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/AlgebraicUtil.java tajo-core/src/test/resources/queries/TestSelectQuery/testNestedPythonFunction.sql tajo-core/src/main/resources/python/controller.py tajo-core/src/main/java/org/apache/tajo/engine/codegen/LegacyFunctionBindingEmitter.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/PatternMatchPredicateEval.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/EvalTreeOptimizationRule.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/OutputHandler.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSplitProcessor.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java CHANGES tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc2.sql tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionInvocation.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/AggregationExec.java tajo-catalog/tajo-catalog-common/src/main/proto/CatalogProtos.proto tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineSerializer.java tajo-core/src/test/resources/results/TestSelectQuery/testNestedPythonFunction.result tajo-core/src/main/java/org/apache/tajo/engine/codegen/EvalCodeGenContext.java tajo-core/src/main/java/org/apache/tajo/worker/Task.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalContext.java tajo-plan/src/main/java/org/apache/tajo/plan/exprrewrite/rules/ConstantFolding.java tajo-plan/src/main/java/org/apache/tajo/plan/function/PythonFunctionInvoke.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc.result tajo-common/src/main/java/org/apache/tajo/json/DatumAdapter.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbySecondAggregationExec.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/MemStore.java tajo-core/src/test/resources/python/test_funcs.py tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/LineSplitProcessor.java tajo-core/src/test/resources/results/TestGroupByQuery/testGroupbyWithPythonFunc2.result tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java tajo-core/src/test/resources/queries/TestSelectQuery/testSelectPythonFuncs.sql tajo-common/src/main/java/org/apache/tajo/datum/AnyDatum.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/FieldSerializerDeserializer.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/ByteBufLineReader.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestSelectQuery.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyHashAggregationExec.java tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HavingExec.java tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestJsonSerDe/testVariousType.json tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/HashLeftOuterJoinExec.java tajo-common/src/main/java/org/apache/tajo/datum/DatumFactory.java tajo-core/src/main/java/org/apache/tajo/engine/codegen/VariablesPreBuilder.java tajo-catalog/tajo-catalog-common/src/test/java/org/apache/tajo/catalog/TestFunctionDesc.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/BSTIndexScanExec.java tajo-core/src/test/java/org/apache/tajo/engine/function/TestPythonFunctions.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerDe.java tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvokeContext.java tajo-plan/src/main/java/org/apache/tajo/plan/function/python/TajoScriptEngine.java tajo-common/src/main/java/org/apache/tajo/datum/BlobDatum.java tajo-core/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java tajo-plan/src/main/java/org/apache/tajo/plan/serder/EvalNodeSerializer.java tajo-plan/src/main/java/org/apache/tajo/plan/function/LegacyScalarFunctionInvoke.java tajo-core/src/main/resources/python/_ init _.py tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/CatalogUtil.java tajo-docs/src/main/sphinx/functions/json_func.rst tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/InputHandler.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextFieldSerializerDeserializer.java tajo-catalog/tajo-catalog-server/src/main/java/org/apache/tajo/catalog/store/AbstractDBStore.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyThirdAggregationExec.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalTreeUtil.java tajo-core/src/main/java/org/apache/tajo/engine/planner/Projector.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/BetweenPredicateEval.java tajo-common/src/main/java/org/apache/tajo/util/datetime/DateTimeFormat.java tajo-storage/tajo-storage-common/src/test/java/org/apache/tajo/storage/TestLazyTuple.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/EvalNode.java tajo-plan/src/main/java/org/apache/tajo/plan/function/FunctionInvoke.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/CommonJoinExec.java tajo-common/src/main/java/org/apache/tajo/util/FileUtil.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/catalog/FunctionDesc.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/CSVLineSerializer.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/PythonInvocationDesc.java tajo-storage/tajo-storage-hdfs/src/test/resources/dataset/TestDelimitedTextFile/testErrorTolerance1.json tajo-catalog/tajo-catalog-server/src/test/java/org/apache/tajo/catalog/TestCatalog.java tajo-core/src/test/resources/queries/TestGroupByQuery/testGroupbyWithPythonFunc.sql tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/TextLineParsingError.java tajo-common/src/main/java/org/apache/tajo/QueryVars.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTree.java tajo-core/src/test/resources/results/TestSelectQuery/testSelectWithPredicateOnPythonFunc.result tajo-plan/src/main/java/org/apache/tajo/plan/serder/LogicalNodeDeserializer.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/DistinctGroupbyFirstAggregationExec.java tajo-core/src/test/java/org/apache/tajo/engine/eval/ExprTestBase.java tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-catalog/tajo-catalog-common/src/main/java/org/apache/tajo/function/FunctionUtil.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/FieldEval.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/SelectionExec.java tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/StreamingUtil.java tajo-core/src/test/java/org/apache/tajo/engine/eval/TestEvalTreeUtil.java tajo-core/src/main/java/org/apache/tajo/master/exec/QueryExecutor.java tajo-plan/src/main/java/org/apache/tajo/plan/expr/GeneralFunctionEval.java tajo-docs/src/main/sphinx/functions.rst tajo-core/src/test/resources/results/TestSelectQuery/testSelectPythonFuncs.result tajo-plan/src/main/java/org/apache/tajo/plan/function/stream/BufferPool.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/EvalExprExec.java
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/526

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/526
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-94012400

          No, Looks good to me. Here is my +1
          Thank you for your work

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-94012400 No, Looks good to me. Here is my +1 Thank you for your work
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-94007286

          I've tested on my laptop.
          @jinossy, do you have any other opinions?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-94007286 I've tested on my laptop. @jinossy, do you have any other opinions?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-93957206

          I got your point and your plan. Please keep going. Here is my +1.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-93957206 I got your point and your plan. Please keep going. Here is my +1.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-93954054

          Thanks @hyunsik. I've changed the function name and removed ```QueryContextUtil``` according to your comment.

          Regarding on refactoring bind() function, I think that it would be better to work in another jira. For this issue, we first should decide what information are required to be passed to EvalNode. I booked another jira (https://issues.apache.org/jira/browse/TAJO-1566).

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-93954054 Thanks @hyunsik. I've changed the function name and removed ```QueryContextUtil``` according to your comment. Regarding on refactoring bind() function, I think that it would be better to work in another jira. For this issue, we first should decide what information are required to be passed to EvalNode. I booked another jira ( https://issues.apache.org/jira/browse/TAJO-1566 ).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-93902909

          I leaved some trivial comments. The patch looks good to me. It's a great job.

          Here is my additional comments. You try to change the signature EvalNode::bind to take EvalContext instance in order to get an instance of a launched script engine.

          Even though this refactoring is already a breaking change. EvalNode still need more information about task. In this chance, it would be great if we refactor EvalNode to take more meta context including TajoConf, task attempt information, and shared resources of workers. Then, we can probably remove OverridableConf parameter from all constructors of EvalNode.

          If you are concerned with a large patch, we can do this work in another jira. But, it will cause the second breaking change. We can choose either two breaking change or a large patch. It's up to you.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-93902909 I leaved some trivial comments. The patch looks good to me. It's a great job. Here is my additional comments. You try to change the signature EvalNode::bind to take EvalContext instance in order to get an instance of a launched script engine. Even though this refactoring is already a breaking change. EvalNode still need more information about task. In this chance, it would be great if we refactor EvalNode to take more meta context including TajoConf, task attempt information, and shared resources of workers. Then, we can probably remove OverridableConf parameter from all constructors of EvalNode. If you are concerned with a large patch, we can do this work in another jira. But, it will cause the second breaking change. We can choose either two breaking change or a large patch. It's up to you.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/526#discussion_r28571327

          — Diff: tajo-core/src/main/java/org/apache/tajo/worker/Task.java —
          @@ -135,7 +138,8 @@ public Task(String taskRunnerId,
          }

          public void initPlan() throws IOException {

          • plan = LogicalNodeDeserializer.deserialize(queryContext, request.getPlan());
            + QueryContextUtil.updatePythonScriptPath(systemConf, queryContext);
              • End diff –

          As I mentioned, script path is immutable config. GeneralFunctionEval can get this property from OverridenConf or TajoConf. You may not need this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/526#discussion_r28571327 — Diff: tajo-core/src/main/java/org/apache/tajo/worker/Task.java — @@ -135,7 +138,8 @@ public Task(String taskRunnerId, } public void initPlan() throws IOException { plan = LogicalNodeDeserializer.deserialize(queryContext, request.getPlan()); + QueryContextUtil.updatePythonScriptPath(systemConf, queryContext); End diff – As I mentioned, script path is immutable config. GeneralFunctionEval can get this property from OverridenConf or TajoConf. You may not need this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/526#discussion_r28571214

          — Diff: tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java —
          @@ -175,6 +174,7 @@ public Expr load(String sql) throws SQLSyntaxError {
          public SubmitQueryResponse executeQuery(Session session, String query, boolean isJson) {
          LOG.info("Query: " + query);
          QueryContext queryContext = createQueryContext(session);
          + QueryContextUtil.updatePythonScriptPath(context.getConf(), queryContext);
          — End diff –

          GeneralFunctionEval can take TajoConf instance, so it may be not necessary.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/526#discussion_r28571214 — Diff: tajo-core/src/main/java/org/apache/tajo/master/GlobalEngine.java — @@ -175,6 +174,7 @@ public Expr load(String sql) throws SQLSyntaxError { public SubmitQueryResponse executeQuery(Session session, String query, boolean isJson) { LOG.info("Query: " + query); QueryContext queryContext = createQueryContext(session); + QueryContextUtil.updatePythonScriptPath(context.getConf(), queryContext); — End diff – GeneralFunctionEval can take TajoConf instance, so it may be not necessary.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on a diff in the pull request:

          https://github.com/apache/tajo/pull/526#discussion_r28569644

          — Diff: tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java —
          @@ -66,7 +80,48 @@
          }
          }

          • return map.values();
            + return map;
            + }
            +
            + /**
            + * Load functions that are optionally defined by users.
            + *
            + * @param conf
            + * @param functionMap
            + * @return
            + * @throws IOException
            + */
            + public static Map<FunctionSignature, FunctionDesc> loadOptionalFunctions(TajoConf conf,
              • End diff –

          In my opinion, 'user-defined function' would be a better word than 'optional function'

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on a diff in the pull request: https://github.com/apache/tajo/pull/526#discussion_r28569644 — Diff: tajo-core/src/main/java/org/apache/tajo/engine/function/FunctionLoader.java — @@ -66,7 +80,48 @@ } } return map.values(); + return map; + } + + /** + * Load functions that are optionally defined by users. + * + * @param conf + * @param functionMap + * @return + * @throws IOException + */ + public static Map<FunctionSignature, FunctionDesc> loadOptionalFunctions(TajoConf conf, End diff – In my opinion, 'user-defined function' would be a better word than 'optional function'
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725855/TAJO-1344_8.patch
          against master revision release-0.9.0-rc0-256-g7b78668.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/736//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/736//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/736//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/736//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725855/TAJO-1344_8.patch against master revision release-0.9.0-rc0-256-g7b78668. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 26 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/736//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/736//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/736//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/736//console This message is automatically generated.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725784/TAJO-1344_7.patch
          against master revision release-0.9.0-rc0-255-g338a2b7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/734//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/734//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/734//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/734//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725784/TAJO-1344_7.patch against master revision release-0.9.0-rc0-255-g338a2b7. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 25 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/734//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/734//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/734//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/734//console This message is automatically generated.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12725782/TAJO-1344_6.patch
          against master revision release-0.9.0-rc0-255-g338a2b7.

          -1 patch. The patch command could not apply the patch.

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/733//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12725782/TAJO-1344_6.patch against master revision release-0.9.0-rc0-255-g338a2b7. -1 patch. The patch command could not apply the patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/733//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-93641173

          Thanks @jinossy and @hyunsik. I've fixed some findbugs warnings related to my patches.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-93641173 Thanks @jinossy and @hyunsik. I've fixed some findbugs warnings related to my patches.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-92194441

          Thank you all guys for your efforts. I also check only some design points.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-92194441 Thank you all guys for your efforts. I also check only some design points.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-92175869

          Great!! @jihoonson
          I will start review. If you fix the findbug warnings in your codes, It would be better

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-92175869 Great!! @jihoonson I will start review. If you fix the findbug warnings in your codes, It would be better
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12724795/TAJO-1344_5.patch
          against master revision release-0.9.0-rc0-250-g6136142.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 26 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 31 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs.

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/726//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-plan.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/726//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12724795/TAJO-1344_5.patch against master revision release-0.9.0-rc0-250-g6136142. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 26 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 31 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan tajo-storage/tajo-storage-common tajo-storage/tajo-storage-hdfs. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/726//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-plan.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/726//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-storage-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/726//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-91983935

          Ok. I think that this patch is ready for review.

          To address @hyunsik's comment, I added a class called ```FunctionInvoke```. This class describes how the functions are executed.

          On executing Python scripts, I used the approach of using an external UDF controller that is responsible for executing python scripts as commented above. When a submitted query involves one or more python UDFs, several UDF controllers are executed to compute UDFs. Input/output tuples are transmitted via stdio. This approach may have an issue on performance, but I think it is inevitable without using Jython.

          Currently, the controller is executed for each Python functions. That is, if a query involves 5 Python functions even some of them are same, at least 5 different controllers are executed during query processing. I chose this architecture due to its simplicity.

          Here are some highlights of changes.

          • ```AnyDatum``` is used to support Python's dynamic typing.
          • ```PythonScriptEngine``` is responsible for maintaining the external controller process. To reduce overhead, the controller should be forked only when UDFs are actually evaluated. In this patch, there are three points where the controller is forked.
          • Constant folding optimization in Tajo master: During constant folding, some UDFs can be evaluated. If necessary, controllers are forked and immediately destroyed after evaluation.
          • Non-from query execution in Tajo master. If the query involves Python UDFs, controllers are forked during query processing.
          • Task execution in worker: If the plan of a stage involves Python UDFs, controllers are forked (destroyed) when a task starts up (shuts down). Due to the simplicity, I chose this architecture rather than sharing controllers among multiple tasks via ```ExecutionBlockSharedResource```.
          • Refactoring the ```EvalNode::bind()``` function. This function now receives ```EvalContext``` in addition to ```Schema```. ```EvalContext``` can contain some information given at runtime such as ```ScriptEngine``` started by each task.

          For reviewers, I apologize for a large patch. But many changes are related to just refactoring of the bind() function and renaming some functions.
          Thanks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-91983935 Ok. I think that this patch is ready for review. To address @hyunsik's comment, I added a class called ```FunctionInvoke```. This class describes how the functions are executed. On executing Python scripts, I used the approach of using an external UDF controller that is responsible for executing python scripts as commented above. When a submitted query involves one or more python UDFs, several UDF controllers are executed to compute UDFs. Input/output tuples are transmitted via stdio. This approach may have an issue on performance, but I think it is inevitable without using Jython. Currently, the controller is executed for each Python functions. That is, if a query involves 5 Python functions even some of them are same, at least 5 different controllers are executed during query processing. I chose this architecture due to its simplicity. Here are some highlights of changes. ```AnyDatum``` is used to support Python's dynamic typing. ```PythonScriptEngine``` is responsible for maintaining the external controller process. To reduce overhead, the controller should be forked only when UDFs are actually evaluated. In this patch, there are three points where the controller is forked. Constant folding optimization in Tajo master: During constant folding, some UDFs can be evaluated. If necessary, controllers are forked and immediately destroyed after evaluation. Non-from query execution in Tajo master. If the query involves Python UDFs, controllers are forked during query processing. Task execution in worker: If the plan of a stage involves Python UDFs, controllers are forked (destroyed) when a task starts up (shuts down). Due to the simplicity, I chose this architecture rather than sharing controllers among multiple tasks via ```ExecutionBlockSharedResource```. Refactoring the ```EvalNode::bind()``` function. This function now receives ```EvalContext``` in addition to ```Schema```. ```EvalContext``` can contain some information given at runtime such as ```ScriptEngine``` started by each task. For reviewers, I apologize for a large patch. But many changes are related to just refactoring of the bind() function and renaming some functions. Thanks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/526#issuecomment-90801615

          There are still some problems with a cluster environment. I'm investigating it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/526#issuecomment-90801615 There are still some problems with a cluster environment. I'm investigating it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jihoonson opened a pull request:

          https://github.com/apache/tajo/pull/526

          TAJO-1344: Python UDF support

          I found some problems of version confliction when using Jython.
          So, I used another approach of using pipe.
          Most codes are borrowed from Pig.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jihoonson/tajo-2 TAJO-1344_3

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/526.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #526


          commit 349b1c23eb9653a7ef0206e9c2a94178953e134f
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2015-03-12T09:53:45Z

          Initial prototype.

          commit b8596e88c58840d2c80c09af604c156ea0073039
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2015-03-12T09:54:02Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into python_udf

          commit 3d4b2b9c5a425d4be027b6fc139f8c469c19a74d
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-23T12:41:51Z

          Merge branch 'python_udf' of https://github.com/hyunsik/tajo into TAJO-1344

          commit 104fa93d4e4fb8752679c94ea06c0357692a95d5
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-23T13:59:15Z

          TAJO-1344

          commit 9241d1ef5a95479772cf192f942905347c3729a4
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-24T08:05:31Z

          TAJO-1344

          commit 7808685f735c56ffb035971630a8865be37e4c7a
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T01:29:28Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit b3d09b82c0288ef6e9c6e85715c50bf5578eea0b
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T10:03:48Z

          TAJO-1344

          commit 06d4a27e65cc75aec2ed1239cdf2215dcf99c00d
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T13:34:40Z

          TAJO-1344

          commit 44afbe25f2021b939aa4498a6791e0f268b6d5e7
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T06:34:27Z

          TAJO-1344

          commit dae49bdbd6d13ff1503168591260569863683e3f
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T10:00:52Z

          TAJO-1344

          commit 88a2fd2da98b78f52d7b2288a68a2e5ac67ad757
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T13:32:24Z

          TAJO-1344

          commit b615a696ab42c75fd4832592772c7219b6a4dccc
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T14:22:10Z

          TAJO-1344

          commit 99ec66a0747413fb2e43a3b15dc142fb4a943b41
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T14:44:11Z

          TAJO-1344

          commit 3457b7830e12f8a8273b9d76f35198bacbee8f90
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T02:09:07Z

          Remove OptionalFunctionContext

          commit d7fa88a3feb96cc4bad800f77a07ac126fecac2e
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T03:07:08Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit 4741be6bd7faf96ed1cc8d22e430388db84c3e4e
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T03:20:09Z

          TAJO-1344

          commit 4a5570597b85f71af630d3c318de87aa03737ae7
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T09:01:40Z

          TAJO-1344

          commit 1d7dab11eb73be96e0c979de118e7e4e7321eaac
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T09:14:47Z

          TAJO-1344

          commit c2fd9b70ba9f5d57514db5ad4ea8316a94f99b32
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T09:54:14Z

          Add more comments

          commit b652351fb27001f94faa3e79afc56974f093df9d
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T14:23:09Z

          Add more tests, rename some variables, and add documentations

          commit e3bceb9ef7ee72a6d7521fe5e32be6f7775c1ccf
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T15:22:09Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit 422f280ecd92d88705c159b2822c49025805af99
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-28T16:20:25Z

          Fix test failure

          commit 14fde9a21a16e182908c4c94e0bccfa55f56bf9b
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-28T16:30:27Z

          Fix findbugs warning

          commit 7f162599f530fbfd60ca3562807db8851e25c120
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-29T11:09:20Z

          TAJO-1344

          commit 34767321066e2117b76809c44871c92823971bb3
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-29T11:49:16Z

          TAJO-1344

          commit 6a474b5b8d1aacbc3cc8e7191c1b87c50c8e3c6f
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-30T09:59:52Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit a71142381718f95714d23c39a40bc2897065491c
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-30T10:20:07Z

          TAJO-1344

          commit 3f2d3f035e2dc8da6d95aea351deafb618d5d725
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-04-02T08:07:17Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          Conflicts:
          tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java

          commit b85bebb15b2e5dc634e2741d4f87b6f41c9d26c9
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-04-02T11:06:33Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit 068d1e0134b029cb15bab9040a5ca224b5e5c17b
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-04-03T01:35:45Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/526 TAJO-1344 : Python UDF support I found some problems of version confliction when using Jython. So, I used another approach of using pipe. Most codes are borrowed from Pig. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1344 _3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/526.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #526 commit 349b1c23eb9653a7ef0206e9c2a94178953e134f Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-03-12T09:53:45Z Initial prototype. commit b8596e88c58840d2c80c09af604c156ea0073039 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-03-12T09:54:02Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into python_udf commit 3d4b2b9c5a425d4be027b6fc139f8c469c19a74d Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-23T12:41:51Z Merge branch 'python_udf' of https://github.com/hyunsik/tajo into TAJO-1344 commit 104fa93d4e4fb8752679c94ea06c0357692a95d5 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-23T13:59:15Z TAJO-1344 commit 9241d1ef5a95479772cf192f942905347c3729a4 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-24T08:05:31Z TAJO-1344 commit 7808685f735c56ffb035971630a8865be37e4c7a Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T01:29:28Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit b3d09b82c0288ef6e9c6e85715c50bf5578eea0b Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T10:03:48Z TAJO-1344 commit 06d4a27e65cc75aec2ed1239cdf2215dcf99c00d Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T13:34:40Z TAJO-1344 commit 44afbe25f2021b939aa4498a6791e0f268b6d5e7 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T06:34:27Z TAJO-1344 commit dae49bdbd6d13ff1503168591260569863683e3f Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T10:00:52Z TAJO-1344 commit 88a2fd2da98b78f52d7b2288a68a2e5ac67ad757 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T13:32:24Z TAJO-1344 commit b615a696ab42c75fd4832592772c7219b6a4dccc Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T14:22:10Z TAJO-1344 commit 99ec66a0747413fb2e43a3b15dc142fb4a943b41 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T14:44:11Z TAJO-1344 commit 3457b7830e12f8a8273b9d76f35198bacbee8f90 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T02:09:07Z Remove OptionalFunctionContext commit d7fa88a3feb96cc4bad800f77a07ac126fecac2e Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T03:07:08Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit 4741be6bd7faf96ed1cc8d22e430388db84c3e4e Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T03:20:09Z TAJO-1344 commit 4a5570597b85f71af630d3c318de87aa03737ae7 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T09:01:40Z TAJO-1344 commit 1d7dab11eb73be96e0c979de118e7e4e7321eaac Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T09:14:47Z TAJO-1344 commit c2fd9b70ba9f5d57514db5ad4ea8316a94f99b32 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T09:54:14Z Add more comments commit b652351fb27001f94faa3e79afc56974f093df9d Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T14:23:09Z Add more tests, rename some variables, and add documentations commit e3bceb9ef7ee72a6d7521fe5e32be6f7775c1ccf Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T15:22:09Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit 422f280ecd92d88705c159b2822c49025805af99 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-28T16:20:25Z Fix test failure commit 14fde9a21a16e182908c4c94e0bccfa55f56bf9b Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-28T16:30:27Z Fix findbugs warning commit 7f162599f530fbfd60ca3562807db8851e25c120 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-29T11:09:20Z TAJO-1344 commit 34767321066e2117b76809c44871c92823971bb3 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-29T11:49:16Z TAJO-1344 commit 6a474b5b8d1aacbc3cc8e7191c1b87c50c8e3c6f Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-30T09:59:52Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit a71142381718f95714d23c39a40bc2897065491c Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-30T10:20:07Z TAJO-1344 commit 3f2d3f035e2dc8da6d95aea351deafb618d5d725 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-04-02T08:07:17Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 Conflicts: tajo-core/src/test/java/org/apache/tajo/TajoTestingCluster.java commit b85bebb15b2e5dc634e2741d4f87b6f41c9d26c9 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-04-02T11:06:33Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit 068d1e0134b029cb15bab9040a5ca224b5e5c17b Author: Jihoon Son <jihoonson@apache.org> Date: 2015-04-03T01:35:45Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson closed the pull request at:

          https://github.com/apache/tajo/pull/473

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson closed the pull request at: https://github.com/apache/tajo/pull/473
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-90789940

          I'll open another PR.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-90789940 I'll open another PR.
          Hide
          hyunsik Hyunsik Choi added a comment - - edited

          Hi Jihoon Son,

          I don't have any free slot yet. My situation would be better after one week Thank you for taking the review on some issues.

          Show
          hyunsik Hyunsik Choi added a comment - - edited Hi Jihoon Son , I don't have any free slot yet. My situation would be better after one week Thank you for taking the review on some issues.
          Hide
          jihoonson Jihoon Son added a comment -

          Thanks @hyunsik. I finally agree with you.
          To avoid redundant work, I'd like to choose the second option. To do so, TAJO-1436 should be committed before this work.
          Unfortunately, you seem to not have much time. If you don't mind, may I review that issue?

          Show
          jihoonson Jihoon Son added a comment - Thanks @hyunsik. I finally agree with you. To avoid redundant work, I'd like to choose the second option. To do so, TAJO-1436 should be committed before this work. Unfortunately, you seem to not have much time. If you don't mind, may I review that issue?
          Hide
          hyunsik Hyunsik Choi added a comment -

          In addition, the purpose of EvalNode::Bind(Schema) is a preparation prior to eval(). Since all preparation fundamentally requires at least Schema, Bind() takes Schema. As you know, eval() took Schema as well as Tuple before TAJO-1436, and it prepares something at the first call and evaluate an expression. Bind() is a trial to separate the role of eval() into both preparation and evaluation.

          Show
          hyunsik Hyunsik Choi added a comment - In addition, the purpose of EvalNode::Bind(Schema) is a preparation prior to eval(). Since all preparation fundamentally requires at least Schema, Bind() takes Schema. As you know, eval() took Schema as well as Tuple before TAJO-1436 , and it prepares something at the first call and evaluate an expression. Bind() is a trial to separate the role of eval() into both preparation and evaluation.
          Hide
          hyunsik Hyunsik Choi added a comment - - edited

          Basing on the current implementation, I suggested the first option. After TAJO-1436, you can choose the second option.

          I think the following points:

          • The conventional wisdom is that a constructor should be lightweight.
          • function invocation implementation is mostly used in worker in order to evaluate expressions.
          • But, if function invocation implementation is initialized in the constructor, the initialization will be repeated several times in the following steps:
          • in logical planner
          • after plan deserialization in query master
          • after plan deserialization in task runner
          Show
          hyunsik Hyunsik Choi added a comment - - edited Basing on the current implementation, I suggested the first option. After TAJO-1436 , you can choose the second option. I think the following points: The conventional wisdom is that a constructor should be lightweight. function invocation implementation is mostly used in worker in order to evaluate expressions. But, if function invocation implementation is initialized in the constructor, the initialization will be repeated several times in the following steps: in logical planner after plan deserialization in query master after plan deserialization in task runner
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-87399705

          @hyunsik, thanks for the nice suggestion. It is definitely a better solution.
          However, I have a different opinion on detailed implementation.
          The first option that initialize each function invocation implementation at the first function call is deprecated, so it is not a good choice.
          And the second option using EvalNode::Bind() seems to initialize schema-dependent variables. Even though we still can change the specification of TAJO-1436, the current one looks good. Since relations have different schemas, schema-dependent variables should be initialized for each schema.

          However, function invocation implementation does not depend on relations' schema. I think that it would be better to initialize when an instance of ```FunctionEval``` is created.
          What do you think?

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-87399705 @hyunsik, thanks for the nice suggestion. It is definitely a better solution. However, I have a different opinion on detailed implementation. The first option that initialize each function invocation implementation at the first function call is deprecated, so it is not a good choice. And the second option using EvalNode::Bind() seems to initialize schema-dependent variables. Even though we still can change the specification of TAJO-1436 , the current one looks good. Since relations have different schemas, schema-dependent variables should be initialized for each schema. However, function invocation implementation does not depend on relations' schema. I think that it would be better to initialize when an instance of ```FunctionEval``` is created. What do you think?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-87281813

          The patch looks good to me.

          I have one suggestion. You make new EvalNode Type ```PYTHON_FUNCTION```. But, it may be not matched to the design of EvalNode, and it will require duplicated codes for EvalNode visitors.

          You seem to do because Python function involcation is different from existing Java function. I think that the best solution is to separate EvalNode and Function Invocation code. In more detail, you should implement some function invocation interface, and separate java function invocation and python function invocation. Each function invocation implementation can be initialized at the first function call, or you can use EvalNode::Bind(), being addressed in TAJO-1436.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-87281813 The patch looks good to me. I have one suggestion. You make new EvalNode Type ```PYTHON_FUNCTION```. But, it may be not matched to the design of EvalNode, and it will require duplicated codes for EvalNode visitors. You seem to do because Python function involcation is different from existing Java function. I think that the best solution is to separate EvalNode and Function Invocation code. In more detail, you should implement some function invocation interface, and separate java function invocation and python function invocation. Each function invocation implementation can be initialized at the first function call, or you can use EvalNode::Bind(), being addressed in TAJO-1436 .
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-87258850

          The test failure looks to be related with the version of Jython.
          I fixed it and verified with ```mvn clean install -Pparallel-test,hcatalog-0.12.0 -Dmaven.fork.count=2```.
          In addition, I fixed findbugs warnings.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-87258850 The test failure looks to be related with the version of Jython. I fixed it and verified with ```mvn clean install -Pparallel-test,hcatalog-0.12.0 -Dmaven.fork.count=2```. In addition, I fixed findbugs warnings.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-87158298

          I'll investigate the test failure.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-87158298 I'll investigate the test failure.
          Hide
          tajoqa Tajo QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12707780/TAJO-1344_3.patch
          against master revision release-0.9.0-rc0-223-g373d53c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 15 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

          +1 checkstyle. The patch generated 0 code style errors.

          -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan:
          org.apache.tajo.engine.query.TestTruncateTable
          org.apache.tajo.engine.query.TestCreateTable
          org.apache.tajo.engine.query.TestInsertQuery
          org.apache.tajo.worker.TestHistory
          org.apache.tajo.engine.eval.TestSQLDateTimeTypes
          org.apache.tajo.cli.tsql.commands.TestHdfsCommand
          org.apache.tajo.engine.query.TestSelectQuery
          org.apache.tajo.cli.tools.TestTajoDump
          org.apache.tajo.ha.TestHAServiceHDFSImpl
          org.apache.tajo.engine.query.TestTableSubQuery
          org.apache.tajo.engine.function.TestConditionalExpressions
          org.apache.tajo.engine.query.TestDropTable
          org.apache.tajo.benchmark.TestTPCH
          org.apache.tajo.engine.query.TestAlterTablespace
          org.apache.tajo.engine.query.TestUnionQuery
          org.apache.tajo.engine.query.TestSortQuery
          org.apache.tajo.engine.function.TestBuiltinFunctions
          org.apache.tajo.jdbc.TestResultSet
          org.apache.tajo.engine.query.TestHBaseTable
          org.apache.tajo.jdbc.TestTajoJdbc
          org.apache.tajo.engine.planner.TestQueryValidation
          org.apache.tajo.client.TestTajoClient
          org.apache.tajo.engine.query.TestCaseByCases
          org.apache.tajo.engine.eval.TestSQLExpression
          org.apache.tajo.engine.query.TestJoinQuery
          org.apache.tajo.engine.query.TestJoinBroadcast
          org.apache.tajo.engine.query.TestNetTypes
          org.apache.tajo.engine.query.TestSetSessionQuery
          org.apache.tajo.engine.query.TestWindowQuery
          org.apache.tajo.master.scheduler.TestFifoScheduler
          org.apache.tajo.engine.eval.TestIntervalType
          org.apache.tajo.engine.query.TestCreateDatabase
          org.apache.tajo.engine.query.TestTablePartitions
          org.apache.tajo.engine.eval.TestPredicates
          org.apache.tajo.engine.eval.TestEvalTree
          org.apache.tajo.engine.query.TestCTASQuery
          org.apache.tajo.cli.tsql.commands.TestExecExternalShellCommand
          org.apache.tajo.util.history.TestHistoryWriterReader
          org.apache.tajo.engine.function.TestMathFunctions
          org.apache.tajo.engine.function.TestPatternMatchingPredicates
          org.apache.tajo.cli.tsql.TestDefaultCliOutputFormatter
          org.apache.tajo.engine.function.TestDateTimeFunctions
          org.apache.tajo.engine.query.TestGroupByQuery
          org.apache.tajo.querymaster.TestQueryProgress
          org.apache.tajo.querymaster.TestTaskStatusUpdate
          org.apache.tajo.engine.function.TestPythonFunctions
          org.apache.tajo.engine.function.TestStringOperatorsAndFunctions
          org.apache.tajo.engine.query.TestNullValues
          org.apache.tajo.jdbc.TestTajoDatabaseMetaData
          org.apache.tajo.engine.planner.physical.TestSortExec
          org.apache.tajo.engine.query.TestAlterTable
          org.apache.tajo.engine.query.TestJoinOnPartitionedTables
          org.apache.tajo.storage.TestRowFile
          org.apache.tajo.engine.codegen.TestEvalCodeGenerator
          org.apache.tajo.cli.tsql.TestTajoCli
          org.apache.tajo.querymaster.TestKillQuery
          org.apache.tajo.master.TestNonForwardQueryResultSystemScanner

          Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/668//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/668//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html
          Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/668//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html
          Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/668//console

          This message is automatically generated.

          Show
          tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12707780/TAJO-1344_3.patch against master revision release-0.9.0-rc0-223-g373d53c. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 2.0.3) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in tajo-catalog/tajo-catalog-common tajo-catalog/tajo-catalog-server tajo-common tajo-core tajo-plan: org.apache.tajo.engine.query.TestTruncateTable org.apache.tajo.engine.query.TestCreateTable org.apache.tajo.engine.query.TestInsertQuery org.apache.tajo.worker.TestHistory org.apache.tajo.engine.eval.TestSQLDateTimeTypes org.apache.tajo.cli.tsql.commands.TestHdfsCommand org.apache.tajo.engine.query.TestSelectQuery org.apache.tajo.cli.tools.TestTajoDump org.apache.tajo.ha.TestHAServiceHDFSImpl org.apache.tajo.engine.query.TestTableSubQuery org.apache.tajo.engine.function.TestConditionalExpressions org.apache.tajo.engine.query.TestDropTable org.apache.tajo.benchmark.TestTPCH org.apache.tajo.engine.query.TestAlterTablespace org.apache.tajo.engine.query.TestUnionQuery org.apache.tajo.engine.query.TestSortQuery org.apache.tajo.engine.function.TestBuiltinFunctions org.apache.tajo.jdbc.TestResultSet org.apache.tajo.engine.query.TestHBaseTable org.apache.tajo.jdbc.TestTajoJdbc org.apache.tajo.engine.planner.TestQueryValidation org.apache.tajo.client.TestTajoClient org.apache.tajo.engine.query.TestCaseByCases org.apache.tajo.engine.eval.TestSQLExpression org.apache.tajo.engine.query.TestJoinQuery org.apache.tajo.engine.query.TestJoinBroadcast org.apache.tajo.engine.query.TestNetTypes org.apache.tajo.engine.query.TestSetSessionQuery org.apache.tajo.engine.query.TestWindowQuery org.apache.tajo.master.scheduler.TestFifoScheduler org.apache.tajo.engine.eval.TestIntervalType org.apache.tajo.engine.query.TestCreateDatabase org.apache.tajo.engine.query.TestTablePartitions org.apache.tajo.engine.eval.TestPredicates org.apache.tajo.engine.eval.TestEvalTree org.apache.tajo.engine.query.TestCTASQuery org.apache.tajo.cli.tsql.commands.TestExecExternalShellCommand org.apache.tajo.util.history.TestHistoryWriterReader org.apache.tajo.engine.function.TestMathFunctions org.apache.tajo.engine.function.TestPatternMatchingPredicates org.apache.tajo.cli.tsql.TestDefaultCliOutputFormatter org.apache.tajo.engine.function.TestDateTimeFunctions org.apache.tajo.engine.query.TestGroupByQuery org.apache.tajo.querymaster.TestQueryProgress org.apache.tajo.querymaster.TestTaskStatusUpdate org.apache.tajo.engine.function.TestPythonFunctions org.apache.tajo.engine.function.TestStringOperatorsAndFunctions org.apache.tajo.engine.query.TestNullValues org.apache.tajo.jdbc.TestTajoDatabaseMetaData org.apache.tajo.engine.planner.physical.TestSortExec org.apache.tajo.engine.query.TestAlterTable org.apache.tajo.engine.query.TestJoinOnPartitionedTables org.apache.tajo.storage.TestRowFile org.apache.tajo.engine.codegen.TestEvalCodeGenerator org.apache.tajo.cli.tsql.TestTajoCli org.apache.tajo.querymaster.TestKillQuery org.apache.tajo.master.TestNonForwardQueryResultSystemScanner Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/668//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/668//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-catalog-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/668//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-common.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/668//console This message is automatically generated.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-86959283

          Great! Really useful feature
          I'll review this weekend.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-86959283 Great! Really useful feature I'll review this weekend.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/473#issuecomment-86956557

          I've added comments and documentations.
          Please review the patch.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/473#issuecomment-86956557 I've added comments and documentations. Please review the patch.
          Hide
          jihoonson Jihoon Son added a comment -

          Added more comments.

          Show
          jihoonson Jihoon Son added a comment - Added more comments.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jihoonson opened a pull request:

          https://github.com/apache/tajo/pull/473

          TAJO-1344: Python UDF support

          To support Python's dynamic typing, I added AnyDatum which can contain a datum of any types.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/jihoonson/tajo-2 TAJO-1344

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/473.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #473


          commit 349b1c23eb9653a7ef0206e9c2a94178953e134f
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2015-03-12T09:53:45Z

          Initial prototype.

          commit b8596e88c58840d2c80c09af604c156ea0073039
          Author: Hyunsik Choi <hyunsik@apache.org>
          Date: 2015-03-12T09:54:02Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into python_udf

          commit 3d4b2b9c5a425d4be027b6fc139f8c469c19a74d
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-23T12:41:51Z

          Merge branch 'python_udf' of https://github.com/hyunsik/tajo into TAJO-1344

          commit 104fa93d4e4fb8752679c94ea06c0357692a95d5
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-23T13:59:15Z

          TAJO-1344

          commit 9241d1ef5a95479772cf192f942905347c3729a4
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-24T08:05:31Z

          TAJO-1344

          commit 7808685f735c56ffb035971630a8865be37e4c7a
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T01:29:28Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit b3d09b82c0288ef6e9c6e85715c50bf5578eea0b
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T10:03:48Z

          TAJO-1344

          commit 06d4a27e65cc75aec2ed1239cdf2215dcf99c00d
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-25T13:34:40Z

          TAJO-1344

          commit 44afbe25f2021b939aa4498a6791e0f268b6d5e7
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T06:34:27Z

          TAJO-1344

          commit dae49bdbd6d13ff1503168591260569863683e3f
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T10:00:52Z

          TAJO-1344

          commit 88a2fd2da98b78f52d7b2288a68a2e5ac67ad757
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T13:32:24Z

          TAJO-1344

          commit b615a696ab42c75fd4832592772c7219b6a4dccc
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T14:22:10Z

          TAJO-1344

          commit 99ec66a0747413fb2e43a3b15dc142fb4a943b41
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-26T14:44:11Z

          TAJO-1344

          commit 3457b7830e12f8a8273b9d76f35198bacbee8f90
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T02:09:07Z

          Remove OptionalFunctionContext

          commit d7fa88a3feb96cc4bad800f77a07ac126fecac2e
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T03:07:08Z

          Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344

          commit 4741be6bd7faf96ed1cc8d22e430388db84c3e4e
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T03:20:09Z

          TAJO-1344

          commit 4a5570597b85f71af630d3c318de87aa03737ae7
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T09:01:40Z

          TAJO-1344

          commit 1d7dab11eb73be96e0c979de118e7e4e7321eaac
          Author: Jihoon Son <jihoonson@apache.org>
          Date: 2015-03-27T09:14:47Z

          TAJO-1344


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jihoonson opened a pull request: https://github.com/apache/tajo/pull/473 TAJO-1344 : Python UDF support To support Python's dynamic typing, I added AnyDatum which can contain a datum of any types. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jihoonson/tajo-2 TAJO-1344 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/473.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #473 commit 349b1c23eb9653a7ef0206e9c2a94178953e134f Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-03-12T09:53:45Z Initial prototype. commit b8596e88c58840d2c80c09af604c156ea0073039 Author: Hyunsik Choi <hyunsik@apache.org> Date: 2015-03-12T09:54:02Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into python_udf commit 3d4b2b9c5a425d4be027b6fc139f8c469c19a74d Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-23T12:41:51Z Merge branch 'python_udf' of https://github.com/hyunsik/tajo into TAJO-1344 commit 104fa93d4e4fb8752679c94ea06c0357692a95d5 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-23T13:59:15Z TAJO-1344 commit 9241d1ef5a95479772cf192f942905347c3729a4 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-24T08:05:31Z TAJO-1344 commit 7808685f735c56ffb035971630a8865be37e4c7a Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T01:29:28Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit b3d09b82c0288ef6e9c6e85715c50bf5578eea0b Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T10:03:48Z TAJO-1344 commit 06d4a27e65cc75aec2ed1239cdf2215dcf99c00d Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-25T13:34:40Z TAJO-1344 commit 44afbe25f2021b939aa4498a6791e0f268b6d5e7 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T06:34:27Z TAJO-1344 commit dae49bdbd6d13ff1503168591260569863683e3f Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T10:00:52Z TAJO-1344 commit 88a2fd2da98b78f52d7b2288a68a2e5ac67ad757 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T13:32:24Z TAJO-1344 commit b615a696ab42c75fd4832592772c7219b6a4dccc Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T14:22:10Z TAJO-1344 commit 99ec66a0747413fb2e43a3b15dc142fb4a943b41 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-26T14:44:11Z TAJO-1344 commit 3457b7830e12f8a8273b9d76f35198bacbee8f90 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T02:09:07Z Remove OptionalFunctionContext commit d7fa88a3feb96cc4bad800f77a07ac126fecac2e Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T03:07:08Z Merge branch 'master' of https://git-wip-us.apache.org/repos/asf/tajo into TAJO-1344 commit 4741be6bd7faf96ed1cc8d22e430388db84c3e4e Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T03:20:09Z TAJO-1344 commit 4a5570597b85f71af630d3c318de87aa03737ae7 Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T09:01:40Z TAJO-1344 commit 1d7dab11eb73be96e0c979de118e7e4e7321eaac Author: Jihoon Son <jihoonson@apache.org> Date: 2015-03-27T09:14:47Z TAJO-1344
          Hide
          hyunsik Hyunsik Choi added a comment -

          I agree too. Static loading would be a good starting point.

          Show
          hyunsik Hyunsik Choi added a comment - I agree too. Static loading would be a good starting point.
          Hide
          jihoonson Jihoon Son added a comment -

          Right. The above grammar is just an example, and even wrong.
          Anywas, I'll proceed this ticket with the first suggestion.

          Show
          jihoonson Jihoon Son added a comment - Right. The above grammar is just an example, and even wrong. Anywas, I'll proceed this ticket with the first suggestion.
          Hide
          hyunsik Hyunsik Choi added a comment - - edited

          I think that we should separate the registration of program codes and the registration of function. This is because we can register more than one function from a program registration. So, we need to rethink the grammar of the second suggestion.

          Show
          hyunsik Hyunsik Choi added a comment - - edited I think that we should separate the registration of program codes and the registration of function. This is because we can register more than one function from a program registration. So, we need to rethink the grammar of the second suggestion.
          Hide
          yongjin.choi Yongjin Choi added a comment -

          +1 for the second.
          Tajo should have a grammar to register functions after all.
          Certainly, it would be even more useful.

          Show
          yongjin.choi Yongjin Choi added a comment - +1 for the second. Tajo should have a grammar to register functions after all. Certainly, it would be even more useful.
          Hide
          jihoonson Jihoon Son added a comment -

          Hi guys, I have a suggestion for this issue.
          To support python UDFs, we need to load python codes.
          Currently, I'm only considering static loading when the tajo cluster starts up for easy development.
          To load python codes, we have two choices as follows:

          • Add some configurations to specify the paths to python codes.
            • Example
              <property>
                <name>tajo.udf.python.path</name>
                <value>/path/to/python/code</value>
              </property>
              
          • Support a new grammar to register python functions.
            • Example
              CREATE EXTERNAL FUNCTION foo USING python LOCATION '/path/to/python/code'
              

          Definitely, the second option looks cool, but I'm worrying about the patch size.
          So, I suggest to proceed it in another Jira.

          Show
          jihoonson Jihoon Son added a comment - Hi guys, I have a suggestion for this issue. To support python UDFs, we need to load python codes. Currently, I'm only considering static loading when the tajo cluster starts up for easy development. To load python codes, we have two choices as follows: Add some configurations to specify the paths to python codes. Example <property> <name>tajo.udf.python.path</name> <value>/path/to/python/code</value> </property> Support a new grammar to register python functions. Example CREATE EXTERNAL FUNCTION foo USING python LOCATION '/path/to/python/code' Definitely, the second option looks cool, but I'm worrying about the patch size. So, I suggest to proceed it in another Jira.
          Hide
          jihoonson Jihoon Son added a comment -

          This is really cool.
          I will happily finish your work.
          Thanks!

          Show
          jihoonson Jihoon Son added a comment - This is really cool. I will happily finish your work. Thanks!
          Hide
          hyunsik Hyunsik Choi added a comment -

          I just did. Thank you again.

          Show
          hyunsik Hyunsik Choi added a comment - I just did. Thank you again.
          Hide
          hyunsik Hyunsik Choi added a comment -

          If so, I greatly appreciate you. Recently, I have no free time slot.

          Show
          hyunsik Hyunsik Choi added a comment - If so, I greatly appreciate you. Recently, I have no free time slot.
          Hide
          jihoonson Jihoon Son added a comment -

          May I take this issue?

          Show
          jihoonson Jihoon Son added a comment - May I take this issue?
          Hide
          hyunsik Hyunsik Choi added a comment -

          I borrowed some of pig code. Here is the first proto type.
          https://github.com/hyunsik/tajo/tree/python_udf

          Show
          hyunsik Hyunsik Choi added a comment - I borrowed some of pig code. Here is the first proto type. https://github.com/hyunsik/tajo/tree/python_udf
          Hide
          hyunsik Hyunsik Choi added a comment -

          I'm going to separate issue into udf and udaf.

          Show
          hyunsik Hyunsik Choi added a comment - I'm going to separate issue into udf and udaf.

            People

            • Assignee:
              jihoonson Jihoon Son
              Reporter:
              hyunsik Hyunsik Choi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development