Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1596

TestPythonFunctions occasionally fails

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Function/UDF
    • Labels:
      None

      Description

      Here is failure test.
      https://travis-ci.org/apache/tajo/builds/61730967

      stacktrace:

      Tests run: 2, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 1.209 sec <<< FAILURE! - in org.apache.tajo.engine.function.TestPythonFunctions
      testNestedFunctions(org.apache.tajo.engine.function.TestPythonFunctions)  Time elapsed: 0.151 sec  <<< ERROR!
      java.lang.RuntimeException: Problem getting output
      	at org.apache.tajo.plan.function.python.PythonScriptEngine.callScalarFunc(PythonScriptEngine.java:479)
      	at org.apache.tajo.plan.function.PythonFunctionInvoke.eval(PythonFunctionInvoke.java:50)
      	at org.apache.tajo.plan.expr.GeneralFunctionEval.eval(GeneralFunctionEval.java:61)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:101)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:32)
      	at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:69)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:68)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:32)
      	at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:38)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:88)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:32)
      	at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:69)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.optimize(ConstantFolding.java:40)
      	at org.apache.tajo.plan.exprrewrite.EvalTreeOptimizer.optimize(EvalTreeOptimizer.java:75)
      	at org.apache.tajo.plan.ExprAnnotator.createEvalNode(ExprAnnotator.java:91)
      	at org.apache.tajo.plan.LogicalPlanner.buildPlanForNoneFromStatement(LogicalPlanner.java:433)
      	at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:230)
      	at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:66)
      	at org.apache.tajo.plan.algebra.BaseAlgebraVisitor.visit(BaseAlgebraVisitor.java:59)
      	at org.apache.tajo.plan.LogicalPlanner.createPlan(LogicalPlanner.java:148)
      	at org.apache.tajo.engine.eval.ExprTestBase.getRawTargets(ExprTestBase.java:151)
      	at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:277)
      	at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:202)
      	at org.apache.tajo.engine.eval.ExprTestBase.testSimpleEval(ExprTestBase.java:182)
      	at org.apache.tajo.engine.function.TestPythonFunctions.testNestedFunctions(TestPythonFunctions.java:41)
      testFunctions(org.apache.tajo.engine.function.TestPythonFunctions)  Time elapsed: 0.063 sec  <<< ERROR!
      java.lang.RuntimeException: Problem getting output
      	at org.apache.tajo.plan.function.python.PythonScriptEngine.callScalarFunc(PythonScriptEngine.java:479)
      	at org.apache.tajo.plan.function.PythonFunctionInvoke.eval(PythonFunctionInvoke.java:50)
      	at org.apache.tajo.plan.expr.GeneralFunctionEval.eval(GeneralFunctionEval.java:61)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:101)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:32)
      	at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:69)
      	at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.optimize(ConstantFolding.java:40)
      	at org.apache.tajo.plan.exprrewrite.EvalTreeOptimizer.optimize(EvalTreeOptimizer.java:75)
      	at org.apache.tajo.plan.ExprAnnotator.createEvalNode(ExprAnnotator.java:91)
      	at org.apache.tajo.plan.LogicalPlanner.buildPlanForNoneFromStatement(LogicalPlanner.java:433)
      	at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:230)
      	at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:66)
      	at org.apache.tajo.plan.algebra.BaseAlgebraVisitor.visit(BaseAlgebraVisitor.java:59)
      	at org.apache.tajo.plan.LogicalPlanner.createPlan(LogicalPlanner.java:148)
      	at org.apache.tajo.engine.eval.ExprTestBase.getRawTargets(ExprTestBase.java:151)
      	at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:277)
      	at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:202)
      	at org.apache.tajo.engine.eval.ExprTestBase.testSimpleEval(ExprTestBase.java:182)
      	at org.apache.tajo.engine.function.TestPythonFunctions.testFunctions(TestPythonFunctions.java:30)
      Running org.apache.tajo.engine.function.TestBuiltinFunctions
      

        Activity

        Hide
        jhkim Jinho Kim added a comment -

        Jihoon Son
        I'd like failure test on travis. assign to me please ?

        Show
        jhkim Jinho Kim added a comment - Jihoon Son I'd like failure test on travis. assign to me please ?
        Hide
        jihoonson Jihoon Son added a comment -

        Thanks Jinho Kim.
        Nowadays, I'm not working on this issue.

        Show
        jihoonson Jihoon Son added a comment - Thanks Jinho Kim . Nowadays, I'm not working on this issue.
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/705

        TAJO-1596: TestPythonFunctions occasionally fails

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1596

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/705.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #705


        commit e886d5ae08b0358710ad2cf6b8b6da80cd7365e8
        Author: Jinho Kim <jhkim@apache.org>
        Date: 2015-08-20T08:18:00Z

        TAJO-1596


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/705 TAJO-1596 : TestPythonFunctions occasionally fails You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1596 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/705.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #705 commit e886d5ae08b0358710ad2cf6b8b6da80cd7365e8 Author: Jinho Kim <jhkim@apache.org> Date: 2015-08-20T08:18:00Z TAJO-1596
        Hide
        jhkim Jinho Kim added a comment -

        Thanks

        Show
        jhkim Jinho Kim added a comment - Thanks
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jinossy commented on the pull request:

        https://github.com/apache/tajo/pull/705#issuecomment-132990185

        I found problem. It caused by 'No such file' of logger path
        I’m going to remove the TODO line
        ```
        org.apache.tajo.engine.function.TestPythonFunctions

        testNestedFunctions(org.apache.tajo.engine.function.TestPythonFunctions) Time elapsed: 1.347 sec <<< ERROR!

        java.lang.RuntimeException: Python exception caused by: Traceback (most recent call last):

        File "/tmp/controller5680807805615667393.py", line 472, in <module>

        sys.argv[5], sys.argv[6], sys.argv[7], sys.argv[8], sys.argv[9])

        File "/tmp/controller5680807805615667393.py", line 117, in main

        self.log_stream = open(output_stream_path, 'a')

        IOError: [Errno 2] No such file or directory: '/tmp/tajo-travis/pythonreturn_one.out'

        at org.apache.tajo.plan.function.python.PythonScriptEngine.throwException(PythonScriptEngine.java:554)

        at org.apache.tajo.plan.function.python.PythonScriptEngine.callScalarFunc(PythonScriptEngine.java:503)

        at org.apache.tajo.plan.function.PythonFunctionInvoke.eval(PythonFunctionInvoke.java:50)

        at org.apache.tajo.plan.expr.GeneralFunctionEval.eval(GeneralFunctionEval.java:60)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:105)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:35)

        at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:70)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:69)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:35)

        at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:39)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:92)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:35)

        at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:70)

        at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.optimize(ConstantFolding.java:41)

        at org.apache.tajo.plan.exprrewrite.EvalTreeOptimizer.optimize(EvalTreeOptimizer.java:75)

        at org.apache.tajo.plan.ExprAnnotator.createEvalNode(ExprAnnotator.java:93)

        at org.apache.tajo.plan.LogicalPlanner.buildPlanForNoneFromStatement(LogicalPlanner.java:454)

        at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:250)

        at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:76)

        at org.apache.tajo.plan.algebra.BaseAlgebraVisitor.visit(BaseAlgebraVisitor.java:60)

        at org.apache.tajo.plan.LogicalPlanner.createPlan(LogicalPlanner.java:170)

        at org.apache.tajo.engine.eval.ExprTestBase.getRawTargets(ExprTestBase.java:150)

        at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:280)

        at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:201)

        at org.apache.tajo.engine.eval.ExprTestBase.testSimpleEval(ExprTestBase.java:181)
        ```

        Show
        githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/705#issuecomment-132990185 I found problem. It caused by 'No such file' of logger path I’m going to remove the TODO line ``` org.apache.tajo.engine.function.TestPythonFunctions testNestedFunctions(org.apache.tajo.engine.function.TestPythonFunctions) Time elapsed: 1.347 sec <<< ERROR! java.lang.RuntimeException: Python exception caused by: Traceback (most recent call last): File "/tmp/controller5680807805615667393.py", line 472, in <module> sys.argv [5] , sys.argv [6] , sys.argv [7] , sys.argv [8] , sys.argv [9] ) File "/tmp/controller5680807805615667393.py", line 117, in main self.log_stream = open(output_stream_path, 'a') IOError: [Errno 2] No such file or directory: '/tmp/tajo-travis/pythonreturn_one.out' at org.apache.tajo.plan.function.python.PythonScriptEngine.throwException(PythonScriptEngine.java:554) at org.apache.tajo.plan.function.python.PythonScriptEngine.callScalarFunc(PythonScriptEngine.java:503) at org.apache.tajo.plan.function.PythonFunctionInvoke.eval(PythonFunctionInvoke.java:50) at org.apache.tajo.plan.expr.GeneralFunctionEval.eval(GeneralFunctionEval.java:60) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:105) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:35) at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:70) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:69) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitUnaryEval(ConstantFolding.java:35) at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:39) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:92) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.visitFuncCall(ConstantFolding.java:35) at org.apache.tajo.plan.expr.SimpleEvalNodeVisitor.visit(SimpleEvalNodeVisitor.java:70) at org.apache.tajo.plan.exprrewrite.rules.ConstantFolding.optimize(ConstantFolding.java:41) at org.apache.tajo.plan.exprrewrite.EvalTreeOptimizer.optimize(EvalTreeOptimizer.java:75) at org.apache.tajo.plan.ExprAnnotator.createEvalNode(ExprAnnotator.java:93) at org.apache.tajo.plan.LogicalPlanner.buildPlanForNoneFromStatement(LogicalPlanner.java:454) at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:250) at org.apache.tajo.plan.LogicalPlanner.visitProjection(LogicalPlanner.java:76) at org.apache.tajo.plan.algebra.BaseAlgebraVisitor.visit(BaseAlgebraVisitor.java:60) at org.apache.tajo.plan.LogicalPlanner.createPlan(LogicalPlanner.java:170) at org.apache.tajo.engine.eval.ExprTestBase.getRawTargets(ExprTestBase.java:150) at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:280) at org.apache.tajo.engine.eval.ExprTestBase.testEval(ExprTestBase.java:201) at org.apache.tajo.engine.eval.ExprTestBase.testSimpleEval(ExprTestBase.java:181) ```
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/705#issuecomment-132990884

        It's really nice work. Thank you for your effort.

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/705#issuecomment-132990884 It's really nice work. Thank you for your effort.
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/705#issuecomment-133027594

        You made it!

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/705#issuecomment-133027594 You made it!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/705#issuecomment-133265752

        +1 ship it!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/705#issuecomment-133265752 +1 ship it!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/705

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/705
        Hide
        jhkim Jinho Kim added a comment -

        committed it
        Thanks for your quick review!

        Show
        jhkim Jinho Kim added a comment - committed it Thanks for your quick review!
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #460 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/460/)
        TAJO-1596: TestPythonFunctions occasionally fails. (jhkim: rev f21d5d677c2ba1b9c493498ecb512d1ddcfe2221)

        • tajo-core/src/main/resources/python/controller.py
        • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
        • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
        • CHANGES
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #460 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/460/ ) TAJO-1596 : TestPythonFunctions occasionally fails. (jhkim: rev f21d5d677c2ba1b9c493498ecb512d1ddcfe2221) tajo-core/src/main/resources/python/controller.py tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java CHANGES
        Hide
        hudson Hudson added a comment -

        ABORTED: Integrated in Tajo-master-build #815 (See https://builds.apache.org/job/Tajo-master-build/815/)
        TAJO-1596: TestPythonFunctions occasionally fails. (jhkim: rev f21d5d677c2ba1b9c493498ecb512d1ddcfe2221)

        • tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java
        • tajo-core/src/main/resources/python/controller.py
        • tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java
        • CHANGES
        Show
        hudson Hudson added a comment - ABORTED: Integrated in Tajo-master-build #815 (See https://builds.apache.org/job/Tajo-master-build/815/ ) TAJO-1596 : TestPythonFunctions occasionally fails. (jhkim: rev f21d5d677c2ba1b9c493498ecb512d1ddcfe2221) tajo-plan/src/main/java/org/apache/tajo/plan/function/python/PythonScriptEngine.java tajo-core/src/main/resources/python/controller.py tajo-core/src/main/java/org/apache/tajo/master/QueryManager.java CHANGES

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development