Hive
  1. Hive
  2. HIVE-4160 Vectorized Query Execution in Hive
  3. HIVE-5604

Fix validation of nested expressions.

        Details

        • Type: Sub-task Sub-task
        • Status: Resolved
        • Priority: Major Major
        • Resolution: Fixed
        • Affects Version/s: None
        • Fix Version/s: 0.13.0
        • Component/s: Vectorization
        • Labels:
          None

          Description

          The bug fixes a few issues related to nested expressions:
          1) The nested expressions were not being validated at all.
          2) UDFRegExp was not handled correctly, but issue was not caught because of the previous issue.
          3) HIVE-5642 will not show up when this jira is fixed, but still added a sanity check to validate the number of arguments.

          1. Text File
            HIVE-5604.tez.patch
            18 kB
            Jitendra Nath Pandey
          2. Text File
            HIVE-5604.4.patch
            19 kB
            Jitendra Nath Pandey
          3. Text File
            HIVE-5604.3.patch
            18 kB
            Jitendra Nath Pandey
          4. Text File
            HIVE-5604.2.patch
            12 kB
            Jitendra Nath Pandey
          5. Text File
            HIVE-5604.1.patch
            6 kB
            Jitendra Nath Pandey

            Issue Links

            incorporates

            Sub-task - The sub-task of the issue HIVE-5642 Exception in UDFs with large number of arguments.

            • Major - Major loss of function.
            • Resolved
            is blocked by

            Sub-task - The sub-task of the issue HIVE-5430 Refactor VectorizationContext and handle NOT expression with nulls.

            • Major - Major loss of function.
            • Resolved

              Activity

              Descending order - Click to sort in ascending order
              Ashutosh Chauhan made changes -
              Status Patch Available [ 10002 ] Resolved [ 5 ]
              Fix Version/s 0.13.0 [ 12324986 ]
              Resolution Fixed [ 1 ]
              Hide
              Permalink
              Ashutosh Chauhan added a comment -

              Committed to trunk. Thanks, Jitendra!

              Show
              Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Jitendra!
              Hide
              Permalink
              Hive QA added a comment -

              Overall: +1 all checks pass

              Here are the results of testing the latest attachment:
              https://issues.apache.org/jira/secure/attachment/12610697/HIVE-5604.4.patch

              SUCCESS: +1 4513 tests passed

              Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1282/testReport
              Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1282/console

              Messages:

              Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase

              This message is automatically generated.

              Show
              Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12610697/HIVE-5604.4.patch SUCCESS: +1 4513 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1282/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1282/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
              Hide
              Permalink
              Ashutosh Chauhan added a comment -

              +1

              Show
              Ashutosh Chauhan added a comment - +1
              Jitendra Nath Pandey made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Jitendra Nath Pandey made changes -
              Attachment HIVE-5604.4.patch [ 12610697 ]
              Hide
              Permalink
              Jitendra Nath Pandey added a comment -

              The comment was removed because the descriptor is not a bit vector anymore, instead it is an immutable object. I have updated the patch to add additional comments.

              Show
              Jitendra Nath Pandey added a comment - The comment was removed because the descriptor is not a bit vector anymore, instead it is an immutable object. I have updated the patch to add additional comments.
              Jitendra Nath Pandey made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Hide
              Permalink
              Ashutosh Chauhan added a comment -

              Comment for Builder class of VectorExprDesc is still valid, right? Looks like you inadvertently removed it.

              Show
              Ashutosh Chauhan added a comment - Comment for Builder class of VectorExprDesc is still valid, right? Looks like you inadvertently removed it.
              Hide
              Permalink
              Jitendra Nath Pandey added a comment -

              Also, updated the description. The crux of the issue was that children expressions were not being validated and that was masking some other issues as well.

              Show
              Jitendra Nath Pandey added a comment - Also, updated the description. The crux of the issue was that children expressions were not being validated and that was masking some other issues as well.
              Hide
              Permalink
              Jitendra Nath Pandey added a comment -

              Review board entry: https://reviews.apache.org/r/14956/

              Show
              Jitendra Nath Pandey added a comment - Review board entry: https://reviews.apache.org/r/14956/
              Jitendra Nath Pandey made changes -
              Description The bug fixes a few issues related to nested expressions:
              1) The nested expressions were not being validated at all.
              2)               		The bug fixes a few issues related to nested expressions:
              1) The nested expressions were not being validated at all.
              2) UDFRegExp was not handled correctly, but issue was not caught because of the previous issue.
              3) HIVE-5642 will not show up when this jira is fixed, but still added a sanity check to validate the number of arguments.
              Jitendra Nath Pandey made changes -
              Description There is a bug that nested expressions are not being validated correctly. The bug fixes a few issues related to nested expressions:
              1) The nested expressions were not being validated at all.
              2)
              Hide
              Permalink
              Ashutosh Chauhan added a comment -

              Can you create a phabricator / review board entry? Also, can you describe a bug a bit.

              Show
              Ashutosh Chauhan added a comment - Can you create a phabricator / review board entry? Also, can you describe a bug a bit.
              Jitendra Nath Pandey made changes -
              Attachment HIVE-5604.tez.patch [ 12610395 ]
              Jitendra Nath Pandey made changes -
              Link This issue incorporates HIVE-5642 [ HIVE-5642 ]
              Jitendra Nath Pandey made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Jitendra Nath Pandey made changes -
              Attachment HIVE-5604.3.patch [ 12610279 ]
              Jitendra Nath Pandey made changes -
              Status Patch Available [ 10002 ] Open [ 1 ]
              Hide
              Permalink
              Hive QA added a comment -

              Overall: -1 at least one tests failed

              Here are the results of testing the latest attachment:
              https://issues.apache.org/jira/secure/attachment/12609952/HIVE-5604.2.patch

              ERROR: -1 due to 3 failed/errored test(s), 4471 tests executed
              Failed tests:

              org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1

              Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1227/testReport
              Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1227/console

              Messages:

              Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests failed with: TestsFailedException: 3 tests failed

              This message is automatically generated.

              Show
              Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12609952/HIVE-5604.2.patch ERROR: -1 due to 3 failed/errored test(s), 4471 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorization_short_regress org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_bucket_num_reducers org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_script_broken_pipe1 Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/1227/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/1227/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests failed with: TestsFailedException: 3 tests failed This message is automatically generated.
              Jitendra Nath Pandey made changes -
              Status Open [ 1 ] Patch Available [ 10002 ]
              Jitendra Nath Pandey made changes -
              Attachment HIVE-5604.2.patch [ 12609952 ]
              Jitendra Nath Pandey made changes -
              Link This issue is blocked by HIVE-5430 [ HIVE-5430 ]
              Jitendra Nath Pandey made changes -
              Field Original Value New Value
              Attachment HIVE-5604.1.patch [ 12609729 ]
              Hide
              Permalink
              Jitendra Nath Pandey added a comment -

              Attached patch. But patch will apply only after HIVE-5430 is committed.

              Show
              Jitendra Nath Pandey added a comment - Attached patch. But patch will apply only after HIVE-5430 is committed.
              Jitendra Nath Pandey created issue -

                People

                • Assignee:
                  Jitendra Nath Pandey
                  Reporter:
                  Jitendra Nath Pandey
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development