Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-4196

Certain builtin functions throw symbol does not exist error when used inside aggregate function

        Details

          Description

          Steps to reproduce this:

          [x.y.z.p:21000] > describe sample_07;
Query: describe sample_07
+-------------+--------+---------+
| name        | type   | comment |
+-------------+--------+---------+
| code        | string |         |
| description | string |         |
| total_emp   | int    |         |
| salary      | int    |         |
+-------------+--------+---------+
Fetched 4 row(s) in 0.02s

          [x.y.z.p:21000] > select count(bitand(total_emp,0)) from sample_07;
Query: select count(bitand(total_emp,0)) from sample_07
+-----------------------------+
| count(bitand(total_emp, 0)) |
+-----------------------------+
| 823                         |
+-----------------------------+
Fetched 1 row(s) in 0.53s
[x.y.z.p:21000] > select count(shiftleft(total_emp,0)) from sample_07;
Query: select count(shiftleft(total_emp,0)) from sample_07
WARNINGS: 
Builtin 'shiftleft' with symbol '_ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_6IntValES6_' does not exist. Verify that all your impalads are the same version.

          [x.y.z.p:21000] > select count(shiftright(total_emp,0)) from sample_07;
Query: select count(shiftright(total_emp,0)) from sample_07
WARNINGS: 
Builtin 'shiftright' with symbol '_ZN6impala16BitByteFunctions10ShiftRightEPN10impala_udf15FunctionContextERKNS1_6IntValES6_' does not exist. Verify that all your impalads are the same version.

            Issue Links

            is blocked by

            Improvement - An improvement or enhancement to an existing feature or task. IMPALA-4233 Revisit codegen in ScalarFnCall::Prepare()

            • Minor - Minor loss of function, or other problem where easy workaround is present.
            • Open

              Activity

              Ascending order - Click to sort in descending order
              Hide
              Permalink
              tarmstrong Tim Armstrong added a comment -

              Do you know if this is running with codegen enabled or disabled?

              Show
              tarmstrong Tim Armstrong added a comment - Do you know if this is running with codegen enabled or disabled?
              Hide
              Permalink
              tarmstrong Tim Armstrong added a comment -

              Also, what operating system?

              Show
              tarmstrong Tim Armstrong added a comment - Also, what operating system?
              Hide
              Permalink
              mala_ck Mala Chikka Kempanna added a comment -

              DISABLE_CODEGEN: [0]
              2.6.32-358.el6.x86_64

              Show
              mala_ck Mala Chikka Kempanna added a comment - DISABLE_CODEGEN: [0] 2.6.32-358.el6.x86_64
              Hide
              Permalink
              tarmstrong Tim Armstrong added a comment -

              I was able to reproduce this. It's very strange since the symbol seems to be defined in the binary's symbol table:

              tarmstrong@tarmstrong-box:~/impala-4196-shiftright$ ls
etc  impala-2.6.0+cdh5.8.0+0-1.cdh5.8.0.p0.119.el6.x86_64.rpm  usr  var
tarmstrong@tarmstrong-box:~/impala-4196-shiftright$ objdump -T ./usr/lib/impala/sbin-retail/impalad  | grep ShiftLeft
0000000000878350 g    DF .text	0000000000000039  Base        _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_10TinyIntValERKNS1_6IntValE
0000000001a8a9b0 g    DF .text	000000000000009b  Base        _ZN4llvm5APInt11tcShiftLeftEPmjj
0000000000878410 g    DF .text	000000000000003a  Base        _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_9BigIntValERKNS1_6IntValE
00000000008783d0 g    DF .text	0000000000000038  Base        _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_6IntValES6_
0000000000878390 g    DF .text	0000000000000039  Base        _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_11SmallIntValERKNS1_6IntValE

              Also the function can be called if it's outside the count():

              select shiftleft(total_emp,0) from sample_07;

1	134354250
2	6003930
3	299160
4	1655410
5	61110
6	36300
7	165240
8	322170
9	47210
10	239360
11	264990
12	484390
13	41780
14	28170
15	58100
              Show
              tarmstrong Tim Armstrong added a comment - I was able to reproduce this. It's very strange since the symbol seems to be defined in the binary's symbol table: tarmstrong@tarmstrong-box:~/impala-4196-shiftright$ ls etc impala-2.6.0+cdh5.8.0+0-1.cdh5.8.0.p0.119.el6.x86_64.rpm usr var tarmstrong@tarmstrong-box:~/impala-4196-shiftright$ objdump -T ./usr/lib/impala/sbin-retail/impalad | grep ShiftLeft 0000000000878350 g DF .text 0000000000000039 Base _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_10TinyIntValERKNS1_6IntValE 0000000001a8a9b0 g DF .text 000000000000009b Base _ZN4llvm5APInt11tcShiftLeftEPmjj 0000000000878410 g DF .text 000000000000003a Base _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_9BigIntValERKNS1_6IntValE 00000000008783d0 g DF .text 0000000000000038 Base _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_6IntValES6_ 0000000000878390 g DF .text 0000000000000039 Base _ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_11SmallIntValERKNS1_6IntValE Also the function can be called if it's outside the count(): select shiftleft(total_emp,0) from sample_07; 1 134354250 2 6003930 3 299160 4 1655410 5 61110 6 36300 7 165240 8 322170 9 47210 10 239360 11 264990 12 484390 13 41780 14 28170 15 58100
              Hide
              Permalink
              tarmstrong Tim Armstrong added a comment -

              Ok, I can reproduce this on master:

              [tarmstrong-box.ca.cloudera.com:21000] > select count(shiftleft(int_col, 1)) from functional_parquet.alltypes;
Query: select count(shiftleft(int_col, 1)) from functional_parquet.alltypes
Query submitted at: 2016-09-23 16:28:40 (Coordinator: http://tarmstrong-box:25000)
Query progress can be monitored at: http://tarmstrong-box:25000/query_plan?query_id=c84ea7a023f46364:4c2b82cf00000000
WARNINGS: 
Builtin 'shiftleft' with symbol '_ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_6IntValES6_' does not exist. Verify that all your impalads are the same version.
              Show
              tarmstrong Tim Armstrong added a comment - Ok, I can reproduce this on master: [tarmstrong-box.ca.cloudera.com:21000] > select count(shiftleft(int_col, 1)) from functional_parquet.alltypes; Query: select count(shiftleft(int_col, 1)) from functional_parquet.alltypes Query submitted at: 2016-09-23 16:28:40 (Coordinator: http: //tarmstrong-box:25000) Query progress can be monitored at: http: //tarmstrong-box:25000/query_plan?query_id=c84ea7a023f46364:4c2b82cf00000000 WARNINGS: Builtin 'shiftleft' with symbol '_ZN6impala16BitByteFunctions9ShiftLeftEPN10impala_udf15FunctionContextERKNS1_6IntValES6_' does not exist. Verify that all your impalads are the same version.
              Hide
              Permalink
              bharathv bharath v added a comment -

              Commit: 0a96ead664b8866a128b370fab9b793e28685df8
              http://github.mtv.cloudera.com/CDH/Impala/commit/0a96ead664b8866a128b370fab9b793e28685df8
              Author: Bharath Vissapragada <bharathv@cloudera.com>
              Date: 2016-09-30 (Fri, 30 Sep 2016)

              Changed paths:
              M be/src/codegen/impala-ir.cc
              M be/src/exprs/CMakeLists.txt
              A be/src/exprs/bit-byte-functions-ir.cc
              R be/src/exprs/bit-byte-functions.cc
              M testdata/workloads/functional-query/queries/QueryTest/exprs.test

              Log Message:
              -----------
              IMPALA-4196: Cross compile bit-byte-functions

              Change-Id: I5a1291bfd202b500405a884e4a62f0ca2447244a
              Reviewed-on: http://gerrit.cloudera.org:8080/4557
              Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com>
              Tested-by: Internal Jenkins

              Show
              bharathv bharath v added a comment - Commit: 0a96ead664b8866a128b370fab9b793e28685df8 http://github.mtv.cloudera.com/CDH/Impala/commit/0a96ead664b8866a128b370fab9b793e28685df8 Author: Bharath Vissapragada <bharathv@cloudera.com> Date: 2016-09-30 (Fri, 30 Sep 2016) Changed paths: M be/src/codegen/impala-ir.cc M be/src/exprs/CMakeLists.txt A be/src/exprs/bit-byte-functions-ir.cc R be/src/exprs/bit-byte-functions.cc M testdata/workloads/functional-query/queries/QueryTest/exprs.test Log Message: ----------- IMPALA-4196 : Cross compile bit-byte-functions Change-Id: I5a1291bfd202b500405a884e4a62f0ca2447244a Reviewed-on: http://gerrit.cloudera.org:8080/4557 Reviewed-by: Bharath Vissapragada <bharathv@cloudera.com> Tested-by: Internal Jenkins

                People

                • Assignee:
                  bharathv bharath v
                  Reporter:
                  mala_ck Mala Chikka Kempanna
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  5 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Development