Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8893

Implement whitelist for builtin UDFs to avoid untrused code execution in multiuser mode

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.0
    • Fix Version/s: 1.1.0
    • Component/s: Authorization, HiveServer2, SQL
    • Labels:
      None

      Description

      The udfs like reflect() or java_method() enables executing a java method as udf. While this offers lot of flexibility in the standalone mode, it can become a security loophole in a secure multiuser environment. For example, in HiveServer2 one can execute any available java code with user hive's credentials.
      We need a whitelist and blacklist to restrict builtin udfs in Hiveserver2.

      1. HIVE-8893.3.patch
        27 kB
        Prasad Mujumdar
      2. HIVE-8893.4.patch
        27 kB
        Prasad Mujumdar
      3. HIVE-8893.5.patch
        27 kB
        Prasad Mujumdar
      4. HIVE-8893.6.patch
        32 kB
        Prasad Mujumdar

        Issue Links

          Activity

          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12681842/HIVE-8893.2.patch

          ERROR: -1 due to 119 failed/errored test(s), 6650 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby_empty
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_semijoin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_stats
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_exists
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_udaf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_views
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_windowing
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table_json
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_function
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_creation
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual2
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual4
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_within_subquery
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_general_queries
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reducesink_dedup
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rename_column
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tables
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_explain_rewrite
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_windowing_expressions
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_compare_java_string
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_index
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_stddev_pop
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_adjust_rowcontainer_sz
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_columnPruning
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_decimal
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_streaming
          org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_row_sequence
          org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udtf_output_on_close
          org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence
          org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_invalid_row_sequence
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_windowing
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table
          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_func_nonexistent
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_function_failure
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_AggrFuncsWithNoGBYNoPartDef
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_AmbiguousWindowDefn
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_HavingLeadWithNoGBYNoWindowing
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_HavingLeadWithPTF
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_InvalidValueBoundary
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_WhereWithRankCond
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_nested_subquery
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_windowing_corr
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_invalid
          org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_windowing_ll_no_neg
          org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistent
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_ambiguous_join_col
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_insert_wrong_number_columns
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_dot
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_function_param2
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_index
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_nonkey_groupby
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column1
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column2
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column3
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column4
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column5
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column6
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function1
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function2
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function3
          org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function4
          org.apache.hive.jdbc.TestJdbcDriver2.testErrorMessages
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1817/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1817/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1817/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 119 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12681842 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12681842/HIVE-8893.2.patch ERROR: -1 due to 119 failed/errored test(s), 6650 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_index org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_gby_empty org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_limit org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_semijoin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_simple_select org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_stats org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_exists org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_subq_not_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_udf_udaf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_union org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_views org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_windowing org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_func1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_create_view org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_describe_table_json org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_drop_function org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_index_creation org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_input3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_alt_syntax org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual1 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual2 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual3 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_cond_pushdown_unqual4 org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_merging org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_leadlag_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_order_within_subquery org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_general_queries org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_reducesink_dedup org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_rename_column org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_functions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_show_tables org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_in_explain_rewrite org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_notin_having org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_unqualcolumnrefs org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_temp_table_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_compare_java_string org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_index org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_stddev_pop org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_mapjoin_reduce org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_adjust_rowcontainer_sz org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_columnPruning org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_decimal org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_expressions org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_windowing_streaming org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udf_row_sequence org.apache.hadoop.hive.cli.TestContribCliDriver.testCliDriver_udtf_output_on_close org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_case_with_row_sequence org.apache.hadoop.hive.cli.TestContribNegativeCliDriver.testNegativeCliDriver_invalid_row_sequence org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_gby_empty org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_join org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_limit org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_semijoin org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_simple_select org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_stats org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_exists org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_subq_not_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_udf_udaf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_union org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_views org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_cbo_windowing org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_subquery_in org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_temp_table org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_mapjoin_reduce org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_func_nonexistent org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_drop_function_failure org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_AggrFuncsWithNoGBYNoPartDef org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_AmbiguousWindowDefn org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_HavingLeadWithNoGBYNoWindowing org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_HavingLeadWithPTF org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_InvalidValueBoundary org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_ptf_negative_WhereWithRankCond org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_nested_subquery org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_subquery_windowing_corr org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_udf_invalid org.apache.hadoop.hive.cli.TestNegativeCliDriver.testNegativeCliDriver_windowing_ll_no_neg org.apache.hadoop.hive.ql.parse.TestMacroSemanticAnalyzer.testDropMacroNonExistent org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_ambiguous_join_col org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_insert_wrong_number_columns org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_dot org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_function_param2 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_invalid_index org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_nonkey_groupby org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column1 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column2 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column3 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column4 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column5 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_column6 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function1 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function2 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function3 org.apache.hadoop.hive.ql.parse.TestParseNegative.testParseNegative_unknown_function4 org.apache.hive.jdbc.TestJdbcDriver2.testErrorMessages Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1817/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1817/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1817/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 119 tests failed This message is automatically generated. ATTACHMENT ID: 12681842 - PreCommit-HIVE-TRUNK-Build
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12682018/HIVE-8893.3.patch

          ERROR: -1 due to 1 failed/errored test(s), 6649 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1827/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1827/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1827/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12682018 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12682018/HIVE-8893.3.patch ERROR: -1 due to 1 failed/errored test(s), 6649 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1827/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1827/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1827/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12682018 - PreCommit-HIVE-TRUNK-Build
          Hide
          szehon Szehon Ho added a comment -

          I think this is very useful, left some minor comments on review board.

          Show
          szehon Szehon Ho added a comment - I think this is very useful, left some minor comments on review board.
          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12682121/HIVE-8893.5.patch

          ERROR: -1 due to 1 failed/errored test(s), 6650 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1833/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1833/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1833/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 1 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12682121 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12682121/HIVE-8893.5.patch ERROR: -1 due to 1 failed/errored test(s), 6650 tests executed Failed tests: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1833/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1833/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1833/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 1 tests failed This message is automatically generated. ATTACHMENT ID: 12682121 - PreCommit-HIVE-TRUNK-Build
          Hide
          prasadm Prasad Mujumdar added a comment -

          The failed test optimize_nullscan passes in my setup.

          Show
          prasadm Prasad Mujumdar added a comment - The failed test optimize_nullscan passes in my setup.
          Hide
          szehon Szehon Ho added a comment -

          Thanks for the changes! Latest patch looks good, +1

          Show
          szehon Szehon Ho added a comment - Thanks for the changes! Latest patch looks good, +1
          Hide
          szehon Szehon Ho added a comment -

          Hi Prasad, sorry about that, was looking at the patch again and whitespace is still there on the latest patch, I didnt notice it.

          Also I took a look and its inconsistent in setupBlockedUdfs() to check for empty for black and white list. While we are changing, can we also use the guava splitter with omitEmptyString() argument for this situation, so the logic is cleaner? Again sorry I didnt look that closely before.

          Show
          szehon Szehon Ho added a comment - Hi Prasad, sorry about that, was looking at the patch again and whitespace is still there on the latest patch, I didnt notice it. Also I took a look and its inconsistent in setupBlockedUdfs() to check for empty for black and white list. While we are changing, can we also use the guava splitter with omitEmptyString() argument for this situation, so the logic is cleaner? Again sorry I didnt look that closely before.
          Hide
          prasadm Prasad Mujumdar added a comment -

          Updated patch that addressed review feedback.

          Show
          prasadm Prasad Mujumdar added a comment - Updated patch that addressed review feedback.
          Hide
          szehon Szehon Ho added a comment -

          Thanks! This looks great, will commit it once tests pass.

          Show
          szehon Szehon Ho added a comment - Thanks! This looks great, will commit it once tests pass.
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12682529/HIVE-8893.6.patch

          SUCCESS: +1 6664 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1847/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1847/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1847/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12682529 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12682529/HIVE-8893.6.patch SUCCESS: +1 6664 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1847/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1847/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1847/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12682529 - PreCommit-HIVE-TRUNK-Build
          Hide
          szehon Szehon Ho added a comment -

          Committed to trunk. Thanks Prasad!

          Show
          szehon Szehon Ho added a comment - Committed to trunk. Thanks Prasad!
          Hide
          szehon Szehon Ho added a comment -

          Adds two properties:

          • hive.server2.builtin.udf.whitelist
          • hive.server2.builtin.udf.blacklist

          Have to doc in this section:

          https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2

          Show
          szehon Szehon Ho added a comment - Adds two properties: hive.server2.builtin.udf.whitelist hive.server2.builtin.udf.blacklist Have to doc in this section: https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-HiveServer2
          Hide
          prasadm Prasad Mujumdar added a comment -

          Documented the new properties on the wiki. Thanks Szehon Ho!

          Show
          prasadm Prasad Mujumdar added a comment - Documented the new properties on the wiki. Thanks Szehon Ho !
          Hide
          leftylev Lefty Leverenz added a comment -
          Show
          leftylev Lefty Leverenz added a comment - Looks good, here are the specific links: hive.server2.builtin.udf.whitelist hive.server2.builtin.udf.blacklist
          Hide
          leftylev Lefty Leverenz added a comment -

          Doc questions: Although I removed the TODOC15 label, I wonder whether this needs more documentation. For example, should hive.server2.builtin.udf.whitelist and hive.server2.builtin.udf.blacklist be added to hive.conf.restricted.list to prevent user changes? And should they be mentioned in the HiveServer2 docs or perhaps the UDF doc?

          Show
          leftylev Lefty Leverenz added a comment - Doc questions: Although I removed the TODOC15 label, I wonder whether this needs more documentation. For example, should hive.server2.builtin.udf.whitelist and hive.server2.builtin.udf.blacklist be added to hive.conf.restricted.list to prevent user changes? And should they be mentioned in the HiveServer2 docs or perhaps the UDF doc?
          Hide
          prasadm Prasad Mujumdar added a comment -

          Lefty Leverenz Thanks for bring that up. We do NOT need to add these new udf.* properties to the restrict list. Thes are only read at the bootup time from the initial configuration. Changing those in a session won't make a difference.
          I will add a note in the doc to mention that. Thanks!

          Show
          prasadm Prasad Mujumdar added a comment - Lefty Leverenz Thanks for bring that up. We do NOT need to add these new udf.* properties to the restrict list. Thes are only read at the bootup time from the initial configuration. Changing those in a session won't make a difference. I will add a note in the doc to mention that. Thanks!
          Hide
          prasadm Prasad Mujumdar added a comment -

          Updated the doc to mention the behavior.

          Show
          prasadm Prasad Mujumdar added a comment - Updated the doc to mention the behavior.
          Hide
          leftylev Lefty Leverenz added a comment -

          Doc issue: Prasad Mujumdar, I noticed that you put hive.server2.builtin.udf.whitelist and hive.server2.builtin.udf.blacklist in the Configuration Properties doc after hive.security.authorization.sqlstd.confwhitelist, which is in the "SQL Standard Based Authorization" section. Don't they belong in the "HiveServer2" section instead? Or do they only apply to SQL standard-based authorization?

          Wherever they go, I'll add links in the "Restricted List and Whitelist" subsection of "Authentication/Authorization" just like the link for hive.security.authorization.sqlstd.confwhitelist. If you have better ideas about how to organize all these parameters, please let me know.

          Quick reference:

          Show
          leftylev Lefty Leverenz added a comment - Doc issue: Prasad Mujumdar , I noticed that you put hive.server2.builtin.udf.whitelist and hive.server2.builtin.udf.blacklist in the Configuration Properties doc after hive.security.authorization.sqlstd.confwhitelist , which is in the "SQL Standard Based Authorization" section. Don't they belong in the "HiveServer2" section instead? Or do they only apply to SQL standard-based authorization? Wherever they go, I'll add links in the "Restricted List and Whitelist" subsection of "Authentication/Authorization" just like the link for hive.security.authorization.sqlstd.confwhitelist . If you have better ideas about how to organize all these parameters, please let me know. Quick reference: hive.security.authorization.sqlstd.confwhitelist followed by hive.server2.builtin.udf.whitelist and hive.server2.builtin.udf.blacklist HiveServer2 Restricted List and Whitelist

            People

            • Assignee:
              prasadm Prasad Mujumdar
              Reporter:
              prasadm Prasad Mujumdar
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development