Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-5413

test_seq_writer_hive_compatibility fails on a real cluster because test user lacks write access

    Details

    • Epic Color:
      ghx-label-8

      Description

      This test uses the method run_stmt_in_hive() from ImpalaTestSuite to perform a select count ( * ) from foo query. As written, this method doesn't permit a username parameter to be passed. If you're running this test on a real cluster, there's no guarantee that the user who runs the test (which might be 'jenkins', for example, in the case of an automated test run) has the necessary access on the test cluster to submit a job via Hive.

      The fix should be to add a parameter for username to run_stmt_in_hive().

       TestTableWriters.test_seq_writer_hive_compatibility[exec_option: {'disable_codegen': False, 'abort_on_error': 1, 'exec_single_node_rows_threshold': 0, 'batch_size': 0, 'num_nodes': 0} | table_format: text/none] 
      [gw0] linux2 -- Python 2.7.6 /data/jenkins/workspace/Impala-Quasar-Executor-New/testing/infra/quasar/deploy/executions/runtime/testing/infra/quasar/execution/common_tests/impala/Impala/bin/../infra/python/env/bin/python
      query_test/test_compressed_formats.py:177: in test_seq_writer_hive_compatibility
          output = self.run_stmt_in_hive('select count(*) from %s' % table_name)
      common/impala_test_suite.py:621: in run_stmt_in_hive
          raise RuntimeError(stderr)
      E   RuntimeError: SLF4J: Class path contains multiple SLF4J bindings.
      E   SLF4J: Found binding in [jar:file:/data/jenkins/workspace/Impala-Quasar-Executor-New/testing/infra/quasar/deploy/executions/runtime/testing/infra/quasar/execution/common_tests/impala/Impala/thirdparty/hbase-1.2.0-cdh5.12.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      E   SLF4J: Found binding in [jar:file:/data/jenkins/workspace/Impala-Quasar-Executor-New/testing/infra/quasar/deploy/executions/runtime/testing/infra/quasar/execution/common_tests/impala/Impala/thirdparty/hadoop-2.6.0-cdh5.12.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
      E   SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
      E   SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
      E   scan complete in 3ms
      E   Connecting to jdbc:hive2://scm-ubuntu1404-mlvdhi-1.vpc.cloudera.com:10000
      E   Connected to: Apache Hive (version 1.1.0-cdh5.12.0)
      E   Driver: Hive JDBC (version 1.1.0-cdh5.12.0)
      E   Transaction isolation: TRANSACTION_REPEATABLE_READ
      E   INFO  : Compiling command(queryId=hive_20170531155353_04ad3f17-cf20-43f7-bb01-78d2feb381e2): select count(*) from test_seq_writer_hive_compatibility_e3728f35.seq_tbl_NONE_RECORD
      E   INFO  : Semantic Analysis Completed
      E   INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:_c0, type:bigint, comment:null)], properties:null)
      E   INFO  : Completed compiling command(queryId=hive_20170531155353_04ad3f17-cf20-43f7-bb01-78d2feb381e2); Time taken: 0.047 seconds
      E   INFO  : Executing command(queryId=hive_20170531155353_04ad3f17-cf20-43f7-bb01-78d2feb381e2): select count(*) from test_seq_writer_hive_compatibility_e3728f35.seq_tbl_NONE_RECORD
      E   INFO  : Query ID = hive_20170531155353_04ad3f17-cf20-43f7-bb01-78d2feb381e2
      E   INFO  : Total jobs = 1
      E   INFO  : Launching Job 1 out of 1
      E   INFO  : Starting task [Stage-1:MAPRED] in serial mode
      E   INFO  : Number of reduce tasks determined at compile time: 1
      E   INFO  : In order to change the average load for a reducer (in bytes):
      E   INFO  :   set hive.exec.reducers.bytes.per.reducer=<number>
      E   INFO  : In order to limit the maximum number of reducers:
      E   INFO  :   set hive.exec.reducers.max=<number>
      E   INFO  : In order to set a constant number of reducers:
      E   INFO  :   set mapreduce.job.reduces=<number>
      E   ERROR : Job Submission failed with exception 'org.apache.hadoop.security.AccessControlException(Permission denied: user=jenkins, access=WRITE, inode="/user":hdfs:supergroup:drwxr-xr-x
      

        Activity

        Hide
        dknupp David Knupp added a comment -
        Show
        dknupp David Knupp added a comment - Posted for review: https://gerrit.cloudera.org/#/c/7046/

          People

          • Assignee:
            dknupp David Knupp
            Reporter:
            dknupp David Knupp
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development