Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-9677

FE Analysis tests using fake S3 bucket fail with AnalysisException

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • Impala 4.0.0
    • Impala 4.0.0
    • Frontend

    Description

      It looks like Hadoop added extra validation for initializing the S3AFilesystem. HADOOP-16711 now defaults to checking for the existence of the bucket and permissions on it. The level of validation is controlled by fs.s3a.bucket.probe. We use a bogus S3 bucket in our analysis tests with the expectation that all we require is a valid URI. Instead, we get this error:

      Error during analysis:
      org.apache.impala.common.AnalysisException: bucket: doesBucketExistV2 on bucket: com.amazonaws.services.s3.model.AmazonS3Exception: The AWS Access Key Id you provided does not exist in our records. (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId; Request ID: 10988D12BA86A604; S3 Extended Request ID: qbfw3pMHGGiPTZyLiAOIAvMlJLpy5HOgkHSN4I1n1pBhX2H3BYSwDXdt+gIi3+xiA+E2w5Itv5o=), S3 Extended Request ID: qbfw3pMHGGiPTZyLiAOIAvMlJLpy5HOgkHSN4I1n1pBhX2H3BYSwDXdt+gIi3+xiA+E2w5Itv5o=:InvalidAccessKeyId
      sql:
      create table tbl (i int) location 's3a://bucket/test-warehouse/new_table'

      Developers frequently don't have S3 credentials set up, so this will always fail.

      Currently, these tests fails:

      org.apache.impala.analysis.AnalyzeDDLTest.TestAlterTableAddDropPartition
      org.apache.impala.analysis.AnalyzeDDLTest.TestCreateDataSource
      org.apache.impala.analysis.AnalyzeDDLTest.TestCreateDb
      org.apache.impala.analysis.AnalyzeDDLTest.TestAlterTableSet
      org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTableLike
      org.apache.impala.analysis.AnalyzeDDLTest.TestCreateTable
      org.apache.impala.planner.ExplainTest.testScanNodeFsScheme
      org.apache.impala.planner.PlannerTest.testScanNodeFsScheme

      It reproduces consistently on jenkins.impala.io: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch-cdp-hive/69/

       

      Attachments

        Activity

          People

            joemcdonnell Joe McDonnell
            joemcdonnell Joe McDonnell
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: