Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-8791

Hive permission inheritance throws exception S3

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.1.0
    • Component/s: Security

      Description

      If Hive permission inheritance is turned on, ACL's are inherited from parent directories. However, on s3 file system, ACL is not supported.

      Hence Hive throws the following exception if hadoop is configured with "dfs.namenode.acls.enabled":

      FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:java.lang.UnsupportedOperationException: NativeS3FileSystem doesn't support getAclStatus)
      

      Workaround is to set this flag to false.

      1. HIVE-8791.2.patch
        3 kB
        Szehon Ho
      2. HIVE-8791.3.patch
        4 kB
        Szehon Ho
      3. HIVE-8791.4.patch
        4 kB
        Szehon Ho
      4. HIVE-8791.4.patch
        4 kB
        Brock Noland
      5. HIVE-8791.patch
        4 kB
        Szehon Ho

        Issue Links

          Activity

          Hide
          hiveqa Hive QA added a comment -

          Overall: -1 at least one tests failed

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12680345/HIVE-8791.patch

          ERROR: -1 due to 3 failed/errored test(s), 6672 tests executed
          Failed tests:

          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_nonacid_from_acid
          org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric
          

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1699/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1699/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1699/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          Tests exited with: TestsFailedException: 3 tests failed
          

          This message is automatically generated.

          ATTACHMENT ID: 12680345 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : -1 at least one tests failed Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12680345/HIVE-8791.patch ERROR: -1 due to 3 failed/errored test(s), 6672 tests executed Failed tests: org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_nonacid_from_acid org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_histogram_numeric Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1699/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1699/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1699/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 3 tests failed This message is automatically generated. ATTACHMENT ID: 12680345 - PreCommit-HIVE-TRUNK-Build
          Hide
          szehon Szehon Ho added a comment -

          Brock Noland can you take a quick look at this? Thanks

          Show
          szehon Szehon Ho added a comment - Brock Noland can you take a quick look at this? Thanks
          Hide
          brocknoland Brock Noland added a comment -

          I think we should log those errors at INFO level. Thoughts?

          Show
          brocknoland Brock Noland added a comment - I think we should log those errors at INFO level. Thoughts?
          Hide
          szehon Szehon Ho added a comment -

          Yep, updated patch

          Show
          szehon Szehon Ho added a comment - Yep, updated patch
          Hide
          brocknoland Brock Noland added a comment -

          Hey,

          I really apologize, I should have mentioned this last time...I was just thinking about this and I am not sure our log message will be as helpful to users as we could be. For example in some setups, users won't know which table is on s3. Perhaps we should add the path to the message as follows:

          LOG.info("Skipping ACL inheritance: File system for path " + target + " does not support ACLs but dfs.namenode.acls.enabled is set to true: " + e, e)
          
          Show
          brocknoland Brock Noland added a comment - Hey, I really apologize, I should have mentioned this last time...I was just thinking about this and I am not sure our log message will be as helpful to users as we could be. For example in some setups, users won't know which table is on s3. Perhaps we should add the path to the message as follows: LOG.info("Skipping ACL inheritance: File system for path " + target + " does not support ACLs but dfs.namenode.acls.enabled is set to true: " + e, e)
          Hide
          szehon Szehon Ho added a comment -

          Sure, attaching patch with better message.

          Show
          szehon Szehon Ho added a comment - Sure, attaching patch with better message.
          Hide
          brocknoland Brock Noland added a comment -

          Thank you Szehon!

          Attached patch makes a trivial change of

          ... true:", e);
          

          to

          ... true: " + e, e);
          

          +1

          Show
          brocknoland Brock Noland added a comment - Thank you Szehon! Attached patch makes a trivial change of ... true:", e); to ... true: " + e, e); +1
          Hide
          szehon Szehon Ho added a comment -

          Attaching same patch again to trigger the pre-commit test as build was down.

          Show
          szehon Szehon Ho added a comment - Attaching same patch again to trigger the pre-commit test as build was down.
          Hide
          hiveqa Hive QA added a comment -

          Overall: +1 all checks pass

          Here are the results of testing the latest attachment:
          https://issues.apache.org/jira/secure/attachment/12681107/HIVE-8791.4.patch

          SUCCESS: +1 6686 tests passed

          Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1757/testReport
          Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1757/console
          Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1757/

          Messages:

          Executing org.apache.hive.ptest.execution.PrepPhase
          Executing org.apache.hive.ptest.execution.ExecutionPhase
          Executing org.apache.hive.ptest.execution.ReportingPhase
          

          This message is automatically generated.

          ATTACHMENT ID: 12681107 - PreCommit-HIVE-TRUNK-Build

          Show
          hiveqa Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12681107/HIVE-8791.4.patch SUCCESS: +1 6686 tests passed Test results: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1757/testReport Console output: http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1757/console Test logs: http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1757/ Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12681107 - PreCommit-HIVE-TRUNK-Build
          Hide
          brocknoland Brock Noland added a comment -

          Thank you Szehon! I have committed this to trunk.

          Show
          brocknoland Brock Noland added a comment - Thank you Szehon! I have committed this to trunk.

            People

            • Assignee:
              szehon Szehon Ho
              Reporter:
              szehon Szehon Ho
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development