Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-4143

REFRESH TABLE METADATA - Permission Issues with metadata files

    Details

      Description

      Summary of Refresh Metadata Issues confirmed by two different users on Drill User Mailing list. (Title: REFRESH TABLE METADATA - Access Denied)

      This issue pertains to table METADATA and revolves around user authentication.

      Basically, when the drill bits are running as one user, and the data is owned by another user, there can be access denied issues on subsequent queries after issuing a REFRESH TABLE METADATA command.

      To troubleshoot what is actually happening, I turned on MapR Auditing (This is a handy feature) and found that when I run a query (that is giving me access denied.. my query is select count(1) from testtable ) Per MapR the user I am logged in as (dataowner) is trying to do a create operation on the .drill.parquet_metadata file and it's failing with status: 17. Per Keys at MapR, "status 17 means errno 17 which means EEXIST. Looks like Drill is trying to create a file that already exists." This seems to indicate that drill is perhaps trying to create the .drill.parquet_metadata on each select as the dataowner user, but the permissions (as seen below) don't allow it.

      Here are the steps to reproduce:

      Enable Authentication.

      Run all drill bits in the cluster as "drillbituser", then have the files owned by "dataowner". Note the root of the table permissions are drwxrwxr-x but as Drill loads each partition it loads them as drwxr-xr-x (all with dataowner:dataowner ownership). That may be something too, the default permissions when creating a table? Another note, in my setup, drillbituser is in the group for dataowner. Thus, they should always have read access.

      1. Authenticated as dataowner (this should have full permissions to all the data)

      Enter username for jdbc:drill:zk=zknode1:5181: dataowner
      Enter password for jdbc:drill:zk=zknode1:5181: **********
      0: jdbc:drill:zk=zknode1> use dfs.dev;
      -------------------------------------------+

      ok summary

      -------------------------------------------+

      true Default schema changed to [dfs.dev]

      -------------------------------------------+
      1 row selected (0.307 seconds)

      1. The query works fine with no table metadata

      0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
      -----------

      EXPR$0

      -----------

      24565203

      -----------
      1 row selected (3.392 seconds)

      1. Refresh of metadata works under with no errors
        0: jdbc:drill:zk=zknode1> refresh table metadata `testtable`;
        ------------------------------------------------------------+
        ok summary

        ------------------------------------------------------------+

        true Successfully updated metadata for table testtable.

        ------------------------------------------------------------+
        1 row selected (5.767 seconds)

      1. Trying to run the same query, it returns a access denied issue.
        0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
        Error: SYSTEM ERROR: IOException: 2127.7646.2950962 /data/dev/testtable/2015-11-12/.drill.parquet_metadata (Permission denied)

      [Error Id: 7bfce2e7-f78d-4fba-b047-f4c85b471de4 on node1:31010] (state=,code=0)

      1. Note how all the files are owned by the drillbituser. Per discussion on list, this is normal

      $ find ./ -type f -name ".drill.parquet_metadata" -exec ls -ls {} \;
      726 -rwxr-xr-x 1 drillbituser drillbituser 742837 Nov 30 14:27 ./2015-11-12/.drill.parquet_metadata
      583 -rwxr-xr-x 1 drillbituser drillbituser 596146 Nov 30 14:27 ./2015-11-29/.drill.parquet_metadata
      756 -rwxr-xr-x 1 drillbituser drillbituser 773811 Nov 30 14:27 ./2015-11-11/.drill.parquet_metadata
      763 -rwxr-xr-x 1 drillbituser drillbituser 780829 Nov 30 14:27 ./2015-11-04/.drill.parquet_metadata
      632 -rwxr-xr-x 1 drillbituser drillbituser 646851 Nov 30 14:27 ./2015-11-08/.drill.parquet_metadata
      845 -rwxr-xr-x 1 drillbituser drillbituser 864421 Nov 30 14:27 ./2015-11-05/.drill.parquet_metadata
      771 -rwxr-xr-x 1 drillbituser drillbituser 788823 Nov 30 14:27 ./2015-11-28/.drill.parquet_metadata
      1273 -rwxr-xr-x 1 drillbituser drillbituser 1303168 Nov 30 14:27 ./2015-11-10/.drill.parquet_metadata
      645 -rwxr-xr-x 1 drillbituser drillbituser 660028 Nov 30 14:27 ./2015-11-22/.drill.parquet_metadata
      1017 -rwxr-xr-x 1 drillbituser drillbituser 1040469 Nov 30 14:27 ./2015-11-13/.drill.parquet_metadata
      1280 -rwxr-xr-x 1 drillbituser drillbituser 1310552 Nov 30 14:27 ./2015-11-03/.drill.parquet_metadata
      585 -rwxr-xr-x 1 drillbituser drillbituser 598973 Nov 30 14:27 ./2015-11-07/.drill.parquet_metadata
      737 -rwxr-xr-x 1 drillbituser drillbituser 754103 Nov 30 14:27 ./2015-11-20/.drill.parquet_metadata
      836 -rwxr-xr-x 1 drillbituser drillbituser 855363 Nov 30 14:27 ./2015-11-16/.drill.parquet_metadata
      646 -rwxr-xr-x 1 drillbituser drillbituser 661134 Nov 30 14:27 ./2015-11-27/.drill.parquet_metadata
      1156 -rwxr-xr-x 1 drillbituser drillbituser 1183378 Nov 30 14:27 ./2015-11-21/.drill.parquet_metadata
      679 -rwxr-xr-x 1 drillbituser drillbituser 694551 Nov 30 14:27 ./2015-11-09/.drill.parquet_metadata
      631 -rwxr-xr-x 1 drillbituser drillbituser 645500 Nov 30 14:27 ./2015-11-26/.drill.parquet_metadata
      190 -rwxr-xr-x 1 drillbituser drillbituser 193798 Nov 30 14:27 ./2015-11-18/.drill.parquet_metadata
      221 -rwxr-xr-x 1 drillbituser drillbituser 225342 Nov 30 14:27 ./2015-11-23/.drill.parquet_metadata
      269 -rwxr-xr-x 1 drillbituser drillbituser 274988 Nov 30 14:27 ./2015-11-25/.drill.parquet_metadata
      845 -rwxr-xr-x 1 drillbituser drillbituser 864615 Nov 30 14:27 ./2015-11-06/.drill.parquet_metadata
      32 -rwxr-xr-x 1 drillbituser drillbituser 32221 Nov 30 14:27 ./2015-11-19/.drill.parquet_metadata
      16468 -rwxr-xr-x 1 drillbituser drillbituser 16862958 Nov 30 14:27 ./.drill.parquet_metadata

      1. Once the files are deleted...

      $ sudo find ./ -type f -name ".drill.parquet_metadata" -exec rm {} \;

      1. Then the query works again
        0: jdbc:drill:zk=zknode1> select count(1) from `testtable`;
        -----------
        EXPR$0

        -----------

        24567700

        -----------
        1 row selected (1.353 seconds)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                cshi Chunhui Shi
                Reporter:
                mandoskippy John Omernik
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: