[HDDS-5193] Permission Deny when using auth:TOKEN

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Resolved
• Affects Version/s: 1.1.0
• Fix Version/s: 1.4.0
• Component/s: OM
• Labels: pull-request-available

Environment

A Ozone (version 1.1 build from source) cluster with 3 master 3 datanode deploy on baremetal(VMs) running CentOS 7 

 

• Flags: Important
• Assignee: Pakapoj Tulsuk
• Reporter: Pakapoj Tulsuk
• Created: 07/May/21 04:48
• Updated: 27/Mar/23 06:27
• Resolved: 27/Mar/23 06:27

Description

Hi I’m got stuck on the permission issue where I gonna write the data, a text file to a ozone path /vol1/bucket1/mykey * with auth:KERBEROS It be able to complete the task

2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for [e-mail] (auth:KERBEROS) from ip.ip.ip.ip:40294 *  with auth:TOKEN It got stuck on this error despite the given permission to /vol1  /bucket1 see below
2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for [e-mail] (auth:TOKEN) from ip.ip.ip.ip:40412
 2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for [e-mail] (auth:TOKEN) from ip.ip.ip.ip:35266
 2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User [e-mail] doesn't have WRITE permission to access key /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
 2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
 PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User [e-mail] doesn't have WRITE permission to access key vol1 bucket1 mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
 at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
 at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
 at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
 at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
 at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
 at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
 at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
 at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
 at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
 at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)

Given Permission

$ ozone sh vol getacl /vol1/
[ {
  "type" : "USER",
  "name" : "pakapoj_tul",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
}, {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
}, {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-admin",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-users",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
} ]
$ ozone sh bucket getacl /vol1/bucket1/
[ {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-admin",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-users",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
}, {
  "type" : "USER",
  "name" : "pakapoj_tul",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
} ]
$ ozone sh key getacl /vol1/bucket1/mykey/
[ {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-admin",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "GROUP",
  "name" : "ozone-users",
  "aclScope" : "ACCESS",
  "aclList" : [ "ALL" ]
}, {
  "type" : "USER",
  "name" : "[e-mail]",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
}, {
  "type" : "USER",
  "name" : "pakapoj_tul",
  "aclScope" : "ACCESS",
  "aclList" : [ "WRITE", "ALL" ]
} ]

 
The spark code was deployed in Kubernetes in spark cluster mode. Then, the error would happed on spark executors side when the do commitKey with auth:TOKEN , BTW the spark driver was using auth:KERBEROS .
 
so I reproduce using ozone java client writing to ozone with OzoneClient using # Token by export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token before running the program

1. Keytab by running /usr/bin/kinit -kt credential/pakapoj_tul.keytab [e-mail] before running the program

the code, output for #1 and #2 (DEBUG) is in attach
 

Issue Links

• is fixed by: HDDS-5043 Consistently use user's short name in secure mode
• relates to: HDDS-5043 Consistently use user's short name in secure mode
• links to: GitHub Pull Request #2343