Uploaded image for project: 'Apache Ozone'
  1. Apache Ozone
  2. HDDS-5193

Permission Deny when using auth:TOKEN

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • 1.1.0
    • None
    • OM
    • A Ozone (version 1.1 build from source) cluster with 3 master 3 datanode deploy on baremetal(VMs) running CentOS 7 

       

    • Important

    Description

      Hi I’m got stuck on the permission issue where I gonna write the data, a text file to a ozone path /vol1/bucket1/mykey * with auth:KERBEROS It be able to complete the task

      2021-04-29 11:49:01,145 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:KERBEROS) from ip.ip.ip.ip:40294 *  with auth:TOKEN It got stuck on this error despite the given permission to /vol1  /bucket1 see below
      2021-04-29 11:49:08,327 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:40412
       2021-04-29 11:49:12,228 Socket Reader #1 for port 9862 INFO SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for pakapoj_tul@DEV.TAP (auth:TOKEN) from ip.ip.ip.ip:35266
       2021-04-29 11:49:14,671 [OM StateMachine ApplyTransaction Thread - 0] WARN org.apache.hadoop.ozone.om.OzoneManager: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key /vol1/bucket1/mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
       2021-04-29 11:49:14,672 [OM StateMachine ApplyTransaction Thread - 0] ERROR org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest: Key commit failed. Volume:vol1, Bucket:bucket1, Key:mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt.
       PERMISSION_DENIED org.apache.hadoop.ozone.om.exceptions.OMException: User pakapoj_tul@DEV.TAP doesn't have WRITE permission to access key vol1 bucket1 mykey/_temporary/0/_temporary/attempt_202104290449105826106778232640855_0000_m_000000_0/part-00000-9f9c4fcc-5e96-43ee-b53e-913a06729109-c000.txt/106146807974133768
       at org.apache.hadoop.ozone.om.OzoneManager.checkAcls(OzoneManager.java:1803)
       at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:207)
       at org.apache.hadoop.ozone.om.request.OMClientRequest.checkAcls(OMClientRequest.java:185)
       at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAcls(OMKeyRequest.java:437)
       at org.apache.hadoop.ozone.om.request.key.OMKeyRequest.checkKeyAclsInOpenKeyTable(OMKeyRequest.java:485)
       at org.apache.hadoop.ozone.om.request.key.OMKeyCommitRequest.validateAndUpdateCache(OMKeyCommitRequest.java:139)
       at org.apache.hadoop.ozone.protocolPB.OzoneManagerRequestHandler.handleWriteRequest(OzoneManagerRequestHandler.java:227)
       at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.runCommand(OzoneManagerStateMachine.java:415)
       at org.apache.hadoop.ozone.om.ratis.OzoneManagerStateMachine.lambda$applyTransaction$1(OzoneManagerStateMachine.java:240)
       at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
       at java.lang.Thread.run(Thread.java:748)
      

      Given Permission

      $ ozone sh vol getacl /vol1/
      [ {
        "type" : "USER",
        "name" : "pakapoj_tul",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      }, {
        "type" : "USER",
        "name" : "pakapoj_tul@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      }, {
        "type" : "USER",
        "name" : "ozone-admin@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-admin",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-users",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      } ]
      $ ozone sh bucket getacl /vol1/bucket1/
      [ {
        "type" : "USER",
        "name" : "ozone-admin@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-admin",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-users",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "USER",
        "name" : "pakapoj_tul@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      }, {
        "type" : "USER",
        "name" : "pakapoj_tul",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      } ]
      $ ozone sh key getacl /vol1/bucket1/mykey/
      [ {
        "type" : "USER",
        "name" : "ozone-admin@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-admin",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "GROUP",
        "name" : "ozone-users",
        "aclScope" : "ACCESS",
        "aclList" : [ "ALL" ]
      }, {
        "type" : "USER",
        "name" : "pakapoj_tul@DEV.TAP",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      }, {
        "type" : "USER",
        "name" : "pakapoj_tul",
        "aclScope" : "ACCESS",
        "aclList" : [ "WRITE", "ALL" ]
      } ]

       
      The spark code was deployed in Kubernetes in spark cluster mode. Then, the error would happed on spark executors side when the do commitKey with auth:TOKEN , BTW the spark driver was using auth:KERBEROS .
       
      so I reproduce using ozone java client writing to ozone with OzoneClient using # Token by export HADOOP_TOKEN_FILE_LOCATION=credential/ozone.token before running the program

      1. Keytab by running /usr/bin/kinit -kt credential/pakapoj_tul.keytab pakapoj_tul@DEV.TAP before running the program

      the code, output for #1 and #2 (DEBUG) is in attach
       

      Attachments

        1. core-site.xml
          3 kB
          Pakapoj Tulsuk
        2. ozone-client-kerberos.log
          95 kB
          Pakapoj Tulsuk
        3. ozone-client-token.log
          92 kB
          Pakapoj Tulsuk
        4. ozone-java-client.java
          2 kB
          Pakapoj Tulsuk

        Issue Links

          Activity

            People

              pakapoj Pakapoj Tulsuk
              pakapoj Pakapoj Tulsuk
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: