Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-858

Support for hadoop kerberos authentication in Tajo

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0, 0.9.0
    • Fix Version/s: 0.12.0, 0.11.2
    • Component/s: None
    • Labels:
      None

      Description

      The hadoop cluster is configured to use kerberos as authentication mechanism. The exception is list below, seems when opening a hdfs file, tajo can't read the security related config items from core-site.xml. It still used SIMPLE authentication.

      2014-05-29 01:00:40,269 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1551)) - PriviledgedActionException as:mzhou (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,270 WARNdomain  ipc.Client (Client.java:run(669)) - Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client canhostnot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,270 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1551)) - PriviledgedActionException as:mzhou (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,278 ERROR worker.Task (Task.java:run(393)) - java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host4.grid.domain.com/172.20.1.34"; destination host is: "host1.grid.domain.com":9000; 
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1410)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1359)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
      	at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      	at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:206)
      	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1131)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1121)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1111)
      	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:272)
      	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:239)
      	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:232)
      	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1279)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:292)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:292)
      	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:765)
      	at org.apache.tajo.storage.CSVFile$CSVScanner.init(CSVFile.java:303)
      	at org.apache.tajo.engine.planner.physical.SeqScanExec.initScanner(SeqScanExec.java:197)
      	at org.apache.tajo.engine.planner.physical.SeqScanExec.init(SeqScanExec.java:179)
      	at org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:52)
      	at org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:52)
      	at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.init(HashShuffleFileWriteExec.java:81)
      	at org.apache.tajo.worker.Task.run(Task.java:383)
      	at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
      	at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:721)
      	at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1377)
      	... 32 more
      Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:170)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:387)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547)
      	at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708)
      	... 35 more
      
      2014-05-29 01:00:40,278 INFO  worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1401325188560_0001_000001_000000_00 is changed to TA_FAILED
      2014-05-29 01:00:40,281 INFO  worker.Task (Task.java:run(447)) - Task Counter - total:1, succeeded: 0, killed: 0, failed: 1
      2014-05-29 01:00:40,282 INFO  worker.TaskRunner (TaskRunner.java:run(332)) - Request GetTask: eb_1401325188560_0001_000001,container_1401325188560_0001_01_000001
      2014-05-29 01:00:40,305 INFO  worker.TaskRunner (TaskRunner.java:run(370)) - Accumulated Received Task: 2
      2014-05-29 01:00:40,305 INFO  worker.TaskRunner (TaskRunner.java:run(379)) - Initializing: ta_1401325188560_0001_000001_000000_01
      2014-05-29 01:00:40,316 INFO  worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1401325188560_0001_000001_000000_01 is changed to TA_PENDING
      2014-05-29 01:00:40,316 INFO  worker.Task (Task.java:<init>(188)) - ==================================
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(189)) - * Subquery ta_1401325188560_0001_000001_000000_01 is initialized
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(190)) - * InterQuery: true, Use HASH_SHUFFLE shuffle
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(193)) - * Fragments (num: 1)
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(194)) - * Fetches (total:0) :
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(198)) - * Local task dir: file:/grid/d/tmp/yarn/usercache/mzhou/appcache/application_1400096295333_0092/container_1400096295333_0092_01_000004/${LOCAL_DIRS}/q_1401325188560_0001/output/1/0_1
      2014-05-29 01:00:40,318 INFO  worker.Task (Task.java:<init>(203)) - ==================================
      2014-05-29 01:00:40,319 INFO  worker.TaskAttemptContext (TaskAttemptContext.java:setState(110)) - Query status of ta_1401325188560_0001_000001_000000_01 is changed to TA_RUNNING
      2014-05-29 01:00:40,319 INFO  planner.PhysicalPlannerImpl (PhysicalPlannerImpl.java:createInMemoryHashAggregation(901)) - The planner chooses [Hash Aggregation]
      2014-05-29 01:00:40,325 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1551)) - PriviledgedActionException as:mzhou (auth:SIMPLE) cause:org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,326 WARN  ipc.Client (Client.java:run(669)) - Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,326 WARN  security.UserGroupInformation (UserGroupInformation.java:doAs(1551)) - PriviledgedActionException as:mzhou (auth:SIMPLE) cause:java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      2014-05-29 01:00:40,328 ERROR worker.Task (Task.java:run(393)) - java.io.IOException: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host4.grid.domain.com/172.20.1.34"; destination host is: "host1.grid.domain.com":9000; 
      	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:764)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1410)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1359)
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
      	at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:606)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
      	at com.sun.proxy.$Proxy9.getBlockLocations(Unknown Source)
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getBlockLocations(ClientNamenodeProtocolTranslatorPB.java:206)
      	at org.apache.hadoop.hdfs.DFSClient.callGetBlockLocations(DFSClient.java:1131)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1121)
      	at org.apache.hadoop.hdfs.DFSClient.getLocatedBlocks(DFSClient.java:1111)
      	at org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:272)
      	at org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:239)
      	at org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:232)
      	at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1279)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:296)
      	at org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:292)
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
      	at org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:292)
      	at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:765)
      	at org.apache.tajo.storage.CSVFile$CSVScanner.init(CSVFile.java:303)
      	at org.apache.tajo.engine.planner.physical.SeqScanExec.initScanner(SeqScanExec.java:197)
      	at org.apache.tajo.engine.planner.physical.SeqScanExec.init(SeqScanExec.java:179)
      	at org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:52)
      	at org.apache.tajo.engine.planner.physical.UnaryPhysicalExec.init(UnaryPhysicalExec.java:52)
      	at org.apache.tajo.engine.planner.physical.HashShuffleFileWriteExec.init(HashShuffleFileWriteExec.java:81)
      	at org.apache.tajo.worker.Task.run(Task.java:383)
      	at org.apache.tajo.worker.TaskRunner$1.run(TaskRunner.java:391)
      	at java.lang.Thread.run(Thread.java:744)
      Caused by: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      	at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:674)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:637)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:721)
      	at org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367)
      	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1458)
      	at org.apache.hadoop.ipc.Client.call(Client.java:1377)
      	... 32 more
      Caused by: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]
      	at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:170)
      	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:387)
      	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547)
      	at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
      	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709)
      	at java.security.AccessController.doPrivileged(Native Method)
      	at javax.security.auth.Subject.doAs(Subject.java:415)
      	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
      	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708)
      	... 35 more
      
      

        Issue Links

          Activity

          Hide
          hyunsik Hyunsik Choi added a comment -

          Thank you for sharing the problem. Security seems a different area. In order to encourage Tajo to be used in production clusters, we must solve this problem.

          Are there are any volunteer who knows Hadoop security? Anyway, I'll also start learning Hadoop security after a while.

          Show
          hyunsik Hyunsik Choi added a comment - Thank you for sharing the problem. Security seems a different area. In order to encourage Tajo to be used in production clusters, we must solve this problem. Are there are any volunteer who knows Hadoop security? Anyway, I'll also start learning Hadoop security after a while.
          Hide
          coderplay Min Zhou added a comment -

          Related to this topic, when I try select * from tbl in the client side. I got an exception

          default> select * from people;
          Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host1.grid.domain.com/172.20.1.37"; destination host is: "host1.grid.domain.com":9000; 
          

          I used btrace to track the stack, here is how the exception was thrown.

          org.apache.hadoop.security.AccessControlException.<init>(AccessControlException.java:47)
          org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:170)
          org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:387)
          org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547)
          org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367)
          org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713)
          org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709)
          java.security.AccessController.doPrivileged(Native Method)
          javax.security.auth.Subject.doAs(Subject.java:415)
          org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
          org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708)
          org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367)
          org.apache.hadoop.ipc.Client.getConnection(Client.java:1458)
          org.apache.hadoop.ipc.Client.call(Client.java:1377)
          org.apache.hadoop.ipc.Client.call(Client.java:1359)
          org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
          com.sun.proxy.$Proxy12.getListing(Unknown Source)
          sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
          sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          java.lang.reflect.Method.invoke(Method.java:606)
          org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
          org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
          com.sun.proxy.$Proxy12.getListing(Unknown Source)
          org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:502)
          org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1727)
          org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1710)
          org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:646)
          org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:98)
          org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708)
          org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:704)
          org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
          org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:704)
          org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484)
          org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524)
          org.apache.tajo.jdbc.TajoResultSet.getFragments(TajoResultSet.java:113)
          org.apache.tajo.jdbc.TajoResultSet.initScanner(TajoResultSet.java:91)
          org.apache.tajo.jdbc.TajoResultSet.<init>(TajoResultSet.java:68)
          org.apache.tajo.jdbc.TajoResultSet.<init>(TajoResultSet.java:74)
          org.apache.tajo.client.TajoClient.createResultSet(TajoClient.java:438)
          org.apache.tajo.cli.TajoCli.executeQuery(TajoCli.java:329)
          org.apache.tajo.cli.TajoCli.executeParsedResults(TajoCli.java:280)
          org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:269)
          org.apache.tajo.cli.TajoCli.main(TajoCli.java:495)
          
          Show
          coderplay Min Zhou added a comment - Related to this topic, when I try select * from tbl in the client side. I got an exception default> select * from people; Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]; Host Details : local host is: "host1.grid.domain.com/172.20.1.37"; destination host is: "host1.grid.domain.com":9000; I used btrace to track the stack, here is how the exception was thrown. org.apache.hadoop.security.AccessControlException.<init>(AccessControlException.java:47) org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:170) org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:387) org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:547) org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:367) org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:713) org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:709) java.security.AccessController.doPrivileged(Native Method) javax.security.auth.Subject.doAs(Subject.java:415) org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:708) org.apache.hadoop.ipc.Client$Connection.access$2700(Client.java:367) org.apache.hadoop.ipc.Client.getConnection(Client.java:1458) org.apache.hadoop.ipc.Client.call(Client.java:1377) org.apache.hadoop.ipc.Client.call(Client.java:1359) org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) com.sun.proxy.$Proxy12.getListing(Unknown Source) sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) java.lang.reflect.Method.invoke(Method.java:606) org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186) org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) com.sun.proxy.$Proxy12.getListing(Unknown Source) org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:502) org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1727) org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:1710) org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:646) org.apache.hadoop.hdfs.DistributedFileSystem.access$600(DistributedFileSystem.java:98) org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:708) org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:704) org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:704) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1484) org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1524) org.apache.tajo.jdbc.TajoResultSet.getFragments(TajoResultSet.java:113) org.apache.tajo.jdbc.TajoResultSet.initScanner(TajoResultSet.java:91) org.apache.tajo.jdbc.TajoResultSet.<init>(TajoResultSet.java:68) org.apache.tajo.jdbc.TajoResultSet.<init>(TajoResultSet.java:74) org.apache.tajo.client.TajoClient.createResultSet(TajoClient.java:438) org.apache.tajo.cli.TajoCli.executeQuery(TajoCli.java:329) org.apache.tajo.cli.TajoCli.executeParsedResults(TajoCli.java:280) org.apache.tajo.cli.TajoCli.runShell(TajoCli.java:269) org.apache.tajo.cli.TajoCli.main(TajoCli.java:495)
          Hide
          coderplay Min Zhou added a comment -

          I can explain the client side exception now, tajo pass a username to FileSystem in FileScanner.java

            public static FileSystem getFileSystem(TajoConf tajoConf, Path path) throws IOException {
              String tajoUser = tajoConf.getVar(TajoConf.ConfVars.USERNAME);
              FileSystem fs;
              if(tajoUser != null) {
                try {
                  fs = FileSystem.get(path.toUri(), tajoConf, tajoUser);
                } catch (InterruptedException e) {
                  LOG.warn("Occur InterruptedException while FileSystem initiating with user[" + tajoUser + "]");
                  fs = FileSystem.get(path.toUri(), tajoConf);
                }
              } else {
                fs = FileSystem.get(path.toUri(), tajoConf);
              }
          
              return fs;
            }
          

          If user didn't config tajo username, by default it will get the $

          {user.name} env variable, so that code never run into the "else" fork.

          That method will call UserGroupInformation.java:getBestUGI to fetch the UGI information, since the client side have no ticket on disk, so the ticketCachePath will be null, and user will be ${user.name}

          . The method below will jump to the third branch , createRemoteUser, which only use SIMPLE authentication.

          /**
             * Find the most appropriate UserGroupInformation to use
             *
             * @param ticketCachePath    The Kerberos ticket cache path, or NULL
             *                           if none is specfied
             * @param user               The user name, or NULL if none is specified.
             *
             * @return                   The most appropriate UserGroupInformation
             */ 
            public static UserGroupInformation getBestUGI(
                String ticketCachePath, String user) throws IOException {
              if (ticketCachePath != null) {
                return getUGIFromTicketCache(ticketCachePath, user);
              } else if (user == null) {
                return getCurrentUser();
              } else {
                return createRemoteUser(user);
              }    
            }
          

          I have confirmed this explaination through btrace.

          Show
          coderplay Min Zhou added a comment - I can explain the client side exception now, tajo pass a username to FileSystem in FileScanner.java public static FileSystem getFileSystem(TajoConf tajoConf, Path path) throws IOException { String tajoUser = tajoConf.getVar(TajoConf.ConfVars.USERNAME); FileSystem fs; if(tajoUser != null) { try { fs = FileSystem.get(path.toUri(), tajoConf, tajoUser); } catch (InterruptedException e) { LOG.warn("Occur InterruptedException while FileSystem initiating with user[" + tajoUser + "]"); fs = FileSystem.get(path.toUri(), tajoConf); } } else { fs = FileSystem.get(path.toUri(), tajoConf); } return fs; } If user didn't config tajo username, by default it will get the $ {user.name} env variable, so that code never run into the "else" fork. That method will call UserGroupInformation.java:getBestUGI to fetch the UGI information, since the client side have no ticket on disk, so the ticketCachePath will be null, and user will be ${user.name} . The method below will jump to the third branch , createRemoteUser, which only use SIMPLE authentication. /** * Find the most appropriate UserGroupInformation to use * * @param ticketCachePath The Kerberos ticket cache path, or NULL * if none is specfied * @param user The user name, or NULL if none is specified. * * @return The most appropriate UserGroupInformation */ public static UserGroupInformation getBestUGI( String ticketCachePath, String user) throws IOException { if (ticketCachePath != null) { return getUGIFromTicketCache(ticketCachePath, user); } else if (user == null) { return getCurrentUser(); } else { return createRemoteUser(user); } } I have confirmed this explaination through btrace.
          Hide
          prafulla Prafulla T added a comment -

          I can help with this. I have some (little) experience with secure hadoop.
          Here is the idea
          We should add couple of parameters to tajo-site.xml
          kerberos principal for dfs namenode (something like username )
          keytab for above principle (something like password)

          TajoMaster will login to hadoop using above two parameters (using UserGroupInformation.loginUserFromKeytabAndReturnUGI)
          This ugi can be used to get filesystem object in secure way.
          FileSystem.getDelegationToken can be used to allow future logins from different processes without using kerberos.
          We can add new message to TajoMasterProtocol to send this delegation token to the workers. Workers will use this token to create filesystem object in secure way.

          Let me know your comments.

          Show
          prafulla Prafulla T added a comment - I can help with this. I have some (little) experience with secure hadoop. Here is the idea We should add couple of parameters to tajo-site.xml kerberos principal for dfs namenode (something like username ) keytab for above principle (something like password) TajoMaster will login to hadoop using above two parameters (using UserGroupInformation.loginUserFromKeytabAndReturnUGI) This ugi can be used to get filesystem object in secure way. FileSystem.getDelegationToken can be used to allow future logins from different processes without using kerberos. We can add new message to TajoMasterProtocol to send this delegation token to the workers. Workers will use this token to create filesystem object in secure way. Let me know your comments.
          Hide
          jihoonson Jihoon Son added a comment -

          Hi Prafulla,
          it seems great to me.
          Thanks!!

          Show
          jihoonson Jihoon Son added a comment - Hi Prafulla, it seems great to me. Thanks!!
          Hide
          prafulla Prafulla T added a comment -

          I have some initial changes done at my github repository.
          Patch can be viewed at following link. (Expand commit message to read detailed message)

          https://github.com/prafullat/tajo/compare/tajo-858?expand=1

          If anyone has any early comments about overall approach/coding style, let me know.
          I will continue to work on this.

          Show
          prafulla Prafulla T added a comment - I have some initial changes done at my github repository. Patch can be viewed at following link. (Expand commit message to read detailed message) https://github.com/prafullat/tajo/compare/tajo-858?expand=1 If anyone has any early comments about overall approach/coding style, let me know. I will continue to work on this.
          Hide
          coderplay Min Zhou added a comment -

          I already solved it simply by ignoring the tajo.username when fetching a Filesystem instance. It works well.

          Here is the patch

          Show
          coderplay Min Zhou added a comment - I already solved it simply by ignoring the tajo.username when fetching a Filesystem instance. It works well. Here is the patch
          Hide
          coderplay Min Zhou added a comment -

          Prafulla T

          Your solution looks better than mine, please go ahead. I will test it on tajo-yarn mode. Thanks.

          Show
          coderplay Min Zhou added a comment - Prafulla T Your solution looks better than mine, please go ahead. I will test it on tajo-yarn mode. Thanks.
          Hide
          coderplay Min Zhou added a comment - - edited

          Prafulla T

          take a quick scan on your patch, seems has some duplicate work if using tajo-yarn mode.

          See
          https://github.com/coderplay/tajo-yarn/blob/master/src/main/java/org/apache/tajo/yarn/WorkerContainerTask.java line 80-89

          Basically, yarn can download credentials for its application daemons. Applications on yarn never use keytab.

          Could you please create an issue on apache tajo github? Then I can make comment more conveniently

          Show
          coderplay Min Zhou added a comment - - edited Prafulla T take a quick scan on your patch, seems has some duplicate work if using tajo-yarn mode. See https://github.com/coderplay/tajo-yarn/blob/master/src/main/java/org/apache/tajo/yarn/WorkerContainerTask.java line 80-89 Basically, yarn can download credentials for its application daemons. Applications on yarn never use keytab. Could you please create an issue on apache tajo github? Then I can make comment more conveniently
          Hide
          prafulla Prafulla T added a comment -

          Hi Min,

          Thanks for your comments. I am currently testing it in normal (non-yarn) mode.
          I have opened pull request at below location so that it is easier for people to comment on specific part of patch.
          https://github.com/apache/tajo/pull/30

          Show
          prafulla Prafulla T added a comment - Hi Min, Thanks for your comments. I am currently testing it in normal (non-yarn) mode. I have opened pull request at below location so that it is easier for people to comment on specific part of patch. https://github.com/apache/tajo/pull/30
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/30#issuecomment-44976584

          Hi Prafulla,
          it looks good!
          Please go ahead.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/30#issuecomment-44976584 Hi Prafulla, it looks good! Please go ahead.
          Hide
          prafulla Prafulla T added a comment -

          Hi,

          I am done with basic changes required to get Tajo working in kerberized hadoop, but I would need help from senior Tajo contributors.
          My patch can be viewed at following pull request.
          https://github.com/apache/tajo/pull/30

          My questions.
          1. I have developed new api to get FileSystem object. I need to replace all Path.getFileSystem calls with this new function. New getFileSystem implement depends on certain paramaters set in TajoConf. How should we reliably make sure that required parameters are set in TajoConf before .getFileSystem is called?
          2. I see that tajo.storage.* class is not accessible in tajo-common. Certain classes in tajo-common requires new API to get filesystem object. Should I move my new class FileSystemUtil to tajo-common ?
          3. Delegation token is considered as some important and confidential information in secured hadoop. We would need some way to securely exchange it in Tajo cluster. I think, This can be done later as separate issue. ( See http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/ )

          Right now, simple query works with kerberized hadoop, but longer query hits following exception when we try to create staging directory.
          java.io.IOException: org.apache.tajo.conf.TajoConfException: Hadoop DFS delegationToken is null, It should have been set in TajoMaster
          at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:188)
          at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:144)
          at org.apache.tajo.master.querymaster.QueryMasterTask.initStagingDir(QueryMasterTask.java:373)
          at org.apache.tajo.master.querymaster.QueryMasterTask.init(QueryMasterTask.java:160)
          at org.apache.tajo.master.querymaster.QueryMaster$QueryStartEventHandler.handle(QueryMaster.java:378)
          at org.apache.tajo.master.querymaster.QueryMaster$QueryStartEventHandler.handle(QueryMaster.java:371)
          at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137)
          at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79)
          at java.lang.Thread.run(Thread.java:724)
          Caused by: org.apache.tajo.conf.TajoConfException: Hadoop DFS delegationToken is null, It should have been set in TajoMaster
          at org.apache.tajo.storage.FileSystemUtil.getDFSUsingDelegationToken(FileSystemUtil.java:56)
          at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:185)
          ... 8 more

          This happens because required parameter is not set in TajoConf. (Problem (1) above).

          Show
          prafulla Prafulla T added a comment - Hi, I am done with basic changes required to get Tajo working in kerberized hadoop, but I would need help from senior Tajo contributors. My patch can be viewed at following pull request. https://github.com/apache/tajo/pull/30 My questions. 1. I have developed new api to get FileSystem object. I need to replace all Path.getFileSystem calls with this new function. New getFileSystem implement depends on certain paramaters set in TajoConf. How should we reliably make sure that required parameters are set in TajoConf before .getFileSystem is called? 2. I see that tajo.storage.* class is not accessible in tajo-common. Certain classes in tajo-common requires new API to get filesystem object. Should I move my new class FileSystemUtil to tajo-common ? 3. Delegation token is considered as some important and confidential information in secured hadoop. We would need some way to securely exchange it in Tajo cluster. I think, This can be done later as separate issue. ( See http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/ ) Right now, simple query works with kerberized hadoop, but longer query hits following exception when we try to create staging directory. java.io.IOException: org.apache.tajo.conf.TajoConfException: Hadoop DFS delegationToken is null, It should have been set in TajoMaster at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:188) at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:144) at org.apache.tajo.master.querymaster.QueryMasterTask.initStagingDir(QueryMasterTask.java:373) at org.apache.tajo.master.querymaster.QueryMasterTask.init(QueryMasterTask.java:160) at org.apache.tajo.master.querymaster.QueryMaster$QueryStartEventHandler.handle(QueryMaster.java:378) at org.apache.tajo.master.querymaster.QueryMaster$QueryStartEventHandler.handle(QueryMaster.java:371) at org.apache.tajo.master.TajoAsyncDispatcher.dispatch(TajoAsyncDispatcher.java:137) at org.apache.tajo.master.TajoAsyncDispatcher$1.run(TajoAsyncDispatcher.java:79) at java.lang.Thread.run(Thread.java:724) Caused by: org.apache.tajo.conf.TajoConfException: Hadoop DFS delegationToken is null, It should have been set in TajoMaster at org.apache.tajo.storage.FileSystemUtil.getDFSUsingDelegationToken(FileSystemUtil.java:56) at org.apache.tajo.storage.FileSystemUtil.getFileSystem(FileSystemUtil.java:185) ... 8 more This happens because required parameter is not set in TajoConf. (Problem (1) above).
          Hide
          jihoonson Jihoon Son added a comment -

          Hi Prafulla,
          I'm not familiar with kerberized hadoop, so the third question is hard to answer for me.
          Followings are my answers for other questions.

          1. I think that required parameters might have some default values. If they are, using TajoConf.ConfVars can be a good solution. ConfVars is an enumeration which contains parameters required for tajo and their default values. These parameters are always added to a TajoConf instance when it is created.

          2. Before deciding move your class to tajo-common, I wonder which classes in tajo-common require to get filesystem object. Would you mind listing some examples?

          Thanks,
          Jihoon

          Show
          jihoonson Jihoon Son added a comment - Hi Prafulla, I'm not familiar with kerberized hadoop, so the third question is hard to answer for me. Followings are my answers for other questions. 1. I think that required parameters might have some default values. If they are, using TajoConf.ConfVars can be a good solution. ConfVars is an enumeration which contains parameters required for tajo and their default values. These parameters are always added to a TajoConf instance when it is created. 2. Before deciding move your class to tajo-common, I wonder which classes in tajo-common require to get filesystem object. Would you mind listing some examples? Thanks, Jihoon
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Jihoon,

          TajoConf uses Path::getFileSystem method in order to obtain some base filesystem that a running Tajo cluster uses. I have no idea about how we eliminate it from TajoConf yet.

          Hi Prafulla,

          Your work looks nice. I leave inline comments on your questions.

          > 1. I have developed new api to get FileSystem object. I need to replace all Path.getFileSystem calls with this new function. New getFileSystem implement depends on certain paramaters set in TajoConf. How should we reliably make sure that required parameters are set in TajoConf before .getFileSystem is called?

          In my opinion, we need to force those who want to kerberos authentication to set required parameters to tajo-site.xml. We can write some guide manual for it. Otherwise, Tajo should be work in non-secured mode. If we need some default parameters, we can use tajo-default.xml in resources directory. I'm not an expert in secured Hadoop. If my opinion does not make sense, please let me know that.

          > 2. I see that tajo.storage.* class is not accessible in tajo-common. Certain classes in tajo-common requires new API to get filesystem object. Should I move my new class FileSystemUtil to tajo-common?

          In my opinion, tajo-common is proper package to include FileSystemUtil and kerberos-related modules. In almost all cases, FileSystem object of HDFS is essential throughout Tajo system. In addition, we also use FileSystem in order to access table data files as well as general files stored in HDFS.

          > 3. Delegation token is considered as some important and confidential information in secured hadoop. We would need some way to securely exchange it in Tajo cluster. I think, This can be done later as separate issue. ( See http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/ )

          Sounds nice!

          Show
          hyunsik Hyunsik Choi added a comment - Hi Jihoon, TajoConf uses Path::getFileSystem method in order to obtain some base filesystem that a running Tajo cluster uses. I have no idea about how we eliminate it from TajoConf yet. Hi Prafulla, Your work looks nice. I leave inline comments on your questions. > 1. I have developed new api to get FileSystem object. I need to replace all Path.getFileSystem calls with this new function. New getFileSystem implement depends on certain paramaters set in TajoConf. How should we reliably make sure that required parameters are set in TajoConf before .getFileSystem is called? In my opinion, we need to force those who want to kerberos authentication to set required parameters to tajo-site.xml. We can write some guide manual for it. Otherwise, Tajo should be work in non-secured mode. If we need some default parameters, we can use tajo-default.xml in resources directory. I'm not an expert in secured Hadoop. If my opinion does not make sense, please let me know that. > 2. I see that tajo.storage.* class is not accessible in tajo-common. Certain classes in tajo-common requires new API to get filesystem object. Should I move my new class FileSystemUtil to tajo-common? In my opinion, tajo-common is proper package to include FileSystemUtil and kerberos-related modules. In almost all cases, FileSystem object of HDFS is essential throughout Tajo system. In addition, we also use FileSystem in order to access table data files as well as general files stored in HDFS. > 3. Delegation token is considered as some important and confidential information in secured hadoop. We would need some way to securely exchange it in Tajo cluster. I think, This can be done later as separate issue. ( See http://hortonworks.com/blog/the-role-of-delegation-tokens-in-apache-hadoop-security/ ) Sounds nice!
          Hide
          prafulla Prafulla T added a comment -

          Hi Jihoon, Hyunsik
          Thanks for your comments.

          I am talking about the parameters which are set at at runtime (Delegation token). Currently, TajoMaster requests this parameter at runtime by contacting hadoop and sets it in TajoConf. TajoWorker or TajoClient can get this token by requesting it from TajoMaster. The problem that I am facing is how TajoWorker or Client can reliably set this token, so that it is available at everyplace where FileSystem object needs to be accessed.

          Show
          prafulla Prafulla T added a comment - Hi Jihoon, Hyunsik Thanks for your comments. I am talking about the parameters which are set at at runtime (Delegation token). Currently, TajoMaster requests this parameter at runtime by contacting hadoop and sets it in TajoConf. TajoWorker or TajoClient can get this token by requesting it from TajoMaster. The problem that I am facing is how TajoWorker or Client can reliably set this token, so that it is available at everyplace where FileSystem object needs to be accessed.
          Hide
          hyunsik Hyunsik Choi added a comment -

          Hi Prafulla,

          I got your point. How about using the session variable? TajoClient can set the session variable, and the session variables will be copied to QueryContext (122 line in GlobalEngine) which will be passed throughout all modules including TajoMaster, QueryMaster, and all Workers which execute tasks. QueryContext is a set of temporal configs for each query. Session variable will be removed from all systems after the client is disconnected.

          Internally, a FileSystem object for a certain FS URI seems to be cached with UserGroupInformation. So, one JVM seems to make and cache multiple FileSystem objects with the same FS URI and different kerberos tokens. So, I expect that your on-going approach is nice and work well.

          Show
          hyunsik Hyunsik Choi added a comment - Hi Prafulla, I got your point. How about using the session variable? TajoClient can set the session variable, and the session variables will be copied to QueryContext (122 line in GlobalEngine) which will be passed throughout all modules including TajoMaster, QueryMaster, and all Workers which execute tasks. QueryContext is a set of temporal configs for each query. Session variable will be removed from all systems after the client is disconnected. Internally, a FileSystem object for a certain FS URI seems to be cached with UserGroupInformation. So, one JVM seems to make and cache multiple FileSystem objects with the same FS URI and different kerberos tokens. So, I expect that your on-going approach is nice and work well.
          Hide
          coderplay Min Zhou added a comment - - edited

          Hi Prafulla,

          Just put my 2 cents here.

          1. Do you intend to achieve that different tajo users will operate hdfs under different hdfs users? If so, I think you need to fetch tokens from namenode on the tajo client side at first. Explicitly retrieve the tokens from tajo conf seems not very practicable due to the tokens will expired. IMHO, you can do it transparent to tajo. Tajo shouldn't bind to HDFS, it should just aware of FileSystem interface rather than its concrete implementations, where tajo would support other types of filesystem, like S3. Actually, FileSystem.get() or UserGroupInformation can retrieve tokens for you according to the items from hadoop's core-site.xml.

          2. As I mentioned, for yarn mode. Yarn's NodeManager will download the tokens for each container, by default the token files will be in the workdir of that container. So the token file path is determined by runtime. User can't statically config the tokens for each container.

          3. For security reasons, explicitly config the token files path in the tajo-site.xml is not safe for users.

          4. Seems we can't use keytab here. If we have 1000 users, we need to enroll every user from the KDC. That's not very scalable I think. If the system has new come user, we need sync his/her keytabs to every nodes of the cluster. Hard to maintain. On the other hand, if you don't intend to distinguish users in tajo, then just use the default keytab for tajo workers seems more reasonable.

          Thanks for your contribution, Prafulla. Let's keep discussing.

          Min

          Show
          coderplay Min Zhou added a comment - - edited Hi Prafulla, Just put my 2 cents here. 1. Do you intend to achieve that different tajo users will operate hdfs under different hdfs users? If so, I think you need to fetch tokens from namenode on the tajo client side at first. Explicitly retrieve the tokens from tajo conf seems not very practicable due to the tokens will expired. IMHO, you can do it transparent to tajo. Tajo shouldn't bind to HDFS, it should just aware of FileSystem interface rather than its concrete implementations, where tajo would support other types of filesystem, like S3. Actually, FileSystem.get() or UserGroupInformation can retrieve tokens for you according to the items from hadoop's core-site.xml. 2. As I mentioned, for yarn mode. Yarn's NodeManager will download the tokens for each container, by default the token files will be in the workdir of that container. So the token file path is determined by runtime. User can't statically config the tokens for each container. 3. For security reasons, explicitly config the token files path in the tajo-site.xml is not safe for users. 4. Seems we can't use keytab here. If we have 1000 users, we need to enroll every user from the KDC. That's not very scalable I think. If the system has new come user, we need sync his/her keytabs to every nodes of the cluster. Hard to maintain. On the other hand, if you don't intend to distinguish users in tajo, then just use the default keytab for tajo workers seems more reasonable. Thanks for your contribution, Prafulla. Let's keep discussing. Min
          Hide
          prafulla Prafulla T added a comment -

          Hi Min,
          Thanks for your comment.
          Here is my response.

          For (1), The way I intend this to work is as follows.
          User will store following information in tajo-site.xml
          A. kerberos principal they can log in to hadoop namenode in dfs.namenode.kerberos.principal parameter
          B. keytab for this principal in tajo.dfs.namenode.keytab.file.
          Tajomaster will log in to hadoop using this information and request dfs delegation token.
          Now this token can be used to access hdfs by any user. Hadoop API that I am using takes only token information but I need to confirm whether current os user information is used/sent.
          The API that I have written is as follows :
          FileSystemUtil.getFileSystem(path, configuration).
          It does not depend on any hadoop specific information. All hadoop authentication related information is hidden inside this API. For new filesystem/storage ( such as s3 ), we can just modify (internal of) FileSystemUtil.getFileSystem to get correct instance of FileSystem and it should work.

          For (2), I am not yet familiar with kerberos in Yarn. I will test it and see if token file found in work directory can be used. That will make it easy.

          For (3), Token file path OR token itself will not be stored in tajo-site.xml OR any file for that matter.
          It will be stored in memory. I think for token expiry, it can be renewed at TajoMaster and that should be enough. I will test this assumption.

          For (4), yes, that's what I intended. User/keytab information will be used only at tajomaster.
          At workers and other nodes, we will just use token to access hdfs.

          Let me know what you think about this.

          Show
          prafulla Prafulla T added a comment - Hi Min, Thanks for your comment. Here is my response. For (1), The way I intend this to work is as follows. User will store following information in tajo-site.xml A. kerberos principal they can log in to hadoop namenode in dfs.namenode.kerberos.principal parameter B. keytab for this principal in tajo.dfs.namenode.keytab.file. Tajomaster will log in to hadoop using this information and request dfs delegation token. Now this token can be used to access hdfs by any user. Hadoop API that I am using takes only token information but I need to confirm whether current os user information is used/sent. The API that I have written is as follows : FileSystemUtil.getFileSystem(path, configuration). It does not depend on any hadoop specific information. All hadoop authentication related information is hidden inside this API. For new filesystem/storage ( such as s3 ), we can just modify (internal of) FileSystemUtil.getFileSystem to get correct instance of FileSystem and it should work. For (2), I am not yet familiar with kerberos in Yarn. I will test it and see if token file found in work directory can be used. That will make it easy. For (3), Token file path OR token itself will not be stored in tajo-site.xml OR any file for that matter. It will be stored in memory. I think for token expiry, it can be renewed at TajoMaster and that should be enough. I will test this assumption. For (4), yes, that's what I intended. User/keytab information will be used only at tajomaster. At workers and other nodes, we will just use token to access hdfs. Let me know what you think about this.
          Hide
          coderplay Min Zhou added a comment -

          Hi Prafulla,

          Thank you for your answer. I feel sorry that still can't catch your meaning. Did you intend to differentiate between multiple tajo users that use the same client machines?

          Thanks,
          Min

          Show
          coderplay Min Zhou added a comment - Hi Prafulla, Thank you for your answer. I feel sorry that still can't catch your meaning. Did you intend to differentiate between multiple tajo users that use the same client machines? Thanks, Min
          Hide
          prafulla Prafulla T added a comment -

          Hi Min,

          No problem.
          The way I am implementing it right now, I would not differentiate between multiple tajo users.
          From hadoop point of view, Tajo cluster will be using single user which is specified in (tajo.dfs.namenode.kerberos.principal).

          How do you suggest differentiating multiple tajo users?
          I will also think more about it.

          Show
          prafulla Prafulla T added a comment - Hi Min, No problem. The way I am implementing it right now, I would not differentiate between multiple tajo users. From hadoop point of view, Tajo cluster will be using single user which is specified in (tajo.dfs.namenode.kerberos.principal). How do you suggest differentiating multiple tajo users? I will also think more about it.
          Hide
          coderplay Min Zhou added a comment -

          Hi Prafulla,

          Thank you for the explanation. Make more sense for me now.

          Seems the original token is from TajoMaster, right? TajoWorker will get that token from TajoMaster through a RPC call, right? Unfortunately, the current RPC implementation in Tajo is lack of an SASL authetication mechanism. In another word, it's not safe. Anyone who can talk to TajoMaster through that RPC can impersonate a TajoWorker, getting those delegation tokens from TajoMaster without any identification. Do you agree that?

          Since you don't want to differentiate between multiple tajo users, I think we can design the security in tajo like HBase does.

          1. For standalone mode, each TajoWorker is a long-term daemon made up of the whole tajo cluster. How about keeping keytab files like tajo.keytab in a directory on each machine, only the OS user named "tajo" have access to it? This design isolate the keytab from the other users. Anyone except tajo can't touch that file. Furthermore, it avoid token passing through an unsafe RPC. This is similar with what HBase does.

          2. For yarn mode, since yarn can help users download a credentials file for each container and set an env variable named "HADOOP_TOKEN_FILE_LOCATION" point to that file. UserGroupInformation.getCurrentUser() can automatically load those credentials according to that env variable. see

          /**
             * Log in a user using the given subject
             * @parma subject the subject to use when logging in a user, or null to 
             * create a new subject.
             * @throws IOException if login fails
             */
            @InterfaceAudience.Public
            @InterfaceStability.Evolving
            public synchronized 
            static void loginUserFromSubject(Subject subject) throws IOException {
             ...
                String fileLocation = System.getenv(HADOOP_TOKEN_FILE_LOCATION);
                if (fileLocation != null) {
                  // Load the token storage file and put all of the tokens into the
                  // user. Don't use the FileSystem API for reading since it has a lock
                  // cycle (HADOOP-9212).
                  Credentials cred = Credentials.readTokenStorageFile(
                      new File(fileLocation), conf);
                  loginUser.addCredentials(cred);
                }
            ...
          }
          

          This is the reason why my patch worked. For this mode, still avoid token passing through unsafe RPC call.

          References
          http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf
          http://hbase.apache.org/book/security.html

          Show
          coderplay Min Zhou added a comment - Hi Prafulla, Thank you for the explanation. Make more sense for me now. Seems the original token is from TajoMaster, right? TajoWorker will get that token from TajoMaster through a RPC call, right? Unfortunately, the current RPC implementation in Tajo is lack of an SASL authetication mechanism. In another word, it's not safe. Anyone who can talk to TajoMaster through that RPC can impersonate a TajoWorker, getting those delegation tokens from TajoMaster without any identification. Do you agree that? Since you don't want to differentiate between multiple tajo users, I think we can design the security in tajo like HBase does. 1. For standalone mode, each TajoWorker is a long-term daemon made up of the whole tajo cluster. How about keeping keytab files like tajo.keytab in a directory on each machine, only the OS user named "tajo" have access to it? This design isolate the keytab from the other users. Anyone except tajo can't touch that file. Furthermore, it avoid token passing through an unsafe RPC. This is similar with what HBase does. 2. For yarn mode, since yarn can help users download a credentials file for each container and set an env variable named "HADOOP_TOKEN_FILE_LOCATION" point to that file. UserGroupInformation.getCurrentUser() can automatically load those credentials according to that env variable. see /** * Log in a user using the given subject * @parma subject the subject to use when logging in a user, or null to * create a new subject. * @throws IOException if login fails */ @InterfaceAudience.Public @InterfaceStability.Evolving public synchronized static void loginUserFromSubject(Subject subject) throws IOException { ... String fileLocation = System.getenv(HADOOP_TOKEN_FILE_LOCATION); if (fileLocation != null) { // Load the token storage file and put all of the tokens into the // user. Don't use the FileSystem API for reading since it has a lock // cycle (HADOOP-9212). Credentials cred = Credentials.readTokenStorageFile( new File(fileLocation), conf); loginUser.addCredentials(cred); } ... } This is the reason why my patch worked. For this mode, still avoid token passing through unsafe RPC call. References http://hortonworks.com/wp-content/uploads/2011/10/security-design_withCover-1.pdf http://hbase.apache.org/book/security.html
          Hide
          prafulla Prafulla T added a comment -

          Thanks Min, Yes I agree that it would not be safe to pass delegation token like the way I pass currently. That's why I had proposed new feature to support some way to safely exchange information within Tajo cluster (https://issues.apache.org/jira/browse/TAJO-858?focusedCommentId=14021514&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14021514 , Point 3 ).

          But your proposed design will not need us to have that feature. I will think more about it and go through links.

          Show
          prafulla Prafulla T added a comment - Thanks Min, Yes I agree that it would not be safe to pass delegation token like the way I pass currently. That's why I had proposed new feature to support some way to safely exchange information within Tajo cluster ( https://issues.apache.org/jira/browse/TAJO-858?focusedCommentId=14021514&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14021514 , Point 3 ). But your proposed design will not need us to have that feature. I will think more about it and go through links.
          Hide
          coderplay Min Zhou added a comment -

          Hi Prafulla,

          My fault hasn't notice you already commented on that point.

          Show
          coderplay Min Zhou added a comment - Hi Prafulla, My fault hasn't notice you already commented on that point.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/30

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/30
          Hide
          jhkim Jinho Kim added a comment -

          +1 Min Zhou's patch.

          Currently, TajoClient is no longer reading directly from hdfs by TAJO-1160. so, we don’t need SIMPLE authentication in FileScanner.
          If there are no objection until tomorrow, I'll commit it

          Thanks

          Show
          jhkim Jinho Kim added a comment - +1 Min Zhou's patch. Currently, TajoClient is no longer reading directly from hdfs by TAJO-1160 . so, we don’t need SIMPLE authentication in FileScanner. If there are no objection until tomorrow, I'll commit it Thanks
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #675 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/675/)
          TAJO-858: Support for hadoop kerberos authentication in Tajo. (jhkim: rev 2fc99117f5fe45f0d8cdc862601cba639da27956)

          • CHANGES
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileScanner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #675 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/675/ ) TAJO-858 : Support for hadoop kerberos authentication in Tajo. (jhkim: rev 2fc99117f5fe45f0d8cdc862601cba639da27956) CHANGES tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileScanner.java
          Hide
          jhkim Jinho Kim added a comment -

          committed it
          Thanks

          Show
          jhkim Jinho Kim added a comment - committed it Thanks
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #1079 (See https://builds.apache.org/job/Tajo-master-build/1079/)
          TAJO-858: Support for hadoop kerberos authentication in Tajo. (jhkim: rev 2fc99117f5fe45f0d8cdc862601cba639da27956)

          • CHANGES
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileScanner.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #1079 (See https://builds.apache.org/job/Tajo-master-build/1079/ ) TAJO-858 : Support for hadoop kerberos authentication in Tajo. (jhkim: rev 2fc99117f5fe45f0d8cdc862601cba639da27956) CHANGES tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/FileScanner.java

            People

            • Assignee:
              prafulla Prafulla T
              Reporter:
              coderplay Min Zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development