Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-2763

Add /user/zeppelin directory for running Spark job on YARN in Zeppelin

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.2.1
    • Component/s: deployment
    • Labels:
      None

      Description

      To make running Spark on YARN with Zeppelin, it requires '/user/zeppelin' home directory for zeppelin user:

      ERROR [2017-04-25 14:47:32,905] ({pool-2-thread-2} Logging.scala[logError]:91) - Error initializing SparkContext.
      org.apache.hadoop.security.AccessControlException: Permission denied: user=zeppelin, access=WRITE, inode="/user/zeppelin/.sparkStaging/application_1493099537608_0001":hdfs:hdfs:drwxr-xr-x
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:320)
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:292)
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:213)
              at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1728)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1712)
              at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1695)
              at org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:71)
              at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3896)
              at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:984)
              at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:622)
              at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
              at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
              at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
              at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
              at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2045)
      
      
      1. BIGTOP-2763.patch
        0.9 kB
        YoungWoo Kim

        Activity

        Hide
        evans_ye Evans Ye added a comment -

        If it's for zeppelin server to write, isn't it better to be something like 744 or else?
        Sorry I'm not familiar with zeppelin at all.

        Show
        evans_ye Evans Ye added a comment - If it's for zeppelin server to write, isn't it better to be something like 744 or else? Sorry I'm not familiar with zeppelin at all.
        Hide
        warwithin YoungWoo Kim added a comment - - edited

        Evans Ye I'm not sure which one is correct. I just gave 777 to /user/zeppelin because existing entries had 777 permission for user's home directory on HDFS, like following:

            ["/user/jenkins","777","jenkins",null],
            ["/user/vagrant","777","vagrant",null],
            ["/user/hive","777",null,null],
            ["/user/root","777","root",null],
            ["/user/hue","777","hue","hue"],
            ["/user/sqoop","777","sqoop",null],
            ["/user/oozie","777","oozie"],
        
        Show
        warwithin YoungWoo Kim added a comment - - edited Evans Ye I'm not sure which one is correct. I just gave 777 to /user/zeppelin because existing entries had 777 permission for user's home directory on HDFS, like following: ["/user/jenkins","777","jenkins",null], ["/user/vagrant","777","vagrant",null], ["/user/hive","777",null,null], ["/user/root","777","root",null], ["/user/hue","777","hue","hue"], ["/user/sqoop","777","sqoop",null], ["/user/oozie","777","oozie"],
        Hide
        evans_ye Evans Ye added a comment -

        OK. Let's let the other JIRA to address this problem.
        I've committed this. Thanks YoungWoo Kim.

        Show
        evans_ye Evans Ye added a comment - OK. Let's let the other JIRA to address this problem. I've committed this. Thanks YoungWoo Kim .

          People

          • Assignee:
            warwithin YoungWoo Kim
            Reporter:
            warwithin YoungWoo Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development