Apache S4
  1. Apache S4
  2. S4-25

Write S4 Application Master to deploy S4 in Yarn

    Details

    • Type: New Feature New Feature
    • Status: Patch Available
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.7
    • Labels:
      None

      Description

      On the lines of s4PigWrapper, write a s4 application master to host s4 piper inside Hadoop Yarn. This could be useful not only for reading data stored in hadoop ( to build or train a model)... But we could make use of the resource manager to deploy s4 instances in remote machine and monitor them. In short, we could make use of most of the resource management , scheduling and other good stuff in Yarn.

      • Yarn is useful to deploy and launch s4 instances.
      • It still requires deploying node managers on each box which means it will
        be useful if one is running more than one s4 process on a node.
      1. S4-YARN-1.patch
        74 kB
        Sean Mackrory
      2. S4-Constants.diff
        2 kB
        Sean Mackrory
      3. S4-Client.diff
        13 kB
        Sean Mackrory
      4. S4-ApplicationMaster.diff
        13 kB
        Sean Mackrory

        Activity

        Hide
        Matthieu Morel added a comment -

        Patch available in branch S4-25, commit 830f5df

        Currently this patch can be applied on top of S4 0.5 and YARN 2.0.2-alpha

        Documentation here: https://cwiki.apache.org/S4/deploying-s4-applications-with-yarn.html

        Show
        Matthieu Morel added a comment - Patch available in branch S4-25 , commit 830f5df Currently this patch can be applied on top of S4 0.5 and YARN 2.0.2-alpha Documentation here: https://cwiki.apache.org/S4/deploying-s4-applications-with-yarn.html
        Hide
        Matthieu Morel added a comment -

        Thanks for the comments, I've uploaded a new version of the patch in branch S4-25, commit 830f5df

        By default, Application Master and S4 node JVMs are started with a maximum heap size of 80% the memory for the container. This can be overridden through JVM parameters, when the user has specific knowledge about the application. We still use Xmx to compute the resource allocation though, since, AFAIK, we cannot compute the exact memory usage automatically with available parameters.

        If the specified memory exceeds the available memory, Yarn will simply log an error message and not start the JVMs.

        Show
        Matthieu Morel added a comment - Thanks for the comments, I've uploaded a new version of the patch in branch S4-25 , commit 830f5df By default, Application Master and S4 node JVMs are started with a maximum heap size of 80% the memory for the container. This can be overridden through JVM parameters, when the user has specific knowledge about the application. We still use Xmx to compute the resource allocation though, since, AFAIK, we cannot compute the exact memory usage automatically with available parameters. If the specified memory exceeds the available memory, Yarn will simply log an error message and not start the JVMs.
        Hide
        Flavio Junqueira added a comment -

        Decoupling seems ok, but they are still related, so it would be good to check that the container memory size is greater than the heap size of the jvm, no?

        Show
        Flavio Junqueira added a comment - Decoupling seems ok, but they are still related, so it would be good to check that the container memory size is greater than the heap size of the jvm, no?
        Hide
        Daniel Gómez Ferro added a comment -

        Thanks for the update.

        I think I wasn't clear enough on my previous comment. My idea was to decouple the "Container Memory" from the "Node/JVM Memory", since the JVM memory we set through -Xmx has to be strictly less than the reserved memory at the container level, otherwise we could run into the issues reported by Frank Zheng on the mailing list (see http://mail-archives.apache.org/mod_mbox/incubator-s4-user/201211.mbox/%3CCAAf2GfeQK_-zx7sCvpP_9euTsDJL0VbDTS08iXMyQw1%3D8%2B1tOg%40mail.gmail.com%3E ) What I propose is to have two separate parameters, one for the container memory and another for the jvm memory.

        Do you think that's too much complexity?

        There's a minor issue in the S4ApplicationMaster, it's not taking into account the -s4NodeMemory parameter to compute the container memory (it always picks YARN's minimum container memory regardless of the passed parameters).

        Show
        Daniel Gómez Ferro added a comment - Thanks for the update. I think I wasn't clear enough on my previous comment. My idea was to decouple the "Container Memory" from the "Node/JVM Memory", since the JVM memory we set through -Xmx has to be strictly less than the reserved memory at the container level, otherwise we could run into the issues reported by Frank Zheng on the mailing list (see http://mail-archives.apache.org/mod_mbox/incubator-s4-user/201211.mbox/%3CCAAf2GfeQK_-zx7sCvpP_9euTsDJL0VbDTS08iXMyQw1%3D8%2B1tOg%40mail.gmail.com%3E ) What I propose is to have two separate parameters, one for the container memory and another for the jvm memory. Do you think that's too much complexity? There's a minor issue in the S4ApplicationMaster, it's not taking into account the -s4NodeMemory parameter to compute the container memory (it always picks YARN's minimum container memory regardless of the passed parameters).
        Hide
        Matthieu Morel added a comment -

        Thanks for the feedback Daniel.

        I pushed some related updates in commit ba7b94b .

        The -Xmx parameter, which is used for setting the "maximum" memory of the process, can now only be specified through a specific parameter (-s4NodeMemory).

        I also asked on yarn-dev whether there is a recommended way to properly kill applications.
        (see http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201211.mbox/%3CE446FDC3-C0A3-4DA8-8281-A8CD15CC8D52%40yahoo-inc.com%3E )

        Show
        Matthieu Morel added a comment - Thanks for the feedback Daniel. I pushed some related updates in commit ba7b94b . The -Xmx parameter, which is used for setting the "maximum" memory of the process, can now only be specified through a specific parameter (-s4NodeMemory). I also asked on yarn-dev whether there is a recommended way to properly kill applications. (see http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201211.mbox/%3CE446FDC3-C0A3-4DA8-8281-A8CD15CC8D52%40yahoo-inc.com%3E )
        Hide
        Daniel Gómez Ferro added a comment -

        The patch looks good and it works perfectly. Great feature!

        I just have a couple of minor comments:

        There is a Thread.sleep(10000) in S4ApplicationMaster.main() which I don't think it's needed.

        For the memory, I'd let the user set it. It he doesn't, I'd just scale the memory reserved for the container by a fixed factor, maybe 0.8. So, for the -Xmx parameter I'd use a different parameter (-jvmMemory, -nodeMemory?) and if it's not set use containerMemory * 0.8.

        I think that YARN stopping only the application master is a YARN bug. I'd open a ticket, if it turns out there's a better way to stop it they'll tell us.

        Show
        Daniel Gómez Ferro added a comment - The patch looks good and it works perfectly. Great feature! I just have a couple of minor comments: There is a Thread.sleep(10000) in S4ApplicationMaster.main() which I don't think it's needed. For the memory, I'd let the user set it. It he doesn't, I'd just scale the memory reserved for the container by a fixed factor, maybe 0.8. So, for the -Xmx parameter I'd use a different parameter (-jvmMemory, -nodeMemory?) and if it's not set use containerMemory * 0.8. I think that YARN stopping only the application master is a YARN bug. I'd open a ticket, if it turns out there's a better way to stop it they'll tell us.
        Hide
        Matthieu Morel added a comment - - edited

        I just uploaded some updates to the previous patch, in commit a021481

        It includes improvements to parameters parsing, and in particular, the handling of memory and other JVM parameters. The memory parameter passed to YARN is used to set the maximum heap size, and thus is actually less than the maximum amount of memory that can be required by S4 nodes, since that would also include thread stack size and other things, and I'm not sure how to exactly determine the total maximum memory of the VM before launching it.

        Other pending issues are still : stopping the application (rather than just the application master) and test dependencies.

        Show
        Matthieu Morel added a comment - - edited I just uploaded some updates to the previous patch, in commit a021481 It includes improvements to parameters parsing, and in particular, the handling of memory and other JVM parameters. The memory parameter passed to YARN is used to set the maximum heap size, and thus is actually less than the maximum amount of memory that can be required by S4 nodes, since that would also include thread stack size and other things, and I'm not sure how to exactly determine the total maximum memory of the VM before launching it. Other pending issues are still : stopping the application (rather than just the application master) and test dependencies.
        Hide
        Matthieu Morel added a comment -

        I uploaded a patch in branch S4-25 (here https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=shortlog;h=refs/heads/S4-25), and added some documentation here : https://cwiki.apache.org/confluence/display/S4/Deploying+S4+applications+with+YARN

        The approach is to preserve S4 deployment model (coordination through ZooKeeper, application loading logic in the S4 nodes), and make a projection on YARN in order to start S4 nodes.

        The patch depends on hadoop-2.0.2-alpha, the latest release.

        The patch adds a new subproject, s4-yarn and provides the s4-yarn command to deploy S4 applications. You can combine S4 parameters as well as YARN specific parameters (num_containers, queue, user etc...)

        I also added a regression test that uses MiniYARNCluster and MiniDFSCluster.

        Pending issues:

        • It's not clear to me how to stop an application. The YarnClientImpl#killApplication method seems to kill the application master, but not the processes launched by this application master
        • I could not figure how to add yarn test dependencies. That may be a gradle issue, or the way hadoop-2.0.2-alpha packages are distributed on maven. Not sure. In the meantime, I added them to a local lib/ directory of the S4 distribution

        Arun: because we used a released version of Yarn, we used the raw API, not YARN-103

        Show
        Matthieu Morel added a comment - I uploaded a patch in branch S4-25 (here https://git-wip-us.apache.org/repos/asf?p=incubator-s4.git;a=shortlog;h=refs/heads/S4-25 ), and added some documentation here : https://cwiki.apache.org/confluence/display/S4/Deploying+S4+applications+with+YARN The approach is to preserve S4 deployment model (coordination through ZooKeeper, application loading logic in the S4 nodes), and make a projection on YARN in order to start S4 nodes. The patch depends on hadoop-2.0.2-alpha, the latest release. The patch adds a new subproject, s4-yarn and provides the s4-yarn command to deploy S4 applications. You can combine S4 parameters as well as YARN specific parameters (num_containers, queue, user etc...) I also added a regression test that uses MiniYARNCluster and MiniDFSCluster. Pending issues: It's not clear to me how to stop an application. The YarnClientImpl#killApplication method seems to kill the application master, but not the processes launched by this application master I could not figure how to add yarn test dependencies. That may be a gradle issue, or the way hadoop-2.0.2-alpha packages are distributed on maven. Not sure. In the meantime, I added them to a local lib/ directory of the S4 distribution Arun: because we used a released version of Yarn, we used the raw API, not YARN-103
        Hide
        Arun C Murthy added a comment -

        Sean and Matthieu, I encourage you guys to look at YARN-103 to help ease your S4-AM implementation and to ensure you don't have to be exposed to the nitty-gritty details. Thanks.

        Show
        Arun C Murthy added a comment - Sean and Matthieu, I encourage you guys to look at YARN-103 to help ease your S4-AM implementation and to ensure you don't have to be exposed to the nitty-gritty details. Thanks.
        Hide
        Sean Mackrory added a comment -

        I'm glad you were able to make it work! I've since learned that my understanding of S4 deployment might have been entirely backwards - so the actions in the App Master and Container might be in the wrong place. I apologize I haven't had time recently to keep working on this, but when I do I'll check in and see what else I might be able to help with.

        Show
        Sean Mackrory added a comment - I'm glad you were able to make it work! I've since learned that my understanding of S4 deployment might have been entirely backwards - so the actions in the App Master and Container might be in the wrong place. I apologize I haven't had time recently to keep working on this, but when I do I'll check in and see what else I might be able to help with.
        Hide
        Matthieu Morel added a comment -

        Thanks Sean for contributing this patch!

        I had a look today, and I was able to make things work. There were a few minor (but blocking) issues in the Client and ApplicationMaster, and things we'll have to polish for the classpath of the submitted S4 job.

        The issue you reported is due to the following (and I agree the error message from Yarn is not very helpful for non-initiated!):

        • When creating a connection to the resource manager, the client needs to use the resource manager ports, not the scheduler ones (YarnConfiguration.RM_ADDRESS,YarnConfiguration.DEFAULT_RM_ADDRESS). The scheduler is initialized differently and cannot understand the "ClientRMProtocolPB"

        I'll have a further look at how to provide a clean integration, but the approach you took looks like a promising one!

        Show
        Matthieu Morel added a comment - Thanks Sean for contributing this patch! I had a look today, and I was able to make things work. There were a few minor (but blocking) issues in the Client and ApplicationMaster, and things we'll have to polish for the classpath of the submitted S4 job. The issue you reported is due to the following (and I agree the error message from Yarn is not very helpful for non-initiated!): When creating a connection to the resource manager, the client needs to use the resource manager ports, not the scheduler ones (YarnConfiguration.RM_ADDRESS,YarnConfiguration.DEFAULT_RM_ADDRESS). The scheduler is initialized differently and cannot understand the "ClientRMProtocolPB" I'll have a further look at how to provide a clean integration, but the approach you took looks like a promising one!
        Hide
        Sean Mackrory added a comment -

        Actually I've gone ahead and attached the diffs from distributedshell to make it easier for anyone familiar with the distributedshell example to see what I've done.

        Show
        Sean Mackrory added a comment - Actually I've gone ahead and attached the diffs from distributedshell to make it easier for anyone familiar with the distributedshell example to see what I've done.
        Hide
        Sean Mackrory added a comment -

        I have attached a first attempt, although it is still a work in progress. I expect to have more free time to further complete this over the next couple of weeks, but I wanted to share my progress so far. I will replace the attachment with new patches as I add functionality.

        This is based very closely on the distributedshell example shipped with YARN 2.0.1. There are a lot of changes that ought to be made (removing code that's only there to demonstrate features, documentation, etc.), but I am waiting until it's functional so I can make a minimal diff of the changes required for S4 actions (which I suggest doing if you're going to look at this code).

        Feedback is obviously welcome - as this is my first work with Gradle, YARN or S4. I've filled out the ApplicationMaster with what I think should be there, although it is completely untested (it does compile) and is probably wrong. I am running Hadoop / YARN in pseudo-distributed mode, and am attempting to deploy the app on the "cluster" with the following command:

        ./s4 yarn \
            -jar `pwd`/subprojects/s4-tools/build/libs/s4-tools-0.5.0-incubating.jar \
            -cluster=cluster1 \
            -nbTasks=2 \
            -flp=12000 \
            -s4r=`pwd`/build/libs/myApp.s4r
        

        Currently it errors out when the Client communicates with the ResourceManager (it first occurs in some of the unncessary demo code, but will eventually occur anyway). The corresponding log message on the Client side is:

        584 [main] FATAL org.apache.s4.tools.yarn.Client  - Error running CLient
        java.lang.reflect.UndeclaredThrowableException
        	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
        	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:123)
        	at org.apache.s4.tools.yarn.Client.run(Client.java:331)
        	at org.apache.s4.tools.yarn.Client.main(Client.java:184)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:55)
        	at org.apache.s4.tools.Tools.main(Tools.java:95)
        Caused by: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409)
        	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
        
        	at org.apache.hadoop.ipc.Client.call(Client.java:1161)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
        	at $Proxy7.getClusterMetrics(Unknown Source)
        	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:121)
        	... 8 more
        

        And the corresponding log message on the ResourceManager side is:

        lPB.getClusterNodes from 127.0.1.1:51347: error: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
        java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371)
        	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409)
        	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
        	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
        	at java.security.AccessController.doPrivileged(Native Method)
        	at javax.security.auth.Subject.doAs(Subject.java:396)
        	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
        

        Please let me know if you have any feedback or hints on the "Unknown protocol" problem.

        Show
        Sean Mackrory added a comment - I have attached a first attempt, although it is still a work in progress. I expect to have more free time to further complete this over the next couple of weeks, but I wanted to share my progress so far. I will replace the attachment with new patches as I add functionality. This is based very closely on the distributedshell example shipped with YARN 2.0.1. There are a lot of changes that ought to be made (removing code that's only there to demonstrate features, documentation, etc.), but I am waiting until it's functional so I can make a minimal diff of the changes required for S4 actions (which I suggest doing if you're going to look at this code). Feedback is obviously welcome - as this is my first work with Gradle, YARN or S4. I've filled out the ApplicationMaster with what I think should be there, although it is completely untested (it does compile) and is probably wrong. I am running Hadoop / YARN in pseudo-distributed mode, and am attempting to deploy the app on the "cluster" with the following command: ./s4 yarn \ -jar `pwd`/subprojects/s4-tools/build/libs/s4-tools-0.5.0-incubating.jar \ -cluster=cluster1 \ -nbTasks=2 \ -flp=12000 \ -s4r=`pwd`/build/libs/myApp.s4r Currently it errors out when the Client communicates with the ResourceManager (it first occurs in some of the unncessary demo code, but will eventually occur anyway). The corresponding log message on the Client side is: 584 [main] FATAL org.apache.s4.tools.yarn.Client - Error running CLient java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:123) at org.apache.s4.tools.yarn.Client.run(Client.java:331) at org.apache.s4.tools.yarn.Client.main(Client.java:184) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:55) at org.apache.s4.tools.Tools.main(Tools.java:95) Caused by: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686) at org.apache.hadoop.ipc.Client.call(Client.java:1161) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184) at $Proxy7.getClusterMetrics(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:121) ... 8 more And the corresponding log message on the ResourceManager side is: lPB.getClusterNodes from 127.0.1.1:51347: error: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686) Please let me know if you have any feedback or hints on the "Unknown protocol" problem.
        Hide
        Sean Mackrory added a comment -

        I'm afraid it's going slowly due to other time constraints at the moment. There have been several people who have said they had already done a basic PoC of this - so if anybody would care to post that code I can pick up where they left off.

        Show
        Sean Mackrory added a comment - I'm afraid it's going slowly due to other time constraints at the moment. There have been several people who have said they had already done a basic PoC of this - so if anybody would care to post that code I can pick up where they left off.
        Hide
        Daniel Gómez Ferro added a comment -

        How are things progressing?

        Adding to the discussion on the mailing list, maybe for the first implementation we should skip the configuration and deployment steps, letting the user do that using the s4 script.

        Show
        Daniel Gómez Ferro added a comment - How are things progressing? Adding to the discussion on the mailing list, maybe for the first implementation we should skip the configuration and deployment steps, letting the user do that using the s4 script.
        Hide
        J Mohamed Zahoor added a comment -

        We did this 6 months back and it worked great. Need to dig out the code from my backup. It might need some refactoring for the latest releases (YARN and piper) though.

        Show
        J Mohamed Zahoor added a comment - We did this 6 months back and it worked great. Need to dig out the code from my backup. It might need some refactoring for the latest releases (YARN and piper) though.
        Hide
        Arun C Murthy added a comment -

        Sounds great Sean! Please LMK and I'm happy to help with any issue you may face while integrating with YARN.

        Show
        Arun C Murthy added a comment - Sounds great Sean! Please LMK and I'm happy to help with any issue you may face while integrating with YARN.
        Hide
        Sean Mackrory added a comment -

        There was some discussion on the mailing list relevant to this: http://mail-archives.apache.org/mod_mbox/incubator-s4-dev/201207.mbox/%3C5006E90A.1020204@yahoo-inc.com%3E.

        I plan on working on this in some of my spare time. When I have a working proof-of-concept I will post the code.

        Show
        Sean Mackrory added a comment - There was some discussion on the mailing list relevant to this: http://mail-archives.apache.org/mod_mbox/incubator-s4-dev/201207.mbox/%3C5006E90A.1020204@yahoo-inc.com%3E . I plan on working on this in some of my spare time. When I have a working proof-of-concept I will post the code.

          People

          • Assignee:
            Unassigned
            Reporter:
            J Mohamed Zahoor
          • Votes:
            0 Vote for this issue
            Watchers:
            10 Start watching this issue

            Dates

            • Created:
              Updated:

              Development