Details
-
New Feature
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
0.5.0-incubating
-
None
Description
I made some extensions to make twill working for mapr. Besides of what I included in github, there are some other issues. Currently, I have to include maprfs library in yarn-example pom file, then related jars will be bundled and deployed into yarn. These libraries, however, are included in the HADOOP_CLASS, I don't know why it is not been loading when twill/yarn application runs. So if we don't are not using bundle jar application while let twill to scan dependent classes (like CDAP startup services did), the jars are missing. The way to make it work right now is a hacking:
1. add new MapRFileSystem() in the code of TwillContainerMain and Hadoop21YarnAppClient class, so that the dependency will be included, even though it does nothing. BTW, I think it works for other hadoop distribution, because there two classes already depends the hdfs related stuff and jars are packaged into container.jar and application jar (which I think is not necessary since very yarn node should already contain these jars)
2. Add twillRunner.setJVMOptions("-Djava.library.path=/opt/mapr/lib") when run the twill application.
I think it is really not the right way to do, but not sure how to avoid that, just let yarn/twill know to include HADOOP_CLASSPATH when run the application?