--- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md 2017-07-12 18:29:30.000000000 -0700 +++ hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md 2017-07-12 22:14:36.000000000 -0700 @@ -159,7 +159,7 @@ |:---- |:---- | |`yarn.federation.enabled` | `true` | Whether federation is enabled or not | |`yarn.federation.state-store.class` | `org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore` | The type of state-store to use. | -|`yarn.federation.state-store.sql.url` | `jdbc:sqlserver://:;database` | For SQLFederationStateStore the name of the DB where the state is stored. | +|`yarn.federation.state-store.sql.url` | `jdbc:sqlserver://:;databaseName=FederationStateStore` | For SQLFederationStateStore the name of the DB where the state is stored. | |`yarn.federation.state-store.sql.jdbc-class` | `com.microsoft.sqlserver.jdbc.SQLServerDataSource` | For SQLFederationStateStore the jdbc class to use. | |`yarn.federation.state-store.sql.username` | `` | For SQLFederationStateStore the username for the DB connection. | |`yarn.federation.state-store.sql.password` | `` | For SQLFederationStateStore the password for the DB connection. | @@ -200,6 +200,7 @@ | Property | Example | Description | |:---- |:---- | |`yarn.router.bind-host` | `0.0.0.0` | Host IP to bind the router to. The actual address the server will bind to. If this optional address is set, the RPC and webapp servers will bind to this address and the port specified in yarn.router.*.address respectively. This is most useful for making Router listen to all interfaces by setting to 0.0.0.0. | +| `yarn.router.clientrm.interceptor-class.pipeline` | `org.apache.hadoop.yarn.server.router.clientrm.FederationClientInterceptor` | A comma-seperated list of interceptor classes to be run at the router when interfacing with the client. The last step of this pipeline must be the Federation Client Interceptor. | Optional: @@ -222,11 +223,42 @@ | Property | Example | Description | |:---- |:---- | -|`yarn.nodemanager.amrmproxy.interceptor-class.pipeline` | `org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor` | A coma-separated list of interceptors to be run at the amrmproxy. For federation the last step in the pipeline should be the FederationInterceptor. | +| `yarn.nodemanager.amrmproxy.enabled` | `true` | Whether or not the AMRMProxy is enabled. +|`yarn.nodemanager.amrmproxy.interceptor-class.pipeline` | `org.apache.hadoop.yarn.server.nodemanager.amrmproxy.FederationInterceptor` | A comma-separated list of interceptors to be run at the amrmproxy. For federation the last step in the pipeline should be the FederationInterceptor. +| `yarn.client.failover.proxy-provider` | `org.apache.hadoop.yarn.server.federation.failover.FederationRMFailoverProxyProvider` | The class used to connect to the federation state-store and get membership information. This must be set even if HA is not enabled.| Optional: | Property | Example | Description | |:---- |:---- | |`yarn.federation.statestore.max-connections` | `1` | The maximum number of parallel connections from each AMRMProxy to the state-store. This value is typically lower than the router one, since we have many AMRMProxy that could burn-through many DB connections quickly. | -|`yarn.federation.cache-ttl.secs` | `300` | The time to leave for the AMRMProxy cache. Typically larger than at the router, as the number of AMRMProxy is large, and we want to limit the load to the centralized state-store. | \ No newline at end of file +|`yarn.federation.cache-ttl.secs` | `300` | The time to leave for the AMRMProxy cache. Typically larger than at the router, as the number of AMRMProxy is large, and we want to limit the load to the centralized state-store. | + +Running a Sample Job +-------------------- +In order to submit jobs to a Federation cluster one must create a seperate set of configs for the client from which said jobs will be submitted. In these, the **conf/yarn-site.xml** should have the following additional configurations: + +| Property | Example | Description | +|:--- |:--- | +| `yarn.resourcemanager.address` | `:8050` | Redirects jobs launched at the client to the router's client RM port. | +| `yarn.resourcemanger.scheduler.address` | `localhost:8049` | Redirects jobs to the federation AMRMProxy port.| + +In order to launch the router, you must run + + $HADOOP_HOME/bin/yarn --daemon start router + +from the router machine. Any YARN jobs for the cluster can be submitted from the client configurations described above. In order to launch a job through federation, first start up all the clusters involved in the federation as described [here](../../hadoop-project-dist/hadoop-common/ClusterSetup.html). Next, start up the router on the router machine with the following command: + + $HADOOP_HOME/bin/yarn --daemon start router + +Now with $HADOOP_CONF_DIR pointing to the client configurations folder that is described above, run your job the usual way. Here is an example of a run of a Pi job on a federation cluster from the client: + + $HADOOP_HOME/bin/yarn jar hadoop-mapreduce-examples-3.0.0.jar pi 16 1000 + +This job is submitted to the router which as described above, uses a generated policy to pick a home RM for the job to which it is +submitted. + +The output from this particular example job should be: + + Estimated value of Pi is 3.14250000...... +