Uploaded image for project: 'Apache Twill'
  1. Apache Twill
  2. TWILL-119

Updates to work on a secure cluster with HA ResourceManager

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: yarn
    • Labels:
      None

      Description

      I have not tried it yet, but it seems like we may need some extra work for Twill to run on a secure cluster that uses HA ResourceManager. The YarnUtils class gets the socket addr for the RM using YarnConfiguration.RM_ADDRESS, but in HA mode the conf contains YarnConfiguration.RM_HA_IDS, with each id having a corresponding hostname setting.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user chtyim opened a pull request:

          https://github.com/apache/twill/pull/38

          (TWILL-119) Make YarnAppClient supports HA RM

          • HA RM is added since Hadoop 2.3.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/chtyim/twill feature/TWILL-119

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/twill/pull/38.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #38


          commit b4edd6bac42ccb58eeff36913a00ee1df3a468f5
          Author: Terence Yim <chtyim@apache.org>
          Date: 2017-03-17T21:17:47Z

          (TWILL-119) Make YarnAppClient supports HA RM

          • HA RM is added since Hadoop 2.3.

          Show
          githubbot ASF GitHub Bot added a comment - GitHub user chtyim opened a pull request: https://github.com/apache/twill/pull/38 ( TWILL-119 ) Make YarnAppClient supports HA RM HA RM is added since Hadoop 2.3. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chtyim/twill feature/ TWILL-119 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/38.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #38 commit b4edd6bac42ccb58eeff36913a00ee1df3a468f5 Author: Terence Yim <chtyim@apache.org> Date: 2017-03-17T21:17:47Z ( TWILL-119 ) Make YarnAppClient supports HA RM HA RM is added since Hadoop 2.3.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/38#discussion_r106763105

          — Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAppClient.java —
          @@ -211,7 +218,6 @@ public YarnApplicationReport getReport() {
          try

          { return new Hadoop20YarnApplicationReport(yarnClient.getApplicationReport(appId)); }

          catch (YarnRemoteException e) {

          • LOG.error("Failed to get application report {}", appId, e);
              • End diff –

          removing the logging here is good. However, the messages could add useful context (here: the appId). If you wrap it into a RuntimeException, why not add the message?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/38#discussion_r106763105 — Diff: twill-yarn/src/main/hadoop20/org/apache/twill/internal/yarn/Hadoop20YarnAppClient.java — @@ -211,7 +218,6 @@ public YarnApplicationReport getReport() { try { return new Hadoop20YarnApplicationReport(yarnClient.getApplicationReport(appId)); } catch (YarnRemoteException e) { LOG.error("Failed to get application report {}", appId, e); End diff – removing the logging here is good. However, the messages could add useful context (here: the appId). If you wrap it into a RuntimeException, why not add the message?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/38#discussion_r106763328

          — Diff: twill-yarn/src/main/hadoop23/org/apache/twill/internal/yarn/Hadoop23YarnAppClient.java —
          @@ -0,0 +1,111 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.yarn;
          +
          +import com.google.common.base.Joiner;
          +import com.google.common.base.Throwables;
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.io.Text;
          +import org.apache.hadoop.security.Credentials;
          +import org.apache.hadoop.security.SecurityUtil;
          +import org.apache.hadoop.security.UserGroupInformation;
          +import org.apache.hadoop.security.token.Token;
          +import org.apache.hadoop.security.token.TokenIdentifier;
          +import org.apache.hadoop.yarn.api.records.ApplicationId;
          +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
          +import org.apache.hadoop.yarn.client.api.YarnClient;
          +import org.apache.hadoop.yarn.conf.HAUtil;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.util.ConverterUtils;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.net.InetSocketAddress;
          +import java.util.ArrayList;
          +import java.util.List;
          +
          +/**
          + * <p>
          + * The service implementation of

          {@link YarnAppClient}

          for Apache Hadoop 2.1 and beyond.
          — End diff –

          2.3

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/38#discussion_r106763328 — Diff: twill-yarn/src/main/hadoop23/org/apache/twill/internal/yarn/Hadoop23YarnAppClient.java — @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.yarn; + +import com.google.common.base.Joiner; +import com.google.common.base.Throwables; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.security.Credentials; +import org.apache.hadoop.security.SecurityUtil; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.security.token.TokenIdentifier; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; +import org.apache.hadoop.yarn.client.api.YarnClient; +import org.apache.hadoop.yarn.conf.HAUtil; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.util.ConverterUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.List; + +/** + * <p> + * The service implementation of {@link YarnAppClient} for Apache Hadoop 2.1 and beyond. — End diff – 2.3
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/38#discussion_r106763336

          — Diff: twill-yarn/src/main/hadoop23/org/apache/twill/internal/yarn/Hadoop23YarnAppClient.java —
          @@ -0,0 +1,111 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.yarn;
          +
          +import com.google.common.base.Joiner;
          +import com.google.common.base.Throwables;
          +import org.apache.hadoop.conf.Configuration;
          +import org.apache.hadoop.io.Text;
          +import org.apache.hadoop.security.Credentials;
          +import org.apache.hadoop.security.SecurityUtil;
          +import org.apache.hadoop.security.UserGroupInformation;
          +import org.apache.hadoop.security.token.Token;
          +import org.apache.hadoop.security.token.TokenIdentifier;
          +import org.apache.hadoop.yarn.api.records.ApplicationId;
          +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext;
          +import org.apache.hadoop.yarn.client.api.YarnClient;
          +import org.apache.hadoop.yarn.conf.HAUtil;
          +import org.apache.hadoop.yarn.conf.YarnConfiguration;
          +import org.apache.hadoop.yarn.util.ConverterUtils;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.net.InetSocketAddress;
          +import java.util.ArrayList;
          +import java.util.List;
          +
          +/**
          + * <p>
          + * The service implementation of

          {@link YarnAppClient}

          for Apache Hadoop 2.1 and beyond.
          + *
          + * The

          {@link VersionDetectYarnAppClientFactory}

          class will decide to return instance of this class for
          + * Apache Hadoop 2.1 and beyond.
          — End diff –

          ditto

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/38#discussion_r106763336 — Diff: twill-yarn/src/main/hadoop23/org/apache/twill/internal/yarn/Hadoop23YarnAppClient.java — @@ -0,0 +1,111 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.yarn; + +import com.google.common.base.Joiner; +import com.google.common.base.Throwables; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.io.Text; +import org.apache.hadoop.security.Credentials; +import org.apache.hadoop.security.SecurityUtil; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.hadoop.security.token.Token; +import org.apache.hadoop.security.token.TokenIdentifier; +import org.apache.hadoop.yarn.api.records.ApplicationId; +import org.apache.hadoop.yarn.api.records.ContainerLaunchContext; +import org.apache.hadoop.yarn.client.api.YarnClient; +import org.apache.hadoop.yarn.conf.HAUtil; +import org.apache.hadoop.yarn.conf.YarnConfiguration; +import org.apache.hadoop.yarn.util.ConverterUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.net.InetSocketAddress; +import java.util.ArrayList; +import java.util.List; + +/** + * <p> + * The service implementation of {@link YarnAppClient} for Apache Hadoop 2.1 and beyond. + * + * The {@link VersionDetectYarnAppClientFactory} class will decide to return instance of this class for + * Apache Hadoop 2.1 and beyond. — End diff – ditto
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on the issue:

          https://github.com/apache/twill/pull/38

          @anew Thanks for the review. Addressed comments, please have another look.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on the issue: https://github.com/apache/twill/pull/38 @anew Thanks for the review. Addressed comments, please have another look.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/twill/pull/38

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/38

            People

            • Assignee:
              chtyim Terence Yim
              Reporter:
              ashau Albert Shau
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development