Uploaded image for project: 'Samza'
  1. Samza
  2. SAMZA-1143

Universal config support for localized resource

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Currently, every time we add a new resource for localization, we need to modify the Samza code, as the preparation for the localResource is coded for each specific localizer, such as "yarn.package.path" for localizing the package.

      But sometimes, we need to get different resources for localization, one use case is to get the certificate for the samza app to communicate with other services. We do not know details until we use them, so it is better to keep the context for that in the configuration so that the users who are using them can control these resource localization.

      A common configuration for localizer resource would be like the following:
      key: yarn.localizer.resource.<resourceVisibility>.<resourceType>.<resourceName>
      e.g. localizer.resource.APPLICATION.FILE.identity
      where the valid resourceVisibility is a string format of LocalResourceVisibility(enum):
      PUBLIC, PRIVATE, or APPLICATION;
      and the valid resourceType is a string format of LocalResourceType (enum):
      ARCHIVE, FILE, PATTERN;
      e.g. localizer.resource.APPLICATION.FILE.identity
      the valid resourceName could be any non empty String.
      value: <resourcePath>
      where resourcePath is a string format of the URL, e.g. http://hostname.com/test, hdfs://host.com/readme, file:///temp/readme

        Issue Links

          Activity

          Hide
          navina Navina Ramesh added a comment -

          Fred Ji Can you perhaps explain what you mean by "resourceVisibility" and "resourceType" ? How will they be used?

          Show
          navina Navina Ramesh added a comment - Fred Ji Can you perhaps explain what you mean by "resourceVisibility" and "resourceType" ? How will they be used?
          Hide
          fredji Fred Ji added a comment -

          Thanks Navina Ramesh, I updated the description to answer your question.

          Show
          fredji Fred Ji added a comment - Thanks Navina Ramesh , I updated the description to answer your question.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user fredji97 opened a pull request:

          https://github.com/apache/samza/pull/90

          SAMZA-1143 Universal config support for localized resource

          More details in https://issues.apache.org/jira/browse/SAMZA-1143

          Tests: ./gradlew clean check successful and all unit tests passed

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/fredji97/samza universalLocalizer

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/samza/pull/90.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #90


          commit 8f87cacf87c0e9a3944a46e8557293d4543b37e0
          Author: Fred Ji <fji@linkedin.com>
          Date: 2017-03-16T22:56:03Z

          SAMZA-1143 Universal config support for localized resource


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user fredji97 opened a pull request: https://github.com/apache/samza/pull/90 SAMZA-1143 Universal config support for localized resource More details in https://issues.apache.org/jira/browse/SAMZA-1143 Tests: ./gradlew clean check successful and all unit tests passed You can merge this pull request into a Git repository by running: $ git pull https://github.com/fredji97/samza universalLocalizer Alternatively you can review and apply these changes as the patch at: https://github.com/apache/samza/pull/90.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #90 commit 8f87cacf87c0e9a3944a46e8557293d4543b37e0 Author: Fred Ji <fji@linkedin.com> Date: 2017-03-16T22:56:03Z SAMZA-1143 Universal config support for localized resource
          Hide
          navina Navina Ramesh added a comment -

          > Thanks Navina Ramesh, I updated the description to answer your question.
          Perfect! Thanks for explaining

          Show
          navina Navina Ramesh added a comment - > Thanks Navina Ramesh, I updated the description to answer your question. Perfect! Thanks for explaining
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/samza/pull/90

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/samza/pull/90
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user fredji97 opened a pull request:

          https://github.com/apache/samza/pull/97

          SAMZA-1143 Include fs.<scheme>.impl.* subkeys to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager

          SAMZA-1143 Include fs.<scheme>.impl.* subkeys, in addition to fs.<scheme>.impl, to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager.

          When there are additional subconfigurations under fs.myScheme.impl, such as fs.myScheme.impl.client, we need to keep the set of configuration completed in YarnJobFactory and YarnClusterResourceManager. When the context is set for localizing the resource in ClientHelper and YarnContainerRunner, it may rely on this configuration to get the FileStatus information, which may or may not depends on fs.<scheme>.impl and all possible fs.<scheme>.impl.* sub-configuration.

          This is an enhanced feature to the PR#90.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/fredji97/samza fsImplSubkeys

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/samza/pull/97.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #97


          commit 988d43573656c7d795b251c824c7fad3c6f5d9f3
          Author: Fred Ji <fji@linkedin.com>
          Date: 2017-03-27T22:19:26Z

          SAMZA-1143 Copy fs.<scheme>.impl.* subkeys, in addition to fs.<scheme>.impl, to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager. When there are additional subconfigurations under fs.<scheme>.impl, such as fs.<scheme>.impl.client, we need to keep the set of configuration completed.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user fredji97 opened a pull request: https://github.com/apache/samza/pull/97 SAMZA-1143 Include fs.<scheme>.impl.* subkeys to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager SAMZA-1143 Include fs.<scheme>.impl.* subkeys, in addition to fs.<scheme>.impl, to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager. When there are additional subconfigurations under fs.myScheme.impl, such as fs.myScheme.impl.client, we need to keep the set of configuration completed in YarnJobFactory and YarnClusterResourceManager. When the context is set for localizing the resource in ClientHelper and YarnContainerRunner, it may rely on this configuration to get the FileStatus information, which may or may not depends on fs.<scheme>.impl and all possible fs.<scheme>.impl.* sub-configuration. This is an enhanced feature to the PR#90. You can merge this pull request into a Git repository by running: $ git pull https://github.com/fredji97/samza fsImplSubkeys Alternatively you can review and apply these changes as the patch at: https://github.com/apache/samza/pull/97.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #97 commit 988d43573656c7d795b251c824c7fad3c6f5d9f3 Author: Fred Ji <fji@linkedin.com> Date: 2017-03-27T22:19:26Z SAMZA-1143 Copy fs.<scheme>.impl.* subkeys, in addition to fs.<scheme>.impl, to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager. When there are additional subconfigurations under fs.<scheme>.impl, such as fs.<scheme>.impl.client, we need to keep the set of configuration completed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/samza/pull/97

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/samza/pull/97

            People

            • Assignee:
              fredji Fred Ji
              Reporter:
              fredji Fred Ji
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development