Uploaded image for project: 'Apache S4'
  1. Apache S4
  2. S4-25

Write S4 Application Master to deploy S4 in Yarn

    Details

    • Type: New Feature
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 0.7
    • Labels:
      None

      Description

      On the lines of s4PigWrapper, write a s4 application master to host s4 piper inside Hadoop Yarn. This could be useful not only for reading data stored in hadoop ( to build or train a model)... But we could make use of the resource manager to deploy s4 instances in remote machine and monitor them. In short, we could make use of most of the resource management , scheduling and other good stuff in Yarn.

      • Yarn is useful to deploy and launch s4 instances.
      • It still requires deploying node managers on each box which means it will
        be useful if one is running more than one s4 process on a node.

        Attachments

        1. S4-YARN-1.patch
          74 kB
          Sean Mackrory
        2. S4-Constants.diff
          2 kB
          Sean Mackrory
        3. S4-Client.diff
          13 kB
          Sean Mackrory
        4. S4-ApplicationMaster.diff
          13 kB
          Sean Mackrory

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              zahoor J Mohamed Zahoor
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated: