Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1091

STELLAR Shell: Stand Alone installation

    Details

    • Type: New Feature
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:

      Description

      We should have a way that the stellar shell can be installed and run easily on any machine that supports java.

      This may just be a new UBER Jar, or something else.

      It may also be useful to have this app start up and initialize a zookeeper instance and other things for a testing or simulated environment.

        Issue Links

          Activity

          Hide
          ottobackwards Otto Fowler added a comment -

          Initial PR will just be stellar core. My hope is that through review we can discuss brining in other modules

          Show
          ottobackwards Otto Fowler added a comment - Initial PR will just be stellar core. My hope is that through review we can discuss brining in other modules
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ottobackwards opened a pull request:

          https://github.com/apache/metron/pull/690

          METRON-1091 Package STELLAR shell as stand alone

          This PR adds metron-deployment/packaging/archive, and the metron-stellar-shell module.
          This module packages a stellar shell environment that allows running stellar on a workstation that is not a dev/build machine, nor is a cluster node.

          The idea is that a metron 'admin' may want to run stellar commands on his local machine.

          Remote management ( config_get etc ) and profile would be useful, but have not been included. I do not believe it is realistic for the expected user to open firewall ports or have the 'confs' setup to have these functions work as they would on a metron cluster node.

          This use case would seem to require REST support of some kind. Maybe a /metron-interface/metron-rest-stellar module. Or maybe when extensions come to stellar we can just do it as an extension.

            1. Testing
          • package
          • take /metron-deployment/packaging/archive/metron-stellar-shell/target/metron-stellar-shell-0.4.1-archive.tar.gz and unpack it in some directory
          • run it
          • `stellar it up
            1. Questions for reviewers
          • Where should the doc for this go?
          • What NOTICE or other Files should be in the deployed env?
          • Do we want to post this as a separate download?
          • Above questions wrt port opening and configuration
            1. Pull Request Checklist

          Thank you for submitting a contribution to Apache Metron.
          Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
          Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.

          In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

              1. For all changes:
              1. For code changes:
          • [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
          • [x] Have you included steps or a guide to how the change may be verified and tested manually?
          • [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
            ```
            mvn -q clean integration-test install && build_utils/verify_licenses.sh
            ```
          • [ ] Have you written or updated unit tests and or integration tests to verify your changes?
          • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
              1. For documentation related changes:
          • [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:

          ```
          cd site-book
          mvn site
          ```

                1. Note:
                  Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
                  It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ottobackwards/metron stellar_stand_alone

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/metron/pull/690.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #690


          commit eab62efeed7188794e5054b2af2ab256a3b6d6fc
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-09T19:25:33Z

          Initial pass at packaging stellar shell for stand alone deployment.

          New archive area under packaging for things packaged as archives
          New stellar script that doesn't rely on having bigtop installed

          commit 077157d39fcbed753428e76aece2b4a03b26cec6
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-11T13:44:39Z

          bash fu so that script will run from anywhere


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ottobackwards opened a pull request: https://github.com/apache/metron/pull/690 METRON-1091 Package STELLAR shell as stand alone This PR adds metron-deployment/packaging/archive, and the metron-stellar-shell module. This module packages a stellar shell environment that allows running stellar on a workstation that is not a dev/build machine, nor is a cluster node. The idea is that a metron 'admin' may want to run stellar commands on his local machine. Remote management ( config_get etc ) and profile would be useful, but have not been included. I do not believe it is realistic for the expected user to open firewall ports or have the 'confs' setup to have these functions work as they would on a metron cluster node. This use case would seem to require REST support of some kind. Maybe a /metron-interface/metron-rest-stellar module. Or maybe when extensions come to stellar we can just do it as an extension. Testing package take /metron-deployment/packaging/archive/metron-stellar-shell/target/metron-stellar-shell-0.4.1-archive.tar.gz and unpack it in some directory run it `stellar it up Questions for reviewers Where should the doc for this go? What NOTICE or other Files should be in the deployed env? Do we want to post this as a separate download? Above questions wrt port opening and configuration Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines] ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235 ) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines] ( https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview ) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: For all changes: [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira] ( https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel ). [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [x] Has your PR been rebased against the latest commit within the target branch (typically master)? For code changes: [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? [x] Have you included steps or a guide to how the change may be verified and tested manually? [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` [ ] Have you written or updated unit tests and or integration tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? For documentation related changes: [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci] ( https://travis-ci.org ) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_stand_alone Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/690.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #690 commit eab62efeed7188794e5054b2af2ab256a3b6d6fc Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-09T19:25:33Z Initial pass at packaging stellar shell for stand alone deployment. New archive area under packaging for things packaged as archives New stellar script that doesn't rely on having bigtop installed commit 077157d39fcbed753428e76aece2b4a03b26cec6 Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-11T13:44:39Z bash fu so that script will run from anywhere
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user JonZeolla commented on the issue:

          https://github.com/apache/metron/pull/690

          +1, visual inspection + a bit of playing around (see below).

          ```
          ~/bin$ ./stellar
          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: Defaulting to no-operation (NOP) logger implementation
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          log4j:WARN No appenders could be found for logger (org.apache.metron.stellar.beanutils.converters.BooleanConverter).
          log4j:WARN Please initialize the log4j system properly.
          log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
          Stellar, Go!
          Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
          [Stellar]>>> Functions loaded, you may refer to functions now...

          [Stellar]>>> TO_DOUBLE(2)
          2.0
          [Stellar]>>> TO_INTEGER(SQRT(64))
          8
          [Stellar]>>> TO_UPPER('ThIS is A TEsT of MetRoN-1091')
          THIS IS A TEST OF METRON-1091
          [Stellar]>>> CHOP('2093j0f29k30fk230f2j309jf')
          2093j0f29k30fk230f2j309j
          [Stellar]>>> COUNT_MATCHES('the quick brown fox jumped over the lazy dog', 'o')
          4
          [Stellar]>>> DAY_OF_YEAR(1502592512)
          18
          [Stellar]>>> DAY_OF_MONTH(1502592512)
          18
          ~$ bin/stellar
          SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
          SLF4J: Defaulting to no-operation (NOP) logger implementation
          SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
          log4j:WARN No appenders could be found for logger (org.apache.metron.stellar.beanutils.converters.BooleanConverter).
          log4j:WARN Please initialize the log4j system properly.
          log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
          Stellar, Go!
          Please note that functions are loading lazily in the background and will be unavailable until loaded fully.
          [Stellar]>>> Functions loaded, you may refer to functions now...

          [Stellar]>>> DAY_OF_YEAR(1502592893)
          18
          [Stellar]>>> DAY_OF_MONTH(1502592893)
          18
          [Stellar]>>> COUNT_MATCHES('the quick brown fox jumped over the lazy dog', 'o')
          4
          ```

          Opened METRON-1107(https://issues.apache.org/jira/browse/METRON-1107) for the DAY_OF_

          {YEAR,MONTH}

          issue, otherwise looks good to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/690 +1, visual inspection + a bit of playing around (see below). ``` ~/bin$ ./stellar SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No appenders could be found for logger (org.apache.metron.stellar.beanutils.converters.BooleanConverter). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Stellar, Go! Please note that functions are loading lazily in the background and will be unavailable until loaded fully. [Stellar] >>> Functions loaded, you may refer to functions now... [Stellar] >>> TO_DOUBLE(2) 2.0 [Stellar] >>> TO_INTEGER(SQRT(64)) 8 [Stellar] >>> TO_UPPER('ThIS is A TEsT of MetRoN-1091') THIS IS A TEST OF METRON-1091 [Stellar] >>> CHOP('2093j0f29k30fk230f2j309jf') 2093j0f29k30fk230f2j309j [Stellar] >>> COUNT_MATCHES('the quick brown fox jumped over the lazy dog', 'o') 4 [Stellar] >>> DAY_OF_YEAR(1502592512) 18 [Stellar] >>> DAY_OF_MONTH(1502592512) 18 ~$ bin/stellar SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder". SLF4J: Defaulting to no-operation (NOP) logger implementation SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details. log4j:WARN No appenders could be found for logger (org.apache.metron.stellar.beanutils.converters.BooleanConverter). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. Stellar, Go! Please note that functions are loading lazily in the background and will be unavailable until loaded fully. [Stellar] >>> Functions loaded, you may refer to functions now... [Stellar] >>> DAY_OF_YEAR(1502592893) 18 [Stellar] >>> DAY_OF_MONTH(1502592893) 18 [Stellar] >>> COUNT_MATCHES('the quick brown fox jumped over the lazy dog', 'o') 4 ``` Opened METRON-1107 ( https://issues.apache.org/jira/browse/METRON-1107 ) for the DAY_OF_ {YEAR,MONTH} issue, otherwise looks good to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          @JonZeolla Thanks for the review!
          There are several questions in the description that I think need to fall out of the review, one of them is where or how to document this.

          Any ideas?

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 @JonZeolla Thanks for the review! There are several questions in the description that I think need to fall out of the review, one of them is where or how to document this. Any ideas?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user JonZeolla commented on the issue:

          https://github.com/apache/metron/pull/690

          Some ideas:

          Show
          githubbot ASF GitHub Bot added a comment - Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/690 Some ideas: Where should the doc for this go? I would suggest under [here] ( https://github.com/apache/metron/tree/master/metron-stellar/stellar-common#stellar-shell ). Do we want to post this as a separate download? Probably not IMO, this is a bit more of a niche tool. Above questions wrt port opening and configuration Probably handle as a follow-on, if at all.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/metron/pull/690

          Are we packaging up the RPMs correctly to ship the stellar shell after this work has been done? Have we tried this in full-dev?

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/metron/pull/690 Are we packaging up the RPMs correctly to ship the stellar shell after this work has been done? Have we tried this in full-dev?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          Casey, this adds a new packaging subfolder -> archive, and generates a new artifact ( the tar.gz ).
          It does not effect the current deployment of the stellar shell as a part of metron.

          I put it in deployment / packaging as default for 2 reasons:

          1. If we do include profiler/management etc that makes this a metron thing and not a stellar thing
          2. We don't have any packaging outside of deployment

          Both of those things are up for discussion.

          an RPM of the shell for RPM deployment to NON metron nodes could be a follow on.

          *Comments from [Reviewable](https://reviewable.io:443/reviews/apache/metron/690#-:-KrW6vGRzz8e1jzn5S1b:b-6c7w3h)*
          <!-- Sent from Reviewable.io -->

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 Casey, this adds a new packaging subfolder -> archive, and generates a new artifact ( the tar.gz ). It does not effect the current deployment of the stellar shell as a part of metron. I put it in deployment / packaging as default for 2 reasons: 1. If we do include profiler/management etc that makes this a metron thing and not a stellar thing 2. We don't have any packaging outside of deployment Both of those things are up for discussion. an RPM of the shell for RPM deployment to NON metron nodes could be a follow on. — *Comments from [Reviewable] ( https://reviewable.io:443/reviews/apache/metron/690#-:-KrW6vGRzz8e1jzn5S1b:b-6c7w3h)* <!-- Sent from Reviewable.io -->
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user cestella commented on the issue:

          https://github.com/apache/metron/pull/690

          I just want to make sure that we didn't break backwards compatibility and have the shell not deployed as part of the mpack, I guess

          Show
          githubbot ASF GitHub Bot added a comment - Github user cestella commented on the issue: https://github.com/apache/metron/pull/690 I just want to make sure that we didn't break backwards compatibility and have the shell not deployed as part of the mpack, I guess
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user mattf-horton commented on a diff in the pull request:

          https://github.com/apache/metron/pull/690#discussion_r133076804

          — Diff: metron-deployment/packaging/archive/metron-stellar-shell/pom.xml —
          @@ -0,0 +1,94 @@
          +<?xml version="1.0" encoding="UTF-8"?>
          +<!--
          + Licensed to the Apache Software
          + Foundation (ASF) under one or more contributor license agreements. See the
          + NOTICE file distributed with this work for additional information regarding
          + copyright ownership. The ASF licenses this file to You under the Apache License,
          + Version 2.0 (the "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
          + Unless required by applicable law or agreed to in writing, software distributed
          + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
          + OR CONDITIONS OF ANY KIND, either express or implied. See the License for
          + the specific language governing permissions and limitations under the License.
          + -->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.metron</groupId>
          + <artifactId>metron-deployment</artifactId>
          + <version>0.4.1</version>
          + <relativePath>../../..</relativePath>
          + </parent>
          + <artifactId>metron-stellar-shell</artifactId>
          + <name>metron-stellar-shell-archive</name>
          + <description>Archive of stellar shell and libraries for stand alone running</description>
          + <url>https://metron.apache.org/</url>
          + <properties>
          + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
          + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
          + <commons.config.version>1.10</commons.config.version>
          + <metron_dir>$

          {project.parent.relativePath}

          /..</metron_dir>
          + </properties>
          — End diff –

          So I'm not a build-and-deployment expert, but shouldn't this pom have a dependency on stellar-common? Perhaps, with or without the exclusions:
          ```xml
          <dependency>
          <groupId>org.apache.metron</groupId>
          <artifactId>stellar-common</artifactId>
          <classifier>uber</classifier>
          <version>$

          {project.parent.version}

          </version>
          <exclusions>
          <exclusion>
          <groupId>log4j</groupId>
          <artifactId>log4j</artifactId>
          </exclusion>
          <exclusion>
          <groupId>org.slf4j</groupId>
          <artifactId>slf4j-log4j12</artifactId>
          </exclusion>
          </exclusions>
          </dependency>
          ```

          The copy-resources could, if desired, then be replaced by usage in the assembly.xml of a dependencySet reference.

          Show
          githubbot ASF GitHub Bot added a comment - Github user mattf-horton commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r133076804 — Diff: metron-deployment/packaging/archive/metron-stellar-shell/pom.xml — @@ -0,0 +1,94 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software + Foundation (ASF) under one or more contributor license agreements. See the + NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to You under the Apache License, + Version 2.0 (the "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES + OR CONDITIONS OF ANY KIND, either express or implied. See the License for + the specific language governing permissions and limitations under the License. + --> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.metron</groupId> + <artifactId>metron-deployment</artifactId> + <version>0.4.1</version> + <relativePath>../../..</relativePath> + </parent> + <artifactId>metron-stellar-shell</artifactId> + <name>metron-stellar-shell-archive</name> + <description>Archive of stellar shell and libraries for stand alone running</description> + <url> https://metron.apache.org/ </url> + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> + <commons.config.version>1.10</commons.config.version> + <metron_dir>$ {project.parent.relativePath} /..</metron_dir> + </properties> — End diff – So I'm not a build-and-deployment expert, but shouldn't this pom have a dependency on stellar-common? Perhaps, with or without the exclusions: ```xml <dependency> <groupId>org.apache.metron</groupId> <artifactId>stellar-common</artifactId> <classifier>uber</classifier> <version>$ {project.parent.version} </version> <exclusions> <exclusion> <groupId>log4j</groupId> <artifactId>log4j</artifactId> </exclusion> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> ``` The copy-resources could, if desired, then be replaced by usage in the assembly.xml of a dependencySet reference.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on a diff in the pull request:

          https://github.com/apache/metron/pull/690#discussion_r133081100

          — Diff: metron-deployment/packaging/archive/metron-stellar-shell/pom.xml —
          @@ -0,0 +1,94 @@
          +<?xml version="1.0" encoding="UTF-8"?>
          +<!--
          + Licensed to the Apache Software
          + Foundation (ASF) under one or more contributor license agreements. See the
          + NOTICE file distributed with this work for additional information regarding
          + copyright ownership. The ASF licenses this file to You under the Apache License,
          + Version 2.0 (the "License"); you may not use this file except in compliance
          + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
          + Unless required by applicable law or agreed to in writing, software distributed
          + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
          + OR CONDITIONS OF ANY KIND, either express or implied. See the License for
          + the specific language governing permissions and limitations under the License.
          + -->
          +
          +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
          + <modelVersion>4.0.0</modelVersion>
          + <parent>
          + <groupId>org.apache.metron</groupId>
          + <artifactId>metron-deployment</artifactId>
          + <version>0.4.1</version>
          + <relativePath>../../..</relativePath>
          + </parent>
          + <artifactId>metron-stellar-shell</artifactId>
          + <name>metron-stellar-shell-archive</name>
          + <description>Archive of stellar shell and libraries for stand alone running</description>
          + <url>https://metron.apache.org/</url>
          + <properties>
          + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
          + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
          + <commons.config.version>1.10</commons.config.version>
          + <metron_dir>$

          {project.parent.relativePath}

          /..</metron_dir>
          + </properties>
          — End diff –

          I'm not an expert either. I copied what that docker-rpm pom does, because it does similar resource things.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on a diff in the pull request: https://github.com/apache/metron/pull/690#discussion_r133081100 — Diff: metron-deployment/packaging/archive/metron-stellar-shell/pom.xml — @@ -0,0 +1,94 @@ +<?xml version="1.0" encoding="UTF-8"?> +<!-- + Licensed to the Apache Software + Foundation (ASF) under one or more contributor license agreements. See the + NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to You under the Apache License, + Version 2.0 (the "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES + OR CONDITIONS OF ANY KIND, either express or implied. See the License for + the specific language governing permissions and limitations under the License. + --> + +<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd "> + <modelVersion>4.0.0</modelVersion> + <parent> + <groupId>org.apache.metron</groupId> + <artifactId>metron-deployment</artifactId> + <version>0.4.1</version> + <relativePath>../../..</relativePath> + </parent> + <artifactId>metron-stellar-shell</artifactId> + <name>metron-stellar-shell-archive</name> + <description>Archive of stellar shell and libraries for stand alone running</description> + <url> https://metron.apache.org/ </url> + <properties> + <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> + <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> + <commons.config.version>1.10</commons.config.version> + <metron_dir>$ {project.parent.relativePath} /..</metron_dir> + </properties> — End diff – I'm not an expert either. I copied what that docker-rpm pom does, because it does similar resource things.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          bump - any comments on this? Any ideas for deployment? can we leave for later?

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 bump - any comments on this? Any ideas for deployment? can we leave for later?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          @JonZeolla I don't want to document it in the stellar-common module.
          Stellar is being broken out of metron. The packaging of stellar as a shell, with the likely inclusion of metron-* stellar functions is a 'metron' thing. not a stellar thing.

          At least that is what I'm thinking. So this is a metron custom deployment of the shell. I could change it to the default stand alone deployment of the shell for stellar, but I didn't think that was the right way to start out, at the time at least.

          We *could* do it as the default archive of stellar common, and then do *another* archive of metron stellar jars.

          So the usage would be:

          unpack shell package
          unpack metron-stellar-package into lib

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 @JonZeolla I don't want to document it in the stellar-common module. Stellar is being broken out of metron. The packaging of stellar as a shell, with the likely inclusion of metron-* stellar functions is a 'metron' thing. not a stellar thing. At least that is what I'm thinking. So this is a metron custom deployment of the shell. I could change it to the default stand alone deployment of the shell for stellar, but I didn't think that was the right way to start out, at the time at least. We * could * do it as the default archive of stellar common, and then do * another * archive of metron stellar jars. So the usage would be: unpack shell package unpack metron-stellar-package into lib
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          I'm going to close this and refactor

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 I'm going to close this and refactor
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards closed the pull request at:

          https://github.com/apache/metron/pull/690

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards closed the pull request at: https://github.com/apache/metron/pull/690
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user JonZeolla commented on the issue:

          https://github.com/apache/metron/pull/690

          Right, I think the disconnect is that I would envision a stellar shell with metron stellar functions added as libs, as you suggested. That's why I was thinking we would doc in stellar-common, I missed that this includes management/profiler functions.

          Show
          githubbot ASF GitHub Bot added a comment - Github user JonZeolla commented on the issue: https://github.com/apache/metron/pull/690 Right, I think the disconnect is that I would envision a stellar shell with metron stellar functions added as libs, as you suggested. That's why I was thinking we would doc in stellar-common, I missed that this includes management/profiler functions.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user ottobackwards reopened a pull request:

          https://github.com/apache/metron/pull/690

          METRON-1091 Package STELLAR shell as stand alone

          This PR adds metron-deployment/packaging/archive, and the metron-stellar-shell module.
          This module packages a stellar shell environment that allows running stellar on a workstation that is not a dev/build machine, nor is a cluster node.

          The idea is that a metron 'admin' may want to run stellar commands on his local machine.

          Remote management ( config_get etc ) and profile would be useful, but have not been included. I do not believe it is realistic for the expected user to open firewall ports or have the 'confs' setup to have these functions work as they would on a metron cluster node.

          This use case would seem to require REST support of some kind. Maybe a /metron-interface/metron-rest-stellar module. Or maybe when extensions come to stellar we can just do it as an extension.

          Also a problem, is that there are management functions that bring in hadoop dependencies mixed in with other functions like SHELL_EDIT() that may be useful on their own.

            1. Testing
          • package
          • take /metron-deployment/packaging/archive/metron-stellar-shell/target/metron-stellar-shell-0.4.1-archive.tar.gz and unpack it in some directory
          • run it
          • `stellar it up
            1. Questions for reviewers
          • Where should the doc for this go?
          • What NOTICE or other Files should be in the deployed env?
          • Do we want to post this as a separate download?
          • Above questions wrt port opening and configuration
            1. Pull Request Checklist

          Thank you for submitting a contribution to Apache Metron.
          Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions.
          Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides.

          In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following:

              1. For all changes:
              1. For code changes:
          • [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
          • [x] Have you included steps or a guide to how the change may be verified and tested manually?
          • [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
            ```
            mvn -q clean integration-test install && build_utils/verify_licenses.sh
            ```
          • [ ] Have you written or updated unit tests and or integration tests to verify your changes?
          • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?
              1. For documentation related changes:
          • [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`:

          ```
          cd site-book
          mvn site
          ```

                1. Note:
                  Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible.
                  It is also recommended that [travis-ci](https://travis-ci.org) is set up for your personal repository such that your branches are built there before submitting a pull request.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/ottobackwards/metron stellar_stand_alone

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/metron/pull/690.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #690


          commit 81c0badc31353813e0d884c860c0c854b2026709
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-09T19:25:33Z

          Initial pass at packaging stellar shell for stand alone deployment.

          New archive area under packaging for things packaged as archives
          New stellar script that doesn't rely on having bigtop installed

          commit a6acc137d9fb31baeb8c2e7d9d4076bde6b098ba
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-11T18:14:58Z

          Merge branch 'master' of https://github.com/apache/metron into stellar_stand_alone

          commit c315e1ce52c6dc44a6eb688b4387f7a89a1cfc97
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-31T19:44:20Z

          refactor to stellar centric deployment

          commit bde23812615a3c24b034d2b48b3127a8f5b5b900
          Author: Otto Fowler <ottobackwards@gmail.com>
          Date: 2017-08-31T19:44:24Z

          Merge branch 'master' of https://github.com/apache/metron into stellar_stand_alone


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user ottobackwards reopened a pull request: https://github.com/apache/metron/pull/690 METRON-1091 Package STELLAR shell as stand alone This PR adds metron-deployment/packaging/archive, and the metron-stellar-shell module. This module packages a stellar shell environment that allows running stellar on a workstation that is not a dev/build machine, nor is a cluster node. The idea is that a metron 'admin' may want to run stellar commands on his local machine. Remote management ( config_get etc ) and profile would be useful, but have not been included. I do not believe it is realistic for the expected user to open firewall ports or have the 'confs' setup to have these functions work as they would on a metron cluster node. This use case would seem to require REST support of some kind. Maybe a /metron-interface/metron-rest-stellar module. Or maybe when extensions come to stellar we can just do it as an extension. Also a problem, is that there are management functions that bring in hadoop dependencies mixed in with other functions like SHELL_EDIT() that may be useful on their own. Testing package take /metron-deployment/packaging/archive/metron-stellar-shell/target/metron-stellar-shell-0.4.1-archive.tar.gz and unpack it in some directory run it `stellar it up Questions for reviewers Where should the doc for this go? What NOTICE or other Files should be in the deployed env? Do we want to post this as a separate download? Above questions wrt port opening and configuration Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines] ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235 ) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines] ( https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview ) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: For all changes: [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira] ( https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel ). [x] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [x] Has your PR been rebased against the latest commit within the target branch (typically master)? For code changes: [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? [x] Have you included steps or a guide to how the change may be verified and tested manually? [x] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: ``` mvn -q clean integration-test install && build_utils/verify_licenses.sh ``` [ ] Have you written or updated unit tests and or integration tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? For documentation related changes: [ ] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` Note: Please ensure that once the PR is submitted, you check travis-ci for build issues and submit an update to your PR as soon as possible. It is also recommended that [travis-ci] ( https://travis-ci.org ) is set up for your personal repository such that your branches are built there before submitting a pull request. You can merge this pull request into a Git repository by running: $ git pull https://github.com/ottobackwards/metron stellar_stand_alone Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/690.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #690 commit 81c0badc31353813e0d884c860c0c854b2026709 Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-09T19:25:33Z Initial pass at packaging stellar shell for stand alone deployment. New archive area under packaging for things packaged as archives New stellar script that doesn't rely on having bigtop installed commit a6acc137d9fb31baeb8c2e7d9d4076bde6b098ba Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-11T18:14:58Z Merge branch 'master' of https://github.com/apache/metron into stellar_stand_alone commit c315e1ce52c6dc44a6eb688b4387f7a89a1cfc97 Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-31T19:44:20Z refactor to stellar centric deployment commit bde23812615a3c24b034d2b48b3127a8f5b5b900 Author: Otto Fowler <ottobackwards@gmail.com> Date: 2017-08-31T19:44:24Z Merge branch 'master' of https://github.com/apache/metron into stellar_stand_alone
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user ottobackwards commented on the issue:

          https://github.com/apache/metron/pull/690

          I have refactored this to be stellar centric.
          We will create a second archive for the stand alone shell from stellar-common.
          I also documented it in the stellar README.md.

          The questions about NOTICE, LIC, README for deployment still apply.

          Show
          githubbot ASF GitHub Bot added a comment - Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/690 I have refactored this to be stellar centric. We will create a second archive for the stand alone shell from stellar-common. I also documented it in the stellar README.md. The questions about NOTICE, LIC, README for deployment still apply.

            People

            • Assignee:
              ottobackwards Otto Fowler
              Reporter:
              ottobackwards Otto Fowler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development