Uploaded image for project: 'Metron'
  1. Metron
  2. METRON-1031

Management UI Cannot Start Topologies in Kerberized Environment

    Details

    • Type: Bug
    • Status: To Do
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Labels:
      None

      Description

      The Metron Management UI cannot start topologies in a kerberized environment. This includes the parser, indexing, and enrichment topologies.

      It seems that Metron REST does not pass "-ksp" to the start topology commands. As a result, topologies that are started with the Management UI cannot start a producer against a kerberized Kafka.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user nickwallen opened a pull request:

          https://github.com/apache/metron/pull/647

          METRON-1031: Management UI Cannot Start Topologies in Kerberized Environment

          The Metron Management UI cannot start topologies in a kerberized environment. This includes the Parser, Enrichment, and Indexing topologies.

          It seems that Metron REST does not pass "-ksp" to the start topology commands. As a result, topologies that are started with the Management UI cannot start a producer against a kerberized Kafka.

              1. Changes
          • Altered the `StormCLIWrapper` to add logging of the actual CLI commands that are executed.
          • The `-ksp` argument is always used (with Kerberos or not) when launching the Parser, Enrichment, and Indexing topologies.
          • Added a Spring property `kafka.security.protocol` that serves as the `-ksp` argument when launching the topologies.
          • Ambari sets this Spring property based on Kafka's own configuration in Ambari.
              1. Testing

          To manually test this, validate that all of the topologies can be stopped and started using the Management UI and Swagger UI both before and after kerberization.

          1. Launch the Full Dev environment.
          1. Stop then start the Parser topologies using the Management UI.
          1. Stop then start the Enrichment and Indexing topologies using the Swagger UI.
          1. Kerberize the Full Dev environment.
          1. Stop then start the Parser topologies using the Management UI.
          1. Stop then start the Enrichment and Indexing topologies using the Swagger UI.

              1. Pull Request Checklist
          • [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
          • [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
          • [ ] Has your PR been rebased against the latest commit within the target branch (typically master)?
          • [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed?
          • [ ] Have you included steps or a guide to how the change may be verified and tested manually?
          • [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via:
          • [ ] Have you written or updated unit tests and or integration tests to verify your changes?
          • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)?
          • [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent?

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/nickwallen/metron METRON-1031

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/metron/pull/647.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #647


          commit 6cff444a1120ca3577776aa4e39f54981f2d43b5
          Author: Nick Allen <nick@nickallen.org>
          Date: 2017-07-11T23:05:26Z

          METRON-1031: Management UI Cannot Start Topologies in Kerberized Environment

          commit a70ad836f9a08e58dc97c0cfc4d5dcaabf5c818b
          Author: Nick Allen <nick@nickallen.org>
          Date: 2017-07-12T13:52:04Z

          Configuration fixup


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user nickwallen opened a pull request: https://github.com/apache/metron/pull/647 METRON-1031 : Management UI Cannot Start Topologies in Kerberized Environment The Metron Management UI cannot start topologies in a kerberized environment. This includes the Parser, Enrichment, and Indexing topologies. It seems that Metron REST does not pass "-ksp" to the start topology commands. As a result, topologies that are started with the Management UI cannot start a producer against a kerberized Kafka. Changes Altered the `StormCLIWrapper` to add logging of the actual CLI commands that are executed. The `-ksp` argument is always used (with Kerberos or not) when launching the Parser, Enrichment, and Indexing topologies. Added a Spring property `kafka.security.protocol` that serves as the `-ksp` argument when launching the topologies. Ambari sets this Spring property based on Kafka's own configuration in Ambari. Testing To manually test this, validate that all of the topologies can be stopped and started using the Management UI and Swagger UI both before and after kerberization. 1. Launch the Full Dev environment. 1. Stop then start the Parser topologies using the Management UI. 1. Stop then start the Enrichment and Indexing topologies using the Swagger UI. 1. Kerberize the Full Dev environment. 1. Stop then start the Parser topologies using the Management UI. 1. Stop then start the Enrichment and Indexing topologies using the Swagger UI. Pull Request Checklist [ ] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira] ( https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel ). [ ] Does your PR title start with METRON-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. [ ] Has your PR been rebased against the latest commit within the target branch (typically master)? [ ] Have you included steps to reproduce the behavior or problem that is being changed or addressed? [ ] Have you included steps or a guide to how the change may be verified and tested manually? [ ] Have you ensured that the full suite of tests and checks have been executed in the root metron folder via: [ ] Have you written or updated unit tests and or integration tests to verify your changes? [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0] ( http://www.apache.org/legal/resolved.html#category-a)? [ ] Have you verified the basic functionality of the build by building and running locally with Vagrant full-dev environment or the equivalent? You can merge this pull request into a Git repository by running: $ git pull https://github.com/nickwallen/metron METRON-1031 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/metron/pull/647.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #647 commit 6cff444a1120ca3577776aa4e39f54981f2d43b5 Author: Nick Allen <nick@nickallen.org> Date: 2017-07-11T23:05:26Z METRON-1031 : Management UI Cannot Start Topologies in Kerberized Environment commit a70ad836f9a08e58dc97c0cfc4d5dcaabf5c818b Author: Nick Allen <nick@nickallen.org> Date: 2017-07-12T13:52:04Z Configuration fixup
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127466486

          — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java —
          @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException {
          protected int runCommand(String[] command) throws RestException {
          ProcessBuilder pb = getProcessBuilder(command);
          pb.inheritIO();

          • Process process = null;
            + LOG.debug("Running command: cmd={}", String.join(" ", command));
            +
            + Process process;
            try { process = pb.start(); process.waitFor(); + }

            catch (Exception e)

            { throw new RestException(e); }
          • return process.exitValue();
            +
            + int exitValue = process.exitValue();
            + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue);
            +
            + return exitValue;
            }

          protected String[] getParserStartCommand(String name)

          { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; }

          protected String[] getEnrichmentStartCommand() {

          • String[] command = new String[1];
            + String[] command = new String[3];
              • End diff –

          I don't think the -ksp flag is needed for the enrichment or indexing topologies. This setting comes from enrichment.properties and elasticsearch.properties.

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127466486 — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java — @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException { protected int runCommand(String[] command) throws RestException { ProcessBuilder pb = getProcessBuilder(command); pb.inheritIO(); Process process = null; + LOG.debug("Running command: cmd={}", String.join(" ", command)); + + Process process; try { process = pb.start(); process.waitFor(); + } catch (Exception e) { throw new RestException(e); } return process.exitValue(); + + int exitValue = process.exitValue(); + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue); + + return exitValue; } protected String[] getParserStartCommand(String name) { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; } protected String[] getEnrichmentStartCommand() { String[] command = new String [1] ; + String[] command = new String [3] ; End diff – I don't think the -ksp flag is needed for the enrichment or indexing topologies. This setting comes from enrichment.properties and elasticsearch.properties.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickwallen commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127469430

          — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java —
          @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException {
          protected int runCommand(String[] command) throws RestException {
          ProcessBuilder pb = getProcessBuilder(command);
          pb.inheritIO();

          • Process process = null;
            + LOG.debug("Running command: cmd={}", String.join(" ", command));
            +
            + Process process;
            try { process = pb.start(); process.waitFor(); + }

            catch (Exception e)

            { throw new RestException(e); }
          • return process.exitValue();
            +
            + int exitValue = process.exitValue();
            + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue);
            +
            + return exitValue;
            }

          protected String[] getParserStartCommand(String name)

          { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; }

          protected String[] getEnrichmentStartCommand() {

          • String[] command = new String[1];
            + String[] command = new String[3];
              • End diff –

          Makes sense now. I will remove -ksp from both Enrichment and Indexing start commands. Thanks!

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127469430 — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java — @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException { protected int runCommand(String[] command) throws RestException { ProcessBuilder pb = getProcessBuilder(command); pb.inheritIO(); Process process = null; + LOG.debug("Running command: cmd={}", String.join(" ", command)); + + Process process; try { process = pb.start(); process.waitFor(); + } catch (Exception e) { throw new RestException(e); } return process.exitValue(); + + int exitValue = process.exitValue(); + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue); + + return exitValue; } protected String[] getParserStartCommand(String name) { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; } protected String[] getEnrichmentStartCommand() { String[] command = new String [1] ; + String[] command = new String [3] ; End diff – Makes sense now. I will remove -ksp from both Enrichment and Indexing start commands. Thanks!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user simonellistonball commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127472746

          — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java —
          @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException {
          protected int runCommand(String[] command) throws RestException {
          ProcessBuilder pb = getProcessBuilder(command);
          pb.inheritIO();

          • Process process = null;
            + LOG.debug("Running command: cmd={}", String.join(" ", command));
            +
            + Process process;
            try { process = pb.start(); process.waitFor(); + }

            catch (Exception e)

            { throw new RestException(e); }
          • return process.exitValue();
            +
            + int exitValue = process.exitValue();
            + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue);
            +
            + return exitValue;
            }

          protected String[] getParserStartCommand(String name)

          { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; }

          protected String[] getEnrichmentStartCommand() {

          • String[] command = new String[1];
            + String[] command = new String[3];
              • End diff –

          Has this changed since 0.4.0 RC1? Didn't work when I tried it until ksp added in the start script.

          Show
          githubbot ASF GitHub Bot added a comment - Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127472746 — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java — @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException { protected int runCommand(String[] command) throws RestException { ProcessBuilder pb = getProcessBuilder(command); pb.inheritIO(); Process process = null; + LOG.debug("Running command: cmd={}", String.join(" ", command)); + + Process process; try { process = pb.start(); process.waitFor(); + } catch (Exception e) { throw new RestException(e); } return process.exitValue(); + + int exitValue = process.exitValue(); + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue); + + return exitValue; } protected String[] getParserStartCommand(String name) { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; } protected String[] getEnrichmentStartCommand() { String[] command = new String [1] ; + String[] command = new String [3] ; End diff – Has this changed since 0.4.0 RC1? Didn't work when I tried it until ksp added in the start script.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127473184

          — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java —
          @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException {
          protected int runCommand(String[] command) throws RestException {
          ProcessBuilder pb = getProcessBuilder(command);
          pb.inheritIO();

          • Process process = null;
            + LOG.debug("Running command: cmd={}", String.join(" ", command));
            +
            + Process process;
            try { process = pb.start(); process.waitFor(); + }

            catch (Exception e)

            { throw new RestException(e); }
          • return process.exitValue();
            +
            + int exitValue = process.exitValue();
            + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue);
            +
            + return exitValue;
            }

          protected String[] getParserStartCommand(String name)

          { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; }

          protected String[] getEnrichmentStartCommand() {

          • String[] command = new String[1];
            + String[] command = new String[3];
              • End diff –

          No it's always been like that. The parser topology start script is the only one that supports the ksp flag. The enrichment and elasticsearch topology start scripts don't expect any input parameters.

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127473184 — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java — @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException { protected int runCommand(String[] command) throws RestException { ProcessBuilder pb = getProcessBuilder(command); pb.inheritIO(); Process process = null; + LOG.debug("Running command: cmd={}", String.join(" ", command)); + + Process process; try { process = pb.start(); process.waitFor(); + } catch (Exception e) { throw new RestException(e); } return process.exitValue(); + + int exitValue = process.exitValue(); + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue); + + return exitValue; } protected String[] getParserStartCommand(String name) { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; } protected String[] getEnrichmentStartCommand() { String[] command = new String [1] ; + String[] command = new String [3] ; End diff – No it's always been like that. The parser topology start script is the only one that supports the ksp flag. The enrichment and elasticsearch topology start scripts don't expect any input parameters.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user simonellistonball commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127474849

          — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java —
          @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException {
          protected int runCommand(String[] command) throws RestException {
          ProcessBuilder pb = getProcessBuilder(command);
          pb.inheritIO();

          • Process process = null;
            + LOG.debug("Running command: cmd={}", String.join(" ", command));
            +
            + Process process;
            try { process = pb.start(); process.waitFor(); + }

            catch (Exception e)

            { throw new RestException(e); }
          • return process.exitValue();
            +
            + int exitValue = process.exitValue();
            + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue);
            +
            + return exitValue;
            }

          protected String[] getParserStartCommand(String name)

          { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; }

          protected String[] getEnrichmentStartCommand() {

          • String[] command = new String[1];
            + String[] command = new String[3];
              • End diff –

          right, my bad, it was just parser script I had to change

          Show
          githubbot ASF GitHub Bot added a comment - Github user simonellistonball commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127474849 — Diff: metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java — @@ -75,37 +81,50 @@ public int stopIndexingTopology(boolean stopNow) throws RestException { protected int runCommand(String[] command) throws RestException { ProcessBuilder pb = getProcessBuilder(command); pb.inheritIO(); Process process = null; + LOG.debug("Running command: cmd={}", String.join(" ", command)); + + Process process; try { process = pb.start(); process.waitFor(); + } catch (Exception e) { throw new RestException(e); } return process.exitValue(); + + int exitValue = process.exitValue(); + LOG.debug("Command completed: cmd={}, exit={}", String.join(" ", command), exitValue); + + return exitValue; } protected String[] getParserStartCommand(String name) { - String[] command = new String[7]; + String[] command = new String[9]; command[0] = environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY); command[1] = "-k"; command[2] = environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY); command[3] = "-z"; command[4] = environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY); command[5] = "-s"; command[6] = name; + command[7] = "-ksp"; + command[8] = environment.getProperty(MetronRestConstants.KAFKA_SECURITY_PROTOCOL_SPRING_PROPERTY); return command; } protected String[] getEnrichmentStartCommand() { String[] command = new String [1] ; + String[] command = new String [3] ; End diff – right, my bad, it was just parser script I had to change
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickwallen commented on the issue:

          https://github.com/apache/metron/pull/647

          • I removed `-ksp` from Enrichment and Indexing.
          • I added `-e` also to the Parsers. This is also required to get them running.
          • Updated the test plan.

          I ran through a full test once successfully, but I changed a few minor things and just need to run through the manual test once more.

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/647 I removed `-ksp` from Enrichment and Indexing. I added `-e` also to the Parsers. This is also required to get them running. Updated the test plan. I ran through a full test once successfully, but I changed a few minor things and just need to run through the manual test once more.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickwallen commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127534192

          — Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2 —
          @@ -37,3 +37,5 @@ SECURITY_ENABLED=security_enabled|lower

          {% endif %}
          {% if metron_keytab_path is defined %}METRON_SERVICE_KEYTAB="metron_keytab_path"
          {% endif %}

          +KAFKA_SECURITY_PROTOCOL="kafka_security_protocol"
          +PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
          — End diff –

          I did not find any existing property that defines the `PARSER_TOPOLOGY_OPTIONS` value. It is hard coded in one of the MPack Python scripts (`parser_commands.py`) as `' -e ~' + self.__params.metron_user + '/.storm/storm.config'`.

          I thought that if I just put the exact value here, we are half way to making it configurable in Ambari, if we choose to do so.

          I also tried to use `~$

          {METRON_HOME}

          /.storm/storm.config`, but the `ProcessBuilder` does not do tilde expansion for user's home directories.

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickwallen commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127534192 — Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2 — @@ -37,3 +37,5 @@ SECURITY_ENABLED= security_enabled|lower {% endif %} {% if metron_keytab_path is defined %}METRON_SERVICE_KEYTAB=" metron_keytab_path " {% endif %} +KAFKA_SECURITY_PROTOCOL=" kafka_security_protocol " +PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config" — End diff – I did not find any existing property that defines the `PARSER_TOPOLOGY_OPTIONS` value. It is hard coded in one of the MPack Python scripts (`parser_commands.py`) as `' -e ~' + self.__params.metron_user + '/.storm/storm.config'`. I thought that if I just put the exact value here, we are half way to making it configurable in Ambari, if we choose to do so. I also tried to use `~$ {METRON_HOME} /.storm/storm.config`, but the `ProcessBuilder` does not do tilde expansion for user's home directories.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on a diff in the pull request:

          https://github.com/apache/metron/pull/647#discussion_r127555429

          — Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2 —
          @@ -37,3 +37,5 @@ SECURITY_ENABLED=security_enabled|lower

          {% endif %}
          {% if metron_keytab_path is defined %}METRON_SERVICE_KEYTAB="metron_keytab_path"
          {% endif %}

          +KAFKA_SECURITY_PROTOCOL="kafka_security_protocol"
          +PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config"
          — End diff –

          I think that's fine. If you really wanted it to mirror the implementation in the MPack, you could do:
          `PARSER_TOPOLOGY_OPTIONS="/home/metron_user/.storm/storm.config"`

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on a diff in the pull request: https://github.com/apache/metron/pull/647#discussion_r127555429 — Diff: metron-deployment/packaging/ambari/metron-mpack/src/main/resources/common-services/METRON/CURRENT/package/templates/metron.j2 — @@ -37,3 +37,5 @@ SECURITY_ENABLED= security_enabled|lower {% endif %} {% if metron_keytab_path is defined %}METRON_SERVICE_KEYTAB=" metron_keytab_path " {% endif %} +KAFKA_SECURITY_PROTOCOL=" kafka_security_protocol " +PARSER_TOPOLOGY_OPTIONS="/home/metron/.storm/storm.config" — End diff – I think that's fine. If you really wanted it to mirror the implementation in the MPack, you could do: `PARSER_TOPOLOGY_OPTIONS="/home/ metron_user /.storm/storm.config"`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickwallen commented on the issue:

          https://github.com/apache/metron/pull/647

          @merrimanr Ran it through a full test as described in the PR description and its looking good. Please take a look when you can.

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickwallen commented on the issue: https://github.com/apache/metron/pull/647 @merrimanr Ran it through a full test as described in the PR description and its looking good. Please take a look when you can.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user merrimanr commented on the issue:

          https://github.com/apache/metron/pull/647

          +1 validated with instructions in the PR description. Nice work!

          Show
          githubbot ASF GitHub Bot added a comment - Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/647 +1 validated with instructions in the PR description. Nice work!
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/metron/pull/647

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/metron/pull/647

            People

            • Assignee:
              nickwallen Nick Allen
              Reporter:
              nickwallen Nick Allen
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:

                Development