Uploaded image for project: 'Apache Twill'
  1. Apache Twill
  2. TWILL-138

Runtime change of Application runnable log level

    Details

    • Type: New Feature
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.9.0
    • Component/s: core
    • Labels:
      None

      Description

      With TWILL-24 is supported for setting log level when starting the Twill application, next enhancement is to able to send command to update the aggregate log level for a particular runnable in a Twill application.

        Issue Links

          Activity

          Hide
          gsps1 Shankar Selvam added a comment -

          hi henry, i would like to look at this issue, let me know if i can take it up from you, if you have not already worked on it. Thanks

          Show
          gsps1 Shankar Selvam added a comment - hi henry, i would like to look at this issue, let me know if i can take it up from you, if you have not already worked on it. Thanks
          Hide
          hsaputra Henry Saputra added a comment -

          Awesome, feel free to work on it.

          Thanks Shankar!

          Show
          hsaputra Henry Saputra added a comment - Awesome, feel free to work on it. Thanks Shankar!
          Hide
          yaojie Yaojie Feng added a comment - - edited

          To do this, we will first finish TWILL-195. After that, we will make following API changes to TwillPreparer and TwillController:
          Changes in TwillPreparer is for changing log levels before application starts.
          We add:

          TwillPreparer setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs);
          

          and

          TwillPreparer setLogLevel(String runnableName, Map<String, LogEntry.Level> logLevelRunnableArgs);
          

          The first method will accept a map which contains logger name and log level that need to be changed for the application.
          The second method will accept a runnableName, and a map of logger name and log level for that runnable.
          Implementation: With TWILL-195, we can pass the runtime specification to application using TwillRuntimeSpecification. So we store the log level arguments as a map: Map<String, Map<String, Log.Entry level>> and pass it with TwillRuntimeSpecification. Therefore, we can get it from the ApplicationMasterService and TwillContainerService and set the log level once the application starts.

          Changes in TwillPreparer is for changing log levels during runtime.
          We add:

          Future<String> setLogLevel(LogEntry.Level logLevel);
          
          Future<String> setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs);
          
          Future<String> setLogLevel(String runnableName, Map<String, LogEntry.Level> logLevels);
          

          The functionalities are similar to those in TwillPreparer. With these methods, we will be able to set log level during runtime by passing to either the whole application or a specific runnable.
          Implementation: Using messages to send out the log level change request. TwillController will send messages to zookeeper. Message will be received by ApplicationMasterService and be propagated to corresponding runnables using RunningContainers. The corresponding TwillContainerService will receive the message and change the log level for corresponding logger. If it is successful, update the corresponding log information in resources for each container.

          Show
          yaojie Yaojie Feng added a comment - - edited To do this, we will first finish TWILL-195 . After that, we will make following API changes to TwillPreparer and TwillController: Changes in TwillPreparer is for changing log levels before application starts. We add: TwillPreparer setLogLevel(Map< String , LogEntry.Level> logLevelAppArgs); and TwillPreparer setLogLevel( String runnableName, Map< String , LogEntry.Level> logLevelRunnableArgs); The first method will accept a map which contains logger name and log level that need to be changed for the application. The second method will accept a runnableName, and a map of logger name and log level for that runnable. Implementation: With TWILL-195 , we can pass the runtime specification to application using TwillRuntimeSpecification. So we store the log level arguments as a map: Map<String, Map<String, Log.Entry level>> and pass it with TwillRuntimeSpecification. Therefore, we can get it from the ApplicationMasterService and TwillContainerService and set the log level once the application starts. Changes in TwillPreparer is for changing log levels during runtime. We add: Future< String > setLogLevel(LogEntry.Level logLevel); Future< String > setLogLevel(Map< String , LogEntry.Level> logLevelAppArgs); Future< String > setLogLevel( String runnableName, Map< String , LogEntry.Level> logLevels); The functionalities are similar to those in TwillPreparer. With these methods, we will be able to set log level during runtime by passing to either the whole application or a specific runnable. Implementation: Using messages to send out the log level change request. TwillController will send messages to zookeeper. Message will be received by ApplicationMasterService and be propagated to corresponding runnables using RunningContainers. The corresponding TwillContainerService will receive the message and change the log level for corresponding logger. If it is successful, update the corresponding log information in resources for each container.
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user yaojiefeng opened a pull request:

          https://github.com/apache/twill/pull/12

          TWILL-138 TWILL-195 Runtime log level change

          JIRA: https://issues.apache.org/jira/browse/TWILL-138
          https://issues.apache.org/jira/browse/TWILL-195

          The first two commits are for TWILL-195, the rest are for TWILL-138.

          Summary: Make API changes to TwillController and TwillPreparer to set the log level for the twill application or specific twill runnables at runtime or before the application starts. The TwillPreparer will serialize the log level change requirements and deserialize them once service starts. The TwillController will use messages to send out log level change request. Details for implementation can be found in JIRAs.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/yaojiefeng/twill feature/changeLogLevel

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/twill/pull/12.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #12


          commit 19a1971eaca72d1d576bd252be364c7be6aebcc4
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-22T21:35:04Z

          add TwillRuntimeSpecification

          commit d64abe694cda37860b6fd5ff6e4641bface3cf42
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-23T01:04:42Z

          remove some env keys

          commit e72746cac4981c0a40edef60dedb542c8d279190
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-23T07:53:50Z

          Log level API change for TwillController

          commit 7d497b8c63689128e985343c37c21b766b7d85db
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-24T00:02:06Z

          Log level API change for TwillPreparer

          commit 3b89cba28182414c788f8fe1ede2761c2ee4778b
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-25T08:47:15Z

          Remove unused imports, add javadoc, and nit

          commit 2c005ec7b7cd14b620753e9c62374c05673dab41
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-09-25T09:25:26Z

          update the unit test


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user yaojiefeng opened a pull request: https://github.com/apache/twill/pull/12 TWILL-138 TWILL-195 Runtime log level change JIRA: https://issues.apache.org/jira/browse/TWILL-138 https://issues.apache.org/jira/browse/TWILL-195 The first two commits are for TWILL-195 , the rest are for TWILL-138 . Summary: Make API changes to TwillController and TwillPreparer to set the log level for the twill application or specific twill runnables at runtime or before the application starts. The TwillPreparer will serialize the log level change requirements and deserialize them once service starts. The TwillController will use messages to send out log level change request. Details for implementation can be found in JIRAs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yaojiefeng/twill feature/changeLogLevel Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/12.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #12 commit 19a1971eaca72d1d576bd252be364c7be6aebcc4 Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-22T21:35:04Z add TwillRuntimeSpecification commit d64abe694cda37860b6fd5ff6e4641bface3cf42 Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-23T01:04:42Z remove some env keys commit e72746cac4981c0a40edef60dedb542c8d279190 Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-23T07:53:50Z Log level API change for TwillController commit 7d497b8c63689128e985343c37c21b766b7d85db Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-24T00:02:06Z Log level API change for TwillPreparer commit 3b89cba28182414c788f8fe1ede2761c2ee4778b Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-25T08:47:15Z Remove unused imports, add javadoc, and nit commit 2c005ec7b7cd14b620753e9c62374c05673dab41 Author: yaojiefeng <yaojie@cask.co> Date: 2016-09-25T09:25:26Z update the unit test
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hsaputra commented on the issue:

          https://github.com/apache/twill/pull/12

          Does TWILL-138 depends on TWILL-195 ?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/12 Does TWILL-138 depends on TWILL-195 ?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on the issue:

          https://github.com/apache/twill/pull/12

          Yes, with TWILL-195, we will be able to pass log level arguments through TwillRuntimeSpecification

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on the issue: https://github.com/apache/twill/pull/12 Yes, with TWILL-195 , we will be able to pass log level arguments through TwillRuntimeSpecification
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on the issue:

          https://github.com/apache/twill/pull/12

          close this to create separate PRs for different JIRAs, will create PR for TWILL-195 first since it is a prerequisite for TWILL-138

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on the issue: https://github.com/apache/twill/pull/12 close this to create separate PRs for different JIRAs, will create PR for TWILL-195 first since it is a prerequisite for TWILL-138
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng closed the pull request at:

          https://github.com/apache/twill/pull/12

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng closed the pull request at: https://github.com/apache/twill/pull/12
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user yaojiefeng opened a pull request:

          https://github.com/apache/twill/pull/14

          TWILL-138 Runtime log level change

          JIRA: https://issues.apache.org/jira/browse/TWILL-138

          Summary: Make API changes to TwillController and TwillPreparer to set the log level for the twill application or specific twill runnables at runtime or before the application starts. The TwillPreparer will serialize the log level change requirements and deserialize them once service starts. The TwillController will use messages to send out log level change request. Details for implementation can be found in JIRAs.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/yaojiefeng/twill feature/loglevelChangeAtRuntime

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/twill/pull/14.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #14


          commit f14ff6a3afca8cd798d824161485cf91cf18b284
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-10-05T23:17:26Z

          (TWILL-138) Change Log level at Runtime

          commit 96a8af0acb6763a3ed54bc1848690b08d8ce0d1b
          Author: yaojiefeng <yaojie@cask.co>
          Date: 2016-10-05T23:19:41Z

          rename


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user yaojiefeng opened a pull request: https://github.com/apache/twill/pull/14 TWILL-138 Runtime log level change JIRA: https://issues.apache.org/jira/browse/TWILL-138 Summary: Make API changes to TwillController and TwillPreparer to set the log level for the twill application or specific twill runnables at runtime or before the application starts. The TwillPreparer will serialize the log level change requirements and deserialize them once service starts. The TwillController will use messages to send out log level change request. Details for implementation can be found in JIRAs. You can merge this pull request into a Git repository by running: $ git pull https://github.com/yaojiefeng/twill feature/loglevelChangeAtRuntime Alternatively you can review and apply these changes as the patch at: https://github.com/apache/twill/pull/14.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14 commit f14ff6a3afca8cd798d824161485cf91cf18b284 Author: yaojiefeng <yaojie@cask.co> Date: 2016-10-05T23:17:26Z ( TWILL-138 ) Change Log level at Runtime commit 96a8af0acb6763a3ed54bc1848690b08d8ce0d1b Author: yaojiefeng <yaojie@cask.co> Date: 2016-10-05T23:19:41Z rename
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098229

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -30,17 +35,26 @@
          private final int memoryMB;
          private final String host;
          private final Integer debugPort;

          • private final Level logLevel;
            + private final Map<String, LogEntry.Level> logLevelArguments;
            + private Level rootLogLevel;
            +
            + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB,
            + String host, Integer debugPort, Level rootLogLevel) { + this(instanceId, containerId, cores, memoryMB, host, debugPort, rootLogLevel, + new HashMap<String, LogEntry.Level>()); + }

          public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB,

          • String host, Integer debugPort, Level logLevel) {
            + String host, Integer debugPort,
            + Level rootLogLevel, Map<String, LogEntry.Level> logLevelArguments) {
            this.instanceId = instanceId;
            this.containerId = containerId;
            this.virtualCores = cores;
            this.memoryMB = memoryMB;
            this.host = host;
            this.debugPort = debugPort;
          • this.logLevel = logLevel;
            + this.rootLogLevel = rootLogLevel;
            + this.logLevelArguments = logLevelArguments;
              • End diff –

          should just call `logLevels`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098229 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -30,17 +35,26 @@ private final int memoryMB; private final String host; private final Integer debugPort; private final Level logLevel; + private final Map<String, LogEntry.Level> logLevelArguments; + private Level rootLogLevel; + + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB, + String host, Integer debugPort, Level rootLogLevel) { + this(instanceId, containerId, cores, memoryMB, host, debugPort, rootLogLevel, + new HashMap<String, LogEntry.Level>()); + } public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB, String host, Integer debugPort, Level logLevel) { + String host, Integer debugPort, + Level rootLogLevel, Map<String, LogEntry.Level> logLevelArguments) { this.instanceId = instanceId; this.containerId = containerId; this.virtualCores = cores; this.memoryMB = memoryMB; this.host = host; this.debugPort = debugPort; this.logLevel = logLevel; + this.rootLogLevel = rootLogLevel; + this.logLevelArguments = logLevelArguments; End diff – should just call `logLevels`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82097429

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,29 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications running in a container.

              • End diff –

          In a container? Which container? Isn't that this affect all containers of all runnables as well as the AM?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82097429 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,29 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications running in a container. End diff – In a container? Which container? Isn't that this affect all containers of all runnables as well as the AM?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098021

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java —
          @@ -59,6 +61,16 @@
          /**

          • @return the enabled log level for the container where the runnable is running in.
            */
          • LogEntry.Level getLogLevel();
            + LogEntry.Level getRootLogLevel();
            +
            + /**
            + * @return the enabled log level arguments for the container where the runnable is running in.
            + */
            + Map<String, LogEntry.Level> getLogLevelArguments();

          + /**
          + * Update the log level arguments for the container where the runnable is running in.
          + * @param logLevelArguments
          + */
          + void updateLogLevel(Map<String, LogEntry.Level> logLevelArguments);
          — End diff –

          Why there is an update method? This interface is supposedly for reporting, hence is immutable.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098021 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java — @@ -59,6 +61,16 @@ /** @return the enabled log level for the container where the runnable is running in. */ LogEntry.Level getLogLevel(); + LogEntry.Level getRootLogLevel(); + + /** + * @return the enabled log level arguments for the container where the runnable is running in. + */ + Map<String, LogEntry.Level> getLogLevelArguments(); + /** + * Update the log level arguments for the container where the runnable is running in. + * @param logLevelArguments + */ + void updateLogLevel(Map<String, LogEntry.Level> logLevelArguments); — End diff – Why there is an update method? This interface is supposedly for reporting, hence is immutable.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82097272

          — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java —
          @@ -61,4 +63,19 @@

          • @return list of services of the application master.
            */
            List<String> getServices();
            +
            + /**
            + * Get the map of the log level arguments of the twill application.
            + *
            + * @return the map of the log level arguments of the twill application
            + */
            + Map<String, Map<String, LogEntry.Level>> getLogLevelArguments();
              • End diff –

          What's the contract? What's the key? Does it always return all log levels even nothing was set?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82097272 — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java — @@ -61,4 +63,19 @@ @return list of services of the application master. */ List<String> getServices(); + + /** + * Get the map of the log level arguments of the twill application. + * + * @return the map of the log level arguments of the twill application + */ + Map<String, Map<String, LogEntry.Level>> getLogLevelArguments(); End diff – What's the contract? What's the key? Does it always return all log levels even nothing was set?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82097947

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java —
          @@ -59,6 +61,16 @@
          /**

          • @return the enabled log level for the container where the runnable is running in.
            */
          • LogEntry.Level getLogLevel();
            + LogEntry.Level getRootLogLevel();
            +
            + /**
            + * @return the enabled log level arguments for the container where the runnable is running in.
            + */
            + Map<String, LogEntry.Level> getLogLevelArguments();
              • End diff –

          What do mean by "LogLevelArguments"? Should be "getLogLevels".

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82097947 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java — @@ -59,6 +61,16 @@ /** @return the enabled log level for the container where the runnable is running in. */ LogEntry.Level getLogLevel(); + LogEntry.Level getRootLogLevel(); + + /** + * @return the enabled log level arguments for the container where the runnable is running in. + */ + Map<String, LogEntry.Level> getLogLevelArguments(); End diff – What do mean by "LogLevelArguments"? Should be "getLogLevels".
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82097869

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,7 +227,7 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications running in a container.
              • End diff –

          Same here

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82097869 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,7 +227,7 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications running in a container. End diff – Same here
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098323

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -90,8 +104,21 @@ public Integer getDebugPort() {
          }

          @Override

          • public Level getLogLevel() {
          • return logLevel;
            + public Level getRootLogLevel() { + return rootLogLevel; + }

            +
            + @Override
            + public Map<String, LogEntry.Level> getLogLevelArguments()

            { + return logLevelArguments; + }

            +
            + @Override
            + public void updateLogLevel(Map<String, Level> logLevelArguments) {

              • End diff –

          Better keep this class immutable. If there is a change in the log level, simply create a new instance.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098323 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -90,8 +104,21 @@ public Integer getDebugPort() { } @Override public Level getLogLevel() { return logLevel; + public Level getRootLogLevel() { + return rootLogLevel; + } + + @Override + public Map<String, LogEntry.Level> getLogLevelArguments() { + return logLevelArguments; + } + + @Override + public void updateLogLevel(Map<String, Level> logLevelArguments) { End diff – Better keep this class immutable. If there is a change in the log level, simply create a new instance.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098250

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -236,6 +236,23 @@
          TwillPreparer setLogLevel(LogEntry.Level logLevel);

          /**
          + * Set the log levels for requested logger names for Twill applications running in a container.
          + *
          + * @param logLevelAppArgs The

          {@link Map}

          contains the requested logger names and log levels that need to be set.
          + * @return This

          {@link TwillPreparer}

          .
          + */
          + TwillPreparer setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs);
          — End diff –

          `setLogLevels`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098250 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -236,6 +236,23 @@ TwillPreparer setLogLevel(LogEntry.Level logLevel); /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevelAppArgs The {@link Map} contains the requested logger names and log levels that need to be set. + * @return This {@link TwillPreparer} . + */ + TwillPreparer setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs); — End diff – `setLogLevels`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098265

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,29 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications running in a container.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> setLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevelAppArgs The

            {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future} that will be completed when the set log level operation has been done.
            + */
            + Future<String> setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevels The {@link Map}

            contains the requested logger name and log level that need to be changed.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> setLogLevel(String runnableName, Map<String, LogEntry.Level> logLevels);

              • End diff –

          `setLogLevels`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098265 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,29 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications running in a container. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> setLogLevel(LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevelAppArgs The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> setLogLevel(Map<String, LogEntry.Level> logLevelAppArgs); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevels The {@link Map} contains the requested logger name and log level that need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> setLogLevel(String runnableName, Map<String, LogEntry.Level> logLevels); End diff – `setLogLevels`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098631

          — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java —
          @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName)

          @Override
          public final ListenableFuture<Set<String>> restartInstances(Map<String,

          • ? extends Set<Integer>> runnableToInstanceIds) {
            + ? extends Set<Integer>> runnableToInstanceIds) {
              • End diff –

          This new line is awkward, even in the old code. The parent actually returns `Future` instead of returning `ListenableFuture`. If you change the return type to `Future`, everything should fits on one line.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098631 — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java — @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName) @Override public final ListenableFuture<Set<String>> restartInstances(Map<String, ? extends Set<Integer>> runnableToInstanceIds) { + ? extends Set<Integer>> runnableToInstanceIds) { End diff – This new line is awkward, even in the old code. The parent actually returns `Future` instead of returning `ListenableFuture`. If you change the return type to `Future`, everything should fits on one line.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82099064

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -72,7 +76,7 @@ public String getZkConnectStr()

          { return zkConnectStr; }
          • public RunId getTwillRunId() {
            + public RunId getTwillAppRunId() {
              • End diff –

          There is only one `RunId`, why not just call if `runId` and `getRunId` for the field name and method name respectively?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82099064 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -72,7 +76,7 @@ public String getZkConnectStr() { return zkConnectStr; } public RunId getTwillRunId() { + public RunId getTwillAppRunId() { End diff – There is only one `RunId`, why not just call if `runId` and `getRunId` for the field name and method name respectively?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82098461

          — Diff: twill-common/src/main/java/org/apache/twill/internal/Constants.java —
          @@ -82,6 +82,14 @@ private Files() {
          }
          }

          + /**
          + * Constants for logLevel system messages.
          + */
          + public static final class SystemMessages {
          — End diff –

          Why need string constants here? All system messages/commands should be defined in `SystemMessages`. You can use object comparison instead of string comparison.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82098461 — Diff: twill-common/src/main/java/org/apache/twill/internal/Constants.java — @@ -82,6 +82,14 @@ private Files() { } } + /** + * Constants for logLevel system messages. + */ + public static final class SystemMessages { — End diff – Why need string constants here? All system messages/commands should be defined in `SystemMessages`. You can use object comparison instead of string comparison.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82099170

          — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java —
          @@ -68,6 +73,21 @@ public static Message updateRunnablesInstances(Command updateCommand)

          { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); }

          + public static Message setLogLevel(Map<String, LogEntry.Level> logLevelArguments) {
          + return setLogLevel(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevelArguments);
          — End diff –

          Why not use the `Message.Scope.RUNNABLES`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82099170 — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java — @@ -68,6 +73,21 @@ public static Message updateRunnablesInstances(Command updateCommand) { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); } + public static Message setLogLevel(Map<String, LogEntry.Level> logLevelArguments) { + return setLogLevel(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevelArguments); — End diff – Why not use the `Message.Scope.RUNNABLES`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82099417

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,85 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.internal.Constants;
          +
          +import java.util.HashMap;
          +import java.util.Map;
          +import java.util.Set;
          +import javax.annotation.Nullable;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) {
          — End diff –

          Seems like this method only used in `SystemMessages`. Better move it there as a private method. Haven't too many "util" class is not good.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82099417 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.internal.Constants; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) { — End diff – Seems like this method only used in `SystemMessages`. Better move it there as a private method. Haven't too many "util" class is not good.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82099614

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,85 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.internal.Constants;
          +
          +import java.util.HashMap;
          +import java.util.Map;
          +import java.util.Set;
          +import javax.annotation.Nullable;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) {
          + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, LogEntry.Level, String>() {
          + @Override
          + public String transformEntry(String loggerName, LogEntry.Level level)

          { + return level.name(); + }

          + });
          + }
          +
          + /**
          + * Convert the log level argument type to LogEntry.Level
          + */
          + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevelArguments) {
          — End diff –

          Similarly this one, it's better having a method in `SystemMessages` that extract the log level map from a `Messages`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82099614 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.internal.Constants; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) { + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, LogEntry.Level, String>() { + @Override + public String transformEntry(String loggerName, LogEntry.Level level) { + return level.name(); + } + }); + } + + /** + * Convert the log level argument type to LogEntry.Level + */ + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevelArguments) { — End diff – Similarly this one, it's better having a method in `SystemMessages` that extract the log level map from a `Messages`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82099632

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,85 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.internal.Constants;
          +
          +import java.util.HashMap;
          +import java.util.Map;
          +import java.util.Set;
          +import javax.annotation.Nullable;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) {
          + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, LogEntry.Level, String>() {
          + @Override
          + public String transformEntry(String loggerName, LogEntry.Level level)

          { + return level.name(); + }

          + });
          + }
          +
          + /**
          + * Convert the log level argument type to LogEntry.Level
          + */
          + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevelArguments) {
          + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, String, LogEntry.Level>() {
          + @Override
          + public LogEntry.Level transformEntry(String loggerName, String level)

          { + return LogEntry.Level.valueOf(level); + }

          + });
          + }
          +
          + /**
          + * Get the log level arguments for a twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevelArguments the arguments for all runnables.
          + * @return the map of the log level arguments for the runnable, empty if there is no argument.
          + */
          + public static Map<String, LogEntry.Level> getLogLevelForRunnable(
          — End diff –

          Same here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82099632 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.internal.Constants; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) { + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, LogEntry.Level, String>() { + @Override + public String transformEntry(String loggerName, LogEntry.Level level) { + return level.name(); + } + }); + } + + /** + * Convert the log level argument type to LogEntry.Level + */ + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevelArguments) { + return Maps.transformEntries(logLevelArguments, new Maps.EntryTransformer<String, String, LogEntry.Level>() { + @Override + public LogEntry.Level transformEntry(String loggerName, String level) { + return LogEntry.Level.valueOf(level); + } + }); + } + + /** + * Get the log level arguments for a twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevelArguments the arguments for all runnables. + * @return the map of the log level arguments for the runnable, empty if there is no argument. + */ + public static Map<String, LogEntry.Level> getLogLevelForRunnable( — End diff – Same here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82101139

          — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java —
          @@ -61,4 +63,19 @@

          • @return list of services of the application master.
            */
            List<String> getServices();
            +
            + /**
            + * Get the map of the log level arguments of the twill application.
            + *
            + * @return the map of the log level arguments of the twill application
            + */
            + Map<String, Map<String, LogEntry.Level>> getLogLevelArguments();
              • End diff –

          The key of the map is the runnable name, and the value of the map is a map of logger name and log level pair. If nothing is set for log level, it will return an empty map.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82101139 — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java — @@ -61,4 +63,19 @@ @return list of services of the application master. */ List<String> getServices(); + + /** + * Get the map of the log level arguments of the twill application. + * + * @return the map of the log level arguments of the twill application + */ + Map<String, Map<String, LogEntry.Level>> getLogLevelArguments(); End diff – The key of the map is the runnable name, and the value of the map is a map of logger name and log level pair. If nothing is set for log level, it will return an empty map.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82101218

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,29 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications running in a container.

              • End diff –

          Yes. It will affect all containers, I will change the javadoc

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82101218 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,29 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications running in a container. End diff – Yes. It will affect all containers, I will change the javadoc
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82102988

          — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java —
          @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName)

          @Override
          public final ListenableFuture<Set<String>> restartInstances(Map<String,

          • ? extends Set<Integer>> runnableToInstanceIds) {
            + ? extends Set<Integer>> runnableToInstanceIds) {
              • End diff –

          I cannot change the type to Future for this method, there is a another method calling it which requires the return type to be a ListenableFuture

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82102988 — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java — @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName) @Override public final ListenableFuture<Set<String>> restartInstances(Map<String, ? extends Set<Integer>> runnableToInstanceIds) { + ? extends Set<Integer>> runnableToInstanceIds) { End diff – I cannot change the type to Future for this method, there is a another method calling it which requires the return type to be a ListenableFuture
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82886998

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,85 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.internal.Constants;
          +
          +import java.util.HashMap;
          +import java.util.Map;
          +import java.util.Set;
          +import javax.annotation.Nullable;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) {
          — End diff –

          The methods in `LogLevelUtil` are not only used in `SystemMessages`. They are also used in `RunningContainers` and `TwillContainerMain`, is it good to have all these methods in `SystemMessages`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82886998 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,85 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.internal.Constants; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevelArguments) { — End diff – The methods in `LogLevelUtil` are not only used in `SystemMessages`. They are also used in `RunningContainers` and `TwillContainerMain`, is it good to have all these methods in `SystemMessages`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82887283

          — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java —
          @@ -68,6 +73,21 @@ public static Message updateRunnablesInstances(Command updateCommand)

          { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); }

          + public static Message setLogLevel(Map<String, LogEntry.Level> logLevelArguments) {
          + return setLogLevel(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevelArguments);
          — End diff –

          The constants about log levels are only used for `Messages`, they are also used in `TwillPreparer`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82887283 — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java — @@ -68,6 +73,21 @@ public static Message updateRunnablesInstances(Command updateCommand) { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); } + public static Message setLogLevel(Map<String, LogEntry.Level> logLevelArguments) { + return setLogLevel(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevelArguments); — End diff – The constants about log levels are only used for `Messages`, they are also used in `TwillPreparer`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82901178

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -298,6 +299,29 @@ public TwillPreparer addSecureStore(SecureStore secureStore) {
          public TwillPreparer setLogLevel(LogEntry.Level logLevel) {
          Preconditions.checkNotNull(logLevel);
          this.logLevel = logLevel;
          + Map<String, LogEntry.Level> logLevels = new HashMap<>();
          + logLevels.put(Logger.ROOT_LOGGER_NAME, logLevel);
          + saveLogLevels(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevels);
          — End diff –

          Why use the `LOG_ALL_RUNNABLES` as a runnable name? We have the `twillSpec` in this class, so it's easy to just expand the log level settings for each runnable.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82901178 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -298,6 +299,29 @@ public TwillPreparer addSecureStore(SecureStore secureStore) { public TwillPreparer setLogLevel(LogEntry.Level logLevel) { Preconditions.checkNotNull(logLevel); this.logLevel = logLevel; + Map<String, LogEntry.Level> logLevels = new HashMap<>(); + logLevels.put(Logger.ROOT_LOGGER_NAME, logLevel); + saveLogLevels(Constants.SystemMessages.LOG_ALL_RUNNABLES, logLevels); — End diff – Why use the `LOG_ALL_RUNNABLES` as a runnable name? We have the `twillSpec` in this class, so it's easy to just expand the log level settings for each runnable.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82901837

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,87 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.internal.Constants;
          +
          +import java.util.HashMap;
          +import java.util.Map;
          +import java.util.Set;
          +import javax.annotation.Nullable;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) {
          + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, LogEntry.Level, String>() {
          + @Override
          + public String transformEntry(String loggerName, LogEntry.Level level)

          { + return level.name(); + }

          + });
          + }
          +
          + /**
          + * Convert the log level argument type to LogEntry.Level
          + */
          + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevels) {
          + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, String, LogEntry.Level>() {
          + @Override
          + public LogEntry.Level transformEntry(String loggerName, String level)

          { + return LogEntry.Level.valueOf(level); + }

          + });
          + }
          +
          + /**
          + * Get the log level arguments for a twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevels the arguments for all runnables.
          + * @return the map of the log level arguments for the runnable, empty if there is no argument.
          + */
          + public static Map<String, LogEntry.Level> getLogLevelForRunnable(
          — End diff –

          We shouldn't have this method as we shouldn't use `LOG_ALL_RUNNABLES` as a special runnable name. It's better to have the log levels populated to each runnable log level map when setting/changing it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82901837 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,87 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.internal.Constants; + +import java.util.HashMap; +import java.util.Map; +import java.util.Set; +import javax.annotation.Nullable; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) { + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, LogEntry.Level, String>() { + @Override + public String transformEntry(String loggerName, LogEntry.Level level) { + return level.name(); + } + }); + } + + /** + * Convert the log level argument type to LogEntry.Level + */ + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevels) { + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, String, LogEntry.Level>() { + @Override + public LogEntry.Level transformEntry(String loggerName, String level) { + return LogEntry.Level.valueOf(level); + } + }); + } + + /** + * Get the log level arguments for a twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevels the arguments for all runnables. + * @return the map of the log level arguments for the runnable, empty if there is no argument. + */ + public static Map<String, LogEntry.Level> getLogLevelForRunnable( — End diff – We shouldn't have this method as we shouldn't use `LOG_ALL_RUNNABLES` as a special runnable name. It's better to have the log levels populated to each runnable log level map when setting/changing it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82899928

          — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java —
          @@ -0,0 +1,237 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.twill.yarn;
          +
          +import com.google.common.base.Stopwatch;
          +import com.google.common.collect.ImmutableMap;
          +import org.apache.twill.api.AbstractTwillRunnable;
          +import org.apache.twill.api.ResourceReport;
          +import org.apache.twill.api.TwillApplication;
          +import org.apache.twill.api.TwillController;
          +import org.apache.twill.api.TwillRunResources;
          +import org.apache.twill.api.TwillSpecification;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.api.logging.PrinterLogHandler;
          +import org.apache.twill.common.Threads;
          +import org.junit.Assert;
          +import org.junit.Test;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.PrintWriter;
          +import java.util.Map;
          +import java.util.concurrent.CountDownLatch;
          +import java.util.concurrent.TimeUnit;
          +
          +/**
          + * Test changing log level for a twill runnable.
          + */
          +public class LogLevelChangeTestRun extends BaseYarnTest {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class);
          +
          + /**
          + * Twill runnable.
          + */
          + public static final class LogLevelTestRunnable extends AbstractTwillRunnable {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class);
          +
          + private volatile Thread runThread;
          +
          + @Override
          + public void run() {
          + this.runThread = Thread.currentThread();
          +
          + // check if the initial log level is DEBUG
          + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled());
          +
          + int i = 0;
          + while (!Thread.interrupted()) {
          — End diff –

          Without sleeping/pausing, the thread can take up all CPU resources, causing starvation for other threads. On machine with single core, unit-test could fail due to this.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82899928 — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java — @@ -0,0 +1,237 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.twill.yarn; + +import com.google.common.base.Stopwatch; +import com.google.common.collect.ImmutableMap; +import org.apache.twill.api.AbstractTwillRunnable; +import org.apache.twill.api.ResourceReport; +import org.apache.twill.api.TwillApplication; +import org.apache.twill.api.TwillController; +import org.apache.twill.api.TwillRunResources; +import org.apache.twill.api.TwillSpecification; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.api.logging.PrinterLogHandler; +import org.apache.twill.common.Threads; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.PrintWriter; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +/** + * Test changing log level for a twill runnable. + */ +public class LogLevelChangeTestRun extends BaseYarnTest { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class); + + /** + * Twill runnable. + */ + public static final class LogLevelTestRunnable extends AbstractTwillRunnable { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class); + + private volatile Thread runThread; + + @Override + public void run() { + this.runThread = Thread.currentThread(); + + // check if the initial log level is DEBUG + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled()); + + int i = 0; + while (!Thread.interrupted()) { — End diff – Without sleeping/pausing, the thread can take up all CPU resources, causing starvation for other threads. On machine with single core, unit-test could fail due to this.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82898688

          — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/ResourceReportTestRun.java —
          @@ -36,6 +36,7 @@
          import org.junit.Test;
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;
          +import scala.reflect.api.Constants;
          — End diff –

          unused import.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82898688 — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/ResourceReportTestRun.java — @@ -36,6 +36,7 @@ import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import scala.reflect.api.Constants; — End diff – unused import.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r82901524

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -17,26 +17,41 @@
          */
          package org.apache.twill.internal.container;

          +import ch.qos.logback.classic.Level;
          +import ch.qos.logback.classic.LoggerContext;
          +import com.google.common.base.Charsets;
          import com.google.common.base.Preconditions;
          +import com.google.common.io.Files;
          +import com.google.common.reflect.TypeToken;
          import com.google.common.util.concurrent.Futures;
          import com.google.common.util.concurrent.ListenableFuture;
          import com.google.common.util.concurrent.SettableFuture;
          +import com.google.gson.Gson;
          import org.apache.twill.api.Command;
          import org.apache.twill.api.RunId;
          import org.apache.twill.api.TwillRunnable;
          import org.apache.twill.api.TwillRunnableSpecification;
          +import org.apache.twill.api.logging.LogEntry;
          import org.apache.twill.common.Threads;
          import org.apache.twill.filesystem.Location;
          import org.apache.twill.internal.BasicTwillContext;
          +import org.apache.twill.internal.Constants;
          import org.apache.twill.internal.ContainerInfo;
          import org.apache.twill.internal.ContainerLiveNodeData;
          import org.apache.twill.internal.state.Message;
          import org.apache.twill.internal.utils.Instances;
          +import org.apache.twill.internal.utils.LogLevelUtil;
          import org.apache.twill.internal.yarn.AbstractYarnTwillService;
          import org.apache.twill.zookeeper.ZKClient;
          +import org.slf4j.ILoggerFactory;
          import org.slf4j.Logger;
          import org.slf4j.LoggerFactory;

          +import java.io.File;
          +import java.io.FileNotFoundException;
          +import java.io.IOException;
          +import java.io.Reader;
          — End diff –

          Unused imports?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r82901524 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -17,26 +17,41 @@ */ package org.apache.twill.internal.container; +import ch.qos.logback.classic.Level; +import ch.qos.logback.classic.LoggerContext; +import com.google.common.base.Charsets; import com.google.common.base.Preconditions; +import com.google.common.io.Files; +import com.google.common.reflect.TypeToken; import com.google.common.util.concurrent.Futures; import com.google.common.util.concurrent.ListenableFuture; import com.google.common.util.concurrent.SettableFuture; +import com.google.gson.Gson; import org.apache.twill.api.Command; import org.apache.twill.api.RunId; import org.apache.twill.api.TwillRunnable; import org.apache.twill.api.TwillRunnableSpecification; +import org.apache.twill.api.logging.LogEntry; import org.apache.twill.common.Threads; import org.apache.twill.filesystem.Location; import org.apache.twill.internal.BasicTwillContext; +import org.apache.twill.internal.Constants; import org.apache.twill.internal.ContainerInfo; import org.apache.twill.internal.ContainerLiveNodeData; import org.apache.twill.internal.state.Message; import org.apache.twill.internal.utils.Instances; +import org.apache.twill.internal.utils.LogLevelUtil; import org.apache.twill.internal.yarn.AbstractYarnTwillService; import org.apache.twill.zookeeper.ZKClient; +import org.slf4j.ILoggerFactory; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import java.io.File; +import java.io.FileNotFoundException; +import java.io.IOException; +import java.io.Reader; — End diff – Unused imports?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83108745

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -41,7 +43,8 @@
          private static final String MEMORY_MB = "memoryMB";
          private static final String VIRTUAL_CORES = "virtualCores";
          private static final String DEBUG_PORT = "debugPort";

          • private static final String LOG_LEVEL = "logLevel";
            + private static final String LOG_LEVEL = "rootLogLevel";
              • End diff –

          Should rename to `ROOT_LOG_LEVEL` to be consistent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83108745 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -41,7 +43,8 @@ private static final String MEMORY_MB = "memoryMB"; private static final String VIRTUAL_CORES = "virtualCores"; private static final String DEBUG_PORT = "debugPort"; private static final String LOG_LEVEL = "logLevel"; + private static final String LOG_LEVEL = "rootLogLevel"; End diff – Should rename to `ROOT_LOG_LEVEL` to be consistent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83108468

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillContainerLauncher.java —
          @@ -84,22 +84,29 @@ public TwillContainerLauncher(RuntimeSpecification runtimeSpec, ContainerInfo co

          • @param classPath The class path to load classes for the container.
          • @return instance of {@link TwillContainerController}

            to control the container run.
            */

          • public TwillContainerController start(RunId runId, int instanceId, Class<?> mainClass, String classPath) {
            + public TwillContainerController start(RunId runId, int instanceId, Class<?> mainClass, String classPath,
            + Location logLevelLocation) {
              • End diff –

          Is `logLevelLocation` nullable? I see you check that in line 104. If that's the case, annotate this parameter with `@Nullable`. Also, please update the javadoc accordingly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83108468 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillContainerLauncher.java — @@ -84,22 +84,29 @@ public TwillContainerLauncher(RuntimeSpecification runtimeSpec, ContainerInfo co @param classPath The class path to load classes for the container. @return instance of {@link TwillContainerController} to control the container run. */ public TwillContainerController start(RunId runId, int instanceId, Class<?> mainClass, String classPath) { + public TwillContainerController start(RunId runId, int instanceId, Class<?> mainClass, String classPath, + Location logLevelLocation) { End diff – Is `logLevelLocation` nullable? I see you check that in line 104. If that's the case, annotate this parameter with `@Nullable`. Also, please update the javadoc accordingly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83108861

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ
          if (src.getDebugPort() != null)

          { json.addProperty(DEBUG_PORT, src.getDebugPort()); }
          • if (src.getLogLevel() != null) {
          • json.addProperty(LOG_LEVEL, src.getLogLevel().toString());
            + if (src.getRootLogLevel() != null) {
            + json.addProperty(LOG_LEVEL, src.getRootLogLevel().toString());
              • End diff –

          for serializing enum, always use `.name()` instead of `.toString()`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83108861 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ if (src.getDebugPort() != null) { json.addProperty(DEBUG_PORT, src.getDebugPort()); } if (src.getLogLevel() != null) { json.addProperty(LOG_LEVEL, src.getLogLevel().toString()); + if (src.getRootLogLevel() != null) { + json.addProperty(LOG_LEVEL, src.getRootLogLevel().toString()); End diff – for serializing enum, always use `.name()` instead of `.toString()`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83079571

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,30 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.

              • End diff –

          What is the `String` in the resulting `Future` represents? Is it the app name or something else?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83079571 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,30 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. End diff – What is the `String` in the resulting `Future` represents? Is it the app name or something else?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83089307

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -149,6 +161,51 @@ public void setServices(List<String> services)

          { return services.get(); }

          + /**
          + * Get the map of the log level arguments of the twill application.
          + *
          + * @return the map of the log level arguments of the twill application
          + */
          + @Override
          + public Map<String, Map<String, LogEntry.Level>> getLogLevels()

          { + return logLevels; + }

          +
          + /**
          + * Save the log levels for the twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevels map of the log level arguments to be saved.
          + */
          + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels);
          + if (!this.logLevels.containsKey(runnableName))

          { + this.logLevels.put(runnableName, newlogLevels); + }

          else

          { + this.logLevels.get(runnableName).putAll(newlogLevels); + }

          + }
          +
          + /**
          + * Replace the resources with new log levels for the specified twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevels map of the log level arguments to be updated.
          + */
          + public void updateLogLevelsRunnable(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + if (usedResources.containsKey(runnableName)) {
          — End diff –

          Same question here, will there be concurrent calls to this method?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83089307 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -149,6 +161,51 @@ public void setServices(List<String> services) { return services.get(); } + /** + * Get the map of the log level arguments of the twill application. + * + * @return the map of the log level arguments of the twill application + */ + @Override + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; + } + + /** + * Save the log levels for the twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevels map of the log level arguments to be saved. + */ + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels); + if (!this.logLevels.containsKey(runnableName)) { + this.logLevels.put(runnableName, newlogLevels); + } else { + this.logLevels.get(runnableName).putAll(newlogLevels); + } + } + + /** + * Replace the resources with new log levels for the specified twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevels map of the log level arguments to be updated. + */ + public void updateLogLevelsRunnable(String runnableName, Map<String, LogEntry.Level> logLevels) { + if (usedResources.containsKey(runnableName)) { — End diff – Same question here, will there be concurrent calls to this method?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83089162

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -149,6 +161,51 @@ public void setServices(List<String> services)

          { return services.get(); }

          + /**
          + * Get the map of the log level arguments of the twill application.
          + *
          + * @return the map of the log level arguments of the twill application
          + */
          + @Override
          + public Map<String, Map<String, LogEntry.Level>> getLogLevels()

          { + return logLevels; + }

          +
          + /**
          + * Save the log levels for the twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevels map of the log level arguments to be saved.
          + */
          + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels);
          — End diff –

          So I assume there will be non-concurrent calls to this method, right? Otherwise one will overwrite each other. Also, if that's the case, you don't need a `ConcurrentHashMap` as the value in the `logLevels` map, but rather an immutable map is sufficient.

          However, if there can be concurrent calls to this method, you'll need to merge the map, instead of replacing it.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83089162 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -149,6 +161,51 @@ public void setServices(List<String> services) { return services.get(); } + /** + * Get the map of the log level arguments of the twill application. + * + * @return the map of the log level arguments of the twill application + */ + @Override + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; + } + + /** + * Save the log levels for the twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevels map of the log level arguments to be saved. + */ + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels); — End diff – So I assume there will be non-concurrent calls to this method, right? Otherwise one will overwrite each other. Also, if that's the case, you don't need a `ConcurrentHashMap` as the value in the `logLevels` map, but rather an immutable map is sufficient. However, if there can be concurrent calls to this method, you'll need to merge the map, instead of replacing it.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83080737

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java —
          @@ -59,6 +61,10 @@
          /**

          • @return the enabled log level for the container where the runnable is running in.
            */
          • LogEntry.Level getLogLevel();
            + LogEntry.Level getRootLogLevel();
              • End diff –

          This is a backward incompatible change. It's better to have both method, and add a `@deprecated` to the old one.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83080737 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java — @@ -59,6 +61,10 @@ /** @return the enabled log level for the container where the runnable is running in. */ LogEntry.Level getLogLevel(); + LogEntry.Level getRootLogLevel(); End diff – This is a backward incompatible change. It's better to have both method, and add a `@deprecated` to the old one.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83108634

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -40,11 +41,13 @@
          private final int reservedMemory;
          private final String rmSchedulerAddr;
          private final LogEntry.Level logLevel;
          + private final Map<String, Map<String, LogEntry.Level>> logLevels;

          public TwillRuntimeSpecification(TwillSpecification twillSpecification, String fsUser, URI twillAppDir,
          String zkConnectStr, RunId twillRunId, String twillAppName,
          int reservedMemory, @Nullable String rmSchedulerAddr,

          • @Nullable LogEntry.Level logLevel) {
            + @Nullable LogEntry.Level logLevel,
              • End diff –

          better rename this as `rootLogLevel`, just like in other classes. Also rename the getter.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83108634 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -40,11 +41,13 @@ private final int reservedMemory; private final String rmSchedulerAddr; private final LogEntry.Level logLevel; + private final Map<String, Map<String, LogEntry.Level>> logLevels; public TwillRuntimeSpecification(TwillSpecification twillSpecification, String fsUser, URI twillAppDir, String zkConnectStr, RunId twillRunId, String twillAppName, int reservedMemory, @Nullable String rmSchedulerAddr, @Nullable LogEntry.Level logLevel) { + @Nullable LogEntry.Level logLevel, End diff – better rename this as `rootLogLevel`, just like in other classes. Also rename the getter.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83083453

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -30,17 +34,33 @@
          private final int memoryMB;
          private final String host;
          private final Integer debugPort;

          • private final Level logLevel;
            + private final Map<String, LogEntry.Level> logLevels;
            + private Level rootLogLevel;
            +
            + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB,
              • End diff –

          Please add javadocs.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83083453 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -30,17 +34,33 @@ private final int memoryMB; private final String host; private final Integer debugPort; private final Level logLevel; + private final Map<String, LogEntry.Level> logLevels; + private Level rootLogLevel; + + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB, End diff – Please add javadocs.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83108717

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -41,7 +43,8 @@
          private static final String MEMORY_MB = "memoryMB";
          private static final String VIRTUAL_CORES = "virtualCores";
          private static final String DEBUG_PORT = "debugPort";

          • private static final String LOG_LEVEL = "logLevel";
            + private static final String LOG_LEVEL = "rootLogLevel";
            + private static final String LOG_LEVEL_ARGUMENTS = "logLevels";
              • End diff –

          Why not call it `LOG_LEVELS`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83108717 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -41,7 +43,8 @@ private static final String MEMORY_MB = "memoryMB"; private static final String VIRTUAL_CORES = "virtualCores"; private static final String DEBUG_PORT = "debugPort"; private static final String LOG_LEVEL = "logLevel"; + private static final String LOG_LEVEL = "rootLogLevel"; + private static final String LOG_LEVEL_ARGUMENTS = "logLevels"; End diff – Why not call it `LOG_LEVELS`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83083471

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -30,17 +34,33 @@
          private final int memoryMB;
          private final String host;
          private final Integer debugPort;

          • private final Level logLevel;
            + private final Map<String, LogEntry.Level> logLevels;
            + private Level rootLogLevel;
            +
            + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB,
            + String host, Integer debugPort, Level rootLogLevel) { + this(instanceId, containerId, cores, memoryMB, host, debugPort, rootLogLevel, + new HashMap<String, Level>()); + }

            +
            + public DefaultTwillRunResources(TwillRunResources twillRunResources, LogEntry.Level rootLogLevel,

              • End diff –

          Same here.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83083471 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -30,17 +34,33 @@ private final int memoryMB; private final String host; private final Integer debugPort; private final Level logLevel; + private final Map<String, LogEntry.Level> logLevels; + private Level rootLogLevel; + + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB, + String host, Integer debugPort, Level rootLogLevel) { + this(instanceId, containerId, cores, memoryMB, host, debugPort, rootLogLevel, + new HashMap<String, Level>()); + } + + public DefaultTwillRunResources(TwillRunResources twillRunResources, LogEntry.Level rootLogLevel, End diff – Same here.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83111652

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,30 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.

              • End diff –

          The `String` in `Future` in `setLogLevel` and `setLogLevels` for the application is the string representation of the log levels passed in, the `String` in `Future` for `setLogLevels` for a runnable is the runnableName.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83111652 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,30 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. End diff – The `String` in `Future` in `setLogLevel` and `setLogLevels` for the application is the string representation of the log levels passed in, the `String` in `Future` for `setLogLevels` for a runnable is the runnableName.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117482

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -149,6 +161,51 @@ public void setServices(List<String> services)

          { return services.get(); }

          + /**
          + * Get the map of the log level arguments of the twill application.
          + *
          + * @return the map of the log level arguments of the twill application
          + */
          + @Override
          + public Map<String, Map<String, LogEntry.Level>> getLogLevels()

          { + return logLevels; + }

          +
          + /**
          + * Save the log levels for the twill runnable.
          + *
          + * @param runnableName name of the runnable.
          + * @param logLevels map of the log level arguments to be saved.
          + */
          + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels);
          — End diff –

          Yes. There will be no concurrent calls since this method is called in the wrap of `ContainerLock`. But immutable map cannot be used since we need to put values to the map if there is update

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117482 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -149,6 +161,51 @@ public void setServices(List<String> services) { return services.get(); } + /** + * Get the map of the log level arguments of the twill application. + * + * @return the map of the log level arguments of the twill application + */ + @Override + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; + } + + /** + * Save the log levels for the twill runnable. + * + * @param runnableName name of the runnable. + * @param logLevels map of the log level arguments to be saved. + */ + public void setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, LogEntry.Level> newlogLevels = new ConcurrentHashMap<>(logLevels); — End diff – Yes. There will be no concurrent calls since this method is called in the wrap of `ContainerLock`. But immutable map cannot be used since we need to put values to the map if there is update
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117233

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java —
          @@ -951,4 +958,34 @@ public void run() {
          }
          };
          }
          +
          + /**
          + * Attempt to change the log level from a runnable or all runnables.
          + *
          + * @return

          {@code true}

          if the message requests changing log levels and

          {@code false}

          otherwise.
          + */
          + private boolean handleLogLevelChange(final Message message, final Runnable completion) {
          + Message.Scope scope = message.getScope();
          + if (message.getType() != Message.Type.SYSTEM ||
          + (scope != Message.Scope.RUNNABLE && scope != Message.Scope.ALL_RUNNABLE))

          { + return false; + }
          +
          + Command command = message.getCommand();
          + if (!command.getCommand().equals(SystemMessages.LOG_LEVEL)) { + return false; + }

          +
          + if (scope == Message.Scope.ALL_RUNNABLE)

          { + runningContainers.sendToAll(message, completion); + }

          else {
          + final String runnableName = message.getRunnableName();
          + if (runnableName == null || runnableName.isEmpty() || !twillSpec.getRunnables().containsKey(runnableName)) {
          — End diff –

          No need to check if `runnableName` is empty.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117233 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java — @@ -951,4 +958,34 @@ public void run() { } }; } + + /** + * Attempt to change the log level from a runnable or all runnables. + * + * @return {@code true} if the message requests changing log levels and {@code false} otherwise. + */ + private boolean handleLogLevelChange(final Message message, final Runnable completion) { + Message.Scope scope = message.getScope(); + if (message.getType() != Message.Type.SYSTEM || + (scope != Message.Scope.RUNNABLE && scope != Message.Scope.ALL_RUNNABLE)) { + return false; + } + + Command command = message.getCommand(); + if (!command.getCommand().equals(SystemMessages.LOG_LEVEL)) { + return false; + } + + if (scope == Message.Scope.ALL_RUNNABLE) { + runningContainers.sendToAll(message, completion); + } else { + final String runnableName = message.getRunnableName(); + if (runnableName == null || runnableName.isEmpty() || !twillSpec.getRunnables().containsKey(runnableName)) { — End diff – No need to check if `runnableName` is empty.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83109907

          — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java —
          @@ -68,6 +73,31 @@ public static Message updateRunnablesInstances(Command updateCommand)

          { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); }

          + /**
          + * Helper method to get System

          {@link Message} for changing the log levels for all runnables.
          + *
          + * @param logLevels The log levels to be changed.
          + * @return An instance of System {@link Message}

          to change the log levels.
          + */
          + public static Message setLogLevels(Map<String, LogEntry.Level> logLevels)

          { + return setLogLevels(null, logLevels); + }

          +
          + /**
          + * Helper method to get System

          {@link Message} for changing the log levels for one or all runnables.
          + *
          + * @param runnableName The name of the runnable to set the log level, ALL if apply to all runnables.
          + * @param logLevels The log levels to be changed.
          + * @return An instance of System {@link Message}

          to change the log levels.
          + */
          + public static Message setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          — End diff –

          Seems like `runnableName` is nullable. Please add the `@Nullable` annotation.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83109907 — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java — @@ -68,6 +73,31 @@ public static Message updateRunnablesInstances(Command updateCommand) { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); } + /** + * Helper method to get System {@link Message} for changing the log levels for all runnables. + * + * @param logLevels The log levels to be changed. + * @return An instance of System {@link Message} to change the log levels. + */ + public static Message setLogLevels(Map<String, LogEntry.Level> logLevels) { + return setLogLevels(null, logLevels); + } + + /** + * Helper method to get System {@link Message} for changing the log levels for one or all runnables. + * + * @param runnableName The name of the runnable to set the log level, ALL if apply to all runnables. + * @param logLevels The log levels to be changed. + * @return An instance of System {@link Message} to change the log levels. + */ + public static Message setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { — End diff – Seems like `runnableName` is nullable. Please add the `@Nullable` annotation.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83109946

          — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java —
          @@ -68,6 +73,31 @@ public static Message updateRunnablesInstances(Command updateCommand)

          { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); }

          + /**
          + * Helper method to get System

          {@link Message} for changing the log levels for all runnables.
          + *
          + * @param logLevels The log levels to be changed.
          + * @return An instance of System {@link Message}

          to change the log levels.
          + */
          + public static Message setLogLevels(Map<String, LogEntry.Level> logLevels)

          { + return setLogLevels(null, logLevels); + }

          +
          + /**
          + * Helper method to get System

          {@link Message} for changing the log levels for one or all runnables.
          + *
          + * @param runnableName The name of the runnable to set the log level, ALL if apply to all runnables.
          + * @param logLevels The log levels to be changed.
          + * @return An instance of System {@link Message}

          to change the log levels.
          + */
          + public static Message setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          — End diff –

          Also document what does null mean.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83109946 — Diff: twill-core/src/main/java/org/apache/twill/internal/state/SystemMessages.java — @@ -68,6 +73,31 @@ public static Message updateRunnablesInstances(Command updateCommand) { return new SimpleMessage(Message.Type.SYSTEM, Message.Scope.RUNNABLES, null, updateCommand); } + /** + * Helper method to get System {@link Message} for changing the log levels for all runnables. + * + * @param logLevels The log levels to be changed. + * @return An instance of System {@link Message} to change the log levels. + */ + public static Message setLogLevels(Map<String, LogEntry.Level> logLevels) { + return setLogLevels(null, logLevels); + } + + /** + * Helper method to get System {@link Message} for changing the log levels for one or all runnables. + * + * @param runnableName The name of the runnable to set the log level, ALL if apply to all runnables. + * @param logLevels The log levels to be changed. + * @return An instance of System {@link Message} to change the log levels. + */ + public static Message setLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { — End diff – Also document what does null mean.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117760

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc
          try {
          int instanceId = getStartInstanceId(runnableName);
          RunId runId = getRunId(runnableName, instanceId);
          + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ?
          + resourceReport.getLogLevels().get(runnableName) : new HashMap<String, LogEntry.Level>();
          — End diff –

          Just call `get` without calling `containsKey` and check for `null`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117760 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc try { int instanceId = getStartInstanceId(runnableName); RunId runId = getRunId(runnableName, instanceId); + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ? + resourceReport.getLogLevels().get(runnableName) : new HashMap<String, LogEntry.Level>(); — End diff – Just call `get` without calling `containsKey` and check for `null`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83112064

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,56 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +
          +import java.util.Map;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) {
          — End diff –

          No need to have this method. Just do a

          ```Maps.transformValues(logLevels, Functions.toStringFunction())```

          when needed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83112064 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; + +import java.util.Map; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) { — End diff – No need to have this method. Just do a ```Maps.transformValues(logLevels, Functions.toStringFunction())``` when needed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83118218

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc
          try {
          int instanceId = getStartInstanceId(runnableName);
          RunId runId = getRunId(runnableName, instanceId);
          + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ?
          + resourceReport.getLogLevels().get(runnableName) : new HashMap<String, LogEntry.Level>();
          + logLevel = logLevelsRunnable.containsKey(Logger.ROOT_LOGGER_NAME) ?
          — End diff –

          Is `ROOT_LOGGER_NAME` a special logger name? If that's the case, we should be consistently using the map only. I see some other classes having both `rootLogLevel` and the map.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83118218 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc try { int instanceId = getStartInstanceId(runnableName); RunId runId = getRunId(runnableName, instanceId); + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ? + resourceReport.getLogLevels().get(runnableName) : new HashMap<String, LogEntry.Level>(); + logLevel = logLevelsRunnable.containsKey(Logger.ROOT_LOGGER_NAME) ? — End diff – Is `ROOT_LOGGER_NAME` a special logger name? If that's the case, we should be consistently using the map only. I see some other classes having both `rootLogLevel` and the map.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83116829

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java —
          @@ -951,4 +958,34 @@ public void run() {
          }
          };
          }
          +
          + /**
          + * Attempt to change the log level from a runnable or all runnables.
          + *
          + * @return

          {@code true}

          if the message requests changing log levels and

          {@code false}

          otherwise.
          + */
          + private boolean handleLogLevelChange(final Message message, final Runnable completion) {
          — End diff –

          Seems like the `final` is unnecessary.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83116829 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java — @@ -951,4 +958,34 @@ public void run() { } }; } + + /** + * Attempt to change the log level from a runnable or all runnables. + * + * @return {@code true} if the message requests changing log levels and {@code false} otherwise. + */ + private boolean handleLogLevelChange(final Message message, final Runnable completion) { — End diff – Seems like the `final` is unnecessary.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83116780

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java —
          @@ -200,7 +201,9 @@ private RunningContainers initRunningContainers(ContainerId appMasterContainerId
          Integer.parseInt(System.getenv(EnvKeys.YARN_CONTAINER_MEMORY_MB)),
          appMasterHost, null, null);
          String appId = appMasterContainerId.getApplicationAttemptId().getApplicationId().toString();

          • return new RunningContainers(appId, appMasterResources, zkClient);
            + return new RunningContainers(appId, appMasterResources, zkClient,
            + twillRuntimeSpec.getLogLevels(), applicationLocation,
            + new ArrayList<>(twillSpec.getRunnables().keySet()));
              • End diff –

          No need to copy since twill spec is immutable.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83116780 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/ApplicationMasterService.java — @@ -200,7 +201,9 @@ private RunningContainers initRunningContainers(ContainerId appMasterContainerId Integer.parseInt(System.getenv(EnvKeys.YARN_CONTAINER_MEMORY_MB)), appMasterHost, null, null); String appId = appMasterContainerId.getApplicationAttemptId().getApplicationId().toString(); return new RunningContainers(appId, appMasterResources, zkClient); + return new RunningContainers(appId, appMasterResources, zkClient, + twillRuntimeSpec.getLogLevels(), applicationLocation, + new ArrayList<>(twillSpec.getRunnables().keySet())); End diff – No need to copy since twill spec is immutable.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117566

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -98,17 +109,23 @@ public Integer apply(BitSet input) {
          private final Condition containerChange;
          private final ZKClient zkClient;
          private final Multimap<String, ContainerInfo> containerStats;
          + private final Location applicationLocation;
          + private final List<String> allRunnableNames;
          — End diff –

          Just call it `runnableNames`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117566 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -98,17 +109,23 @@ public Integer apply(BitSet input) { private final Condition containerChange; private final ZKClient zkClient; private final Multimap<String, ContainerInfo> containerStats; + private final Location applicationLocation; + private final List<String> allRunnableNames; — End diff – Just call it `runnableNames`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117872

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc
          try {
          int instanceId = getStartInstanceId(runnableName);
          RunId runId = getRunId(runnableName, instanceId);
          + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ?
          — End diff –

          The variable should be called `runnableLogLevels`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117872 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -132,16 +149,22 @@ void start(String runnableName, ContainerInfo containerInfo, TwillContainerLaunc try { int instanceId = getStartInstanceId(runnableName); RunId runId = getRunId(runnableName, instanceId); + Map<String, LogEntry.Level> logLevelsRunnable = resourceReport.getLogLevels().containsKey(runnableName) ? — End diff – The variable should be called `runnableLogLevels`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83117619

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -98,17 +109,23 @@ public Integer apply(BitSet input) {
          private final Condition containerChange;
          private final ZKClient zkClient;
          private final Multimap<String, ContainerInfo> containerStats;
          + private final Location applicationLocation;
          + private final List<String> allRunnableNames;

          • RunningContainers(String appId, TwillRunResources appMasterResources, ZKClient zookeeperClient) {
            + RunningContainers(String appId, TwillRunResources appMasterResources, ZKClient zookeeperClient,
            + Map<String, Map<String, LogEntry.Level>> logLevels, Location appLocation,
            + List<String> allRunnables) {
            containers = HashBasedTable.create();
            runnableInstances = Maps.newHashMap();
            completedContainerCount = Maps.newHashMap();
            startSequence = Lists.newLinkedList();
            containerLock = new ReentrantLock();
            containerChange = containerLock.newCondition();
          • resourceReport = new DefaultResourceReport(appId, appMasterResources);
            + resourceReport = new DefaultResourceReport(appId, appMasterResources, logLevels);
            zkClient = zookeeperClient;
            containerStats = HashMultimap.create();
            + applicationLocation = appLocation;
            + allRunnableNames = allRunnables;
              • End diff –

          Better have the argument and the field named the same for clarity.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83117619 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -98,17 +109,23 @@ public Integer apply(BitSet input) { private final Condition containerChange; private final ZKClient zkClient; private final Multimap<String, ContainerInfo> containerStats; + private final Location applicationLocation; + private final List<String> allRunnableNames; RunningContainers(String appId, TwillRunResources appMasterResources, ZKClient zookeeperClient) { + RunningContainers(String appId, TwillRunResources appMasterResources, ZKClient zookeeperClient, + Map<String, Map<String, LogEntry.Level>> logLevels, Location appLocation, + List<String> allRunnables) { containers = HashBasedTable.create(); runnableInstances = Maps.newHashMap(); completedContainerCount = Maps.newHashMap(); startSequence = Lists.newLinkedList(); containerLock = new ReentrantLock(); containerChange = containerLock.newCondition(); resourceReport = new DefaultResourceReport(appId, appMasterResources); + resourceReport = new DefaultResourceReport(appId, appMasterResources, logLevels); zkClient = zookeeperClient; containerStats = HashMultimap.create(); + applicationLocation = appLocation; + allRunnableNames = allRunnables; End diff – Better have the argument and the field named the same for clarity.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83112357

          — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java —
          @@ -0,0 +1,56 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +package org.apache.twill.internal.utils;
          +
          +import com.google.common.collect.Maps;
          +import org.apache.twill.api.logging.LogEntry;
          +
          +import java.util.Map;
          +
          +/**
          + * Collections of helper functions for log level change.
          + */
          +public final class LogLevelUtil {
          +
          + /**
          + * Convert the log level argument value type to string.
          + */
          + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) {
          + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, LogEntry.Level, String>() {
          + @Override
          + public String transformEntry(String loggerName, LogEntry.Level level)

          { + return level.name(); + }

          + });
          + }
          +
          + /**
          + * Convert the log level argument type to LogEntry.Level
          + */
          + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevels) {
          — End diff –

          Seems like the only usage is in `RunningContainers`. Move it there as a private method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83112357 — Diff: twill-core/src/main/java/org/apache/twill/internal/utils/LogLevelUtil.java — @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.twill.internal.utils; + +import com.google.common.collect.Maps; +import org.apache.twill.api.logging.LogEntry; + +import java.util.Map; + +/** + * Collections of helper functions for log level change. + */ +public final class LogLevelUtil { + + /** + * Convert the log level argument value type to string. + */ + public static Map<String, String> convertLogLevelValuesToString(Map<String, LogEntry.Level> logLevels) { + return Maps.transformEntries(logLevels, new Maps.EntryTransformer<String, LogEntry.Level, String>() { + @Override + public String transformEntry(String loggerName, LogEntry.Level level) { + return level.name(); + } + }); + } + + /** + * Convert the log level argument type to LogEntry.Level + */ + public static Map<String, LogEntry.Level> convertLogLevelValuesToLogEntry(Map<String, String> logLevels) { — End diff – Seems like the only usage is in `RunningContainers`. Move it there as a private method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83471666

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,34 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the

            {@link Map}

            of the log level as a result with ROOT as the logger name and the log level as value.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel);

              • End diff –

          Also, for the `Future`, it seems like it's better just to carry `LogEntry.Level`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83471666 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,34 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the {@link Map} of the log level as a result with ROOT as the logger name and the log level as value. + */ + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel); End diff – Also, for the `Future`, it seems like it's better just to carry `LogEntry.Level`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83471254

          — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java —
          @@ -61,4 +63,11 @@

          • @return list of services of the application master.
            */
            List<String> getServices();
            +
            + /**
            + * Get the map of the log levels enabled for the twill application.
            + *
            + * @return the map of the log level arguments of the twill application
            + */
            + Map<String, Map<String, LogEntry.Level>> getLogLevels();
              • End diff –

          Why have this method? The log levels are already contained inside individual `TwillRunResources`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83471254 — Diff: twill-api/src/main/java/org/apache/twill/api/ResourceReport.java — @@ -61,4 +63,11 @@ @return list of services of the application master. */ List<String> getServices(); + + /** + * Get the map of the log levels enabled for the twill application. + * + * @return the map of the log level arguments of the twill application + */ + Map<String, Map<String, LogEntry.Level>> getLogLevels(); End diff – Why have this method? The log levels are already contained inside individual `TwillRunResources`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83471522

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,34 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the

            {@link Map}

            of the log level as a result with ROOT as the logger name and the log level as value.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel);

              • End diff –

          So there is no way to set the root log level for a particular runnable?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83471522 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,34 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the {@link Map} of the log level as a result with ROOT as the logger name and the log level as value. + */ + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel); End diff – So there is no way to set the root log level for a particular runnable?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83471846

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,7 +227,7 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications in all containers.
              • End diff –

          Same as above, do we have a way to set the root log level for a particular runnablE?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83471846 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,7 +227,7 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications in all containers. End diff – Same as above, do we have a way to set the root log level for a particular runnablE?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83472284

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -129,7 +152,7 @@ public String toString() {
          ", memoryMB=" + memoryMB +
          ", host='" + host + '\'' +
          ", debugPort=" + debugPort +

          • ", logLevel=" + logLevel +
            + ", rootLogLevel=" + getRootLogLevel() +
              • End diff –

          Use the `logLevels` field instead of calling the get method to be consistent.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83472284 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -129,7 +152,7 @@ public String toString() { ", memoryMB=" + memoryMB + ", host='" + host + '\'' + ", debugPort=" + debugPort + ", logLevel=" + logLevel + + ", rootLogLevel=" + getRootLogLevel() + End diff – Use the `logLevels` field instead of calling the get method to be consistent.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83472057

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java —
          @@ -59,6 +61,16 @@
          /**

          • @return the enabled log level for the container where the runnable is running in.
            */
            + @Deprecated
              • End diff –

          Why deprecating a method, also add a `@deprecated` in the javadoc section to tell what's the alternative. E.g.

          ```
          /**

          • ....
          • @deprecated Use {@link #getRootLogLevel()}

            instead.
            */

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83472057 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java — @@ -59,6 +61,16 @@ /** @return the enabled log level for the container where the runnable is running in. */ + @Deprecated End diff – Why deprecating a method, also add a `@deprecated` in the javadoc section to tell what's the alternative. E.g. ``` /** .... @deprecated Use {@link #getRootLogLevel()} instead. */
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83485638

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,34 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the

            {@link Map}

            of the log level as a result with ROOT as the logger name and the log level as value.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel);

              • End diff –

          For a particular runnable, user can use setLogLevels(runnableName, ImmutableMap.of("ROOT", loglevel)) to change the root log level.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83485638 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,34 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the {@link Map} of the log level as a result with ROOT as the logger name and the log level as value. + */ + Future<Map<String, LogEntry.Level>> setLogLevel(LogEntry.Level logLevel); End diff – For a particular runnable, user can use setLogLevels(runnableName, ImmutableMap.of("ROOT", loglevel)) to change the root log level.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83519511

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -90,8 +103,18 @@ public Integer getDebugPort() {
          }

          @Override
          + @Deprecated
          public Level getLogLevel()

          { - return logLevel; + return getRootLogLevel(); + }

          +
          + @Override
          + public Level getRootLogLevel()

          { + return getLogLevels().get(Logger.ROOT_LOGGER_NAME); + }

          +
          + public Map<String, LogEntry.Level> getLogLevels() {
          — End diff –

          Add `@Override`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83519511 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -90,8 +103,18 @@ public Integer getDebugPort() { } @Override + @Deprecated public Level getLogLevel() { - return logLevel; + return getRootLogLevel(); + } + + @Override + public Level getRootLogLevel() { + return getLogLevels().get(Logger.ROOT_LOGGER_NAME); + } + + public Map<String, LogEntry.Level> getLogLevels() { — End diff – Add `@Override`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83519594

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -39,25 +44,33 @@
          private final TwillRunResources appMasterResources;
          private final String applicationId;
          private final AtomicReference<List<String>> services;
          + private final Map<String, Map<String, LogEntry.Level>> logLevels;

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources)

          { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - }

          -

          • public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          • Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); }

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          Map<String, Collection<TwillRunResources>> resources, List<String> services) {
          this.applicationId = applicationId;
          this.appMasterResources = masterResources;

          • this.usedResources = HashMultimap.create();
            + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create());
            for (Map.Entry<String, Collection<TwillRunResources>> entry : resources.entrySet()) { this.usedResources.putAll(entry.getKey(), entry.getValue()); }

            this.services = new AtomicReference<>(services);
            + this.logLevels = new ConcurrentHashMap<>();
            + }
            +
            + public synchronized void setRunnableLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLevels) {

              • End diff –

          better name it as `updateRunnableLogLevels`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83519594 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -39,25 +44,33 @@ private final TwillRunResources appMasterResources; private final String applicationId; private final AtomicReference<List<String>> services; + private final Map<String, Map<String, LogEntry.Level>> logLevels; public DefaultResourceReport(String applicationId, TwillRunResources masterResources) { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - } - public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); } public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources, List<String> services) { this.applicationId = applicationId; this.appMasterResources = masterResources; this.usedResources = HashMultimap.create(); + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create()); for (Map.Entry<String, Collection<TwillRunResources>> entry : resources.entrySet()) { this.usedResources.putAll(entry.getKey(), entry.getValue()); } this.services = new AtomicReference<>(services); + this.logLevels = new ConcurrentHashMap<>(); + } + + public synchronized void setRunnableLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLevels) { End diff – better name it as `updateRunnableLogLevels`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83519663

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -39,25 +44,33 @@
          private final TwillRunResources appMasterResources;
          private final String applicationId;
          private final AtomicReference<List<String>> services;
          + private final Map<String, Map<String, LogEntry.Level>> logLevels;

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources)

          { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - }

          -

          • public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          • Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); }

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          Map<String, Collection<TwillRunResources>> resources, List<String> services) {
          this.applicationId = applicationId;
          this.appMasterResources = masterResources;

          • this.usedResources = HashMultimap.create();
            + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create());
            for (Map.Entry<String, Collection<TwillRunResources>> entry : resources.entrySet()) { this.usedResources.putAll(entry.getKey(), entry.getValue()); }

            this.services = new AtomicReference<>(services);
            + this.logLevels = new ConcurrentHashMap<>();
            + }
            +
            + public synchronized void setRunnableLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLevels) {

              • End diff –

          Also, maybe good to just return if the `runnableLogLevels` map is empty.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83519663 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -39,25 +44,33 @@ private final TwillRunResources appMasterResources; private final String applicationId; private final AtomicReference<List<String>> services; + private final Map<String, Map<String, LogEntry.Level>> logLevels; public DefaultResourceReport(String applicationId, TwillRunResources masterResources) { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - } - public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); } public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources, List<String> services) { this.applicationId = applicationId; this.appMasterResources = masterResources; this.usedResources = HashMultimap.create(); + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create()); for (Map.Entry<String, Collection<TwillRunResources>> entry : resources.entrySet()) { this.usedResources.putAll(entry.getKey(), entry.getValue()); } this.services = new AtomicReference<>(services); + this.logLevels = new ConcurrentHashMap<>(); + } + + public synchronized void setRunnableLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLevels) { End diff – Also, maybe good to just return if the `runnableLogLevels` map is empty.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r83522325

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ
          if (src.getDebugPort() != null)

          { json.addProperty(DEBUG_PORT, src.getDebugPort()); }
          • if (src.getLogLevel() != null) {
          • json.addProperty(LOG_LEVEL, src.getLogLevel().toString());
            + if (src.getRootLogLevel() != null) { + json.addProperty(ROOT_LOG_LEVEL, src.getRootLogLevel().name()); }

            + json.add(LOG_LEVELS, context.serialize(src.getLogLevels(),
            + new TypeToken<Map<String, String>>() { }.getType()));

              • End diff –

          The type should be `Map<String, LogEntry.Level>`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r83522325 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ if (src.getDebugPort() != null) { json.addProperty(DEBUG_PORT, src.getDebugPort()); } if (src.getLogLevel() != null) { json.addProperty(LOG_LEVEL, src.getLogLevel().toString()); + if (src.getRootLogLevel() != null) { + json.addProperty(ROOT_LOG_LEVEL, src.getRootLogLevel().name()); } + json.add(LOG_LEVELS, context.serialize(src.getLogLevels(), + new TypeToken<Map<String, String>>() { }.getType())); End diff – The type should be `Map<String, LogEntry.Level>`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275759

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType()))

          { + return null; + }
          +
          + if (!(type.getType() instanceof ParameterizedType)) { + return null; + }

          +
          + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments()[0]);
          + return (TypeAdapter<T>) collectionAdapter(gson, valueType);
          + }
          +
          + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) {
          + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType);
          + return new TypeAdapter<Collection<V>>() {
          + @Override
          + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException {
          + if (collection == null)

          { + writer.nullValue(); + return; + }

          + writer.beginArray();
          + for (V value : collection)

          { + valueAdapter.write(writer, value); + }

          + writer.endArray();
          + }
          +
          + @Override
          + public Collection<V> read(JsonReader reader) throws IOException {
          + if (reader.peek() == JsonToken.NULL)

          { + reader.nextNull(); + return null; + }

          + if (reader.peek() != JsonToken.BEGIN_ARRAY) {
          + return null;
          — End diff –

          shouldn't this throw an exception? It is a violation of the codec.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275759 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; + } + + if (!(type.getType() instanceof ParameterizedType)) { + return null; + } + + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments() [0] ); + return (TypeAdapter<T>) collectionAdapter(gson, valueType); + } + + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) { + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType); + return new TypeAdapter<Collection<V>>() { + @Override + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException { + if (collection == null) { + writer.nullValue(); + return; + } + writer.beginArray(); + for (V value : collection) { + valueAdapter.write(writer, value); + } + writer.endArray(); + } + + @Override + public Collection<V> read(JsonReader reader) throws IOException { + if (reader.peek() == JsonToken.NULL) { + reader.nextNull(); + return null; + } + if (reader.peek() != JsonToken.BEGIN_ARRAY) { + return null; — End diff – shouldn't this throw an exception? It is a violation of the codec.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86274829

          — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java —
          @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName)

          @Override
          public final ListenableFuture<Set<String>> restartInstances(Map<String,

          • ? extends Set<Integer>> runnableToInstanceIds) {
            + ? extends Set<Integer>> runnableToInstanceIds) {
              • End diff –

          I agree the formatting is odd. Better move the entire Map<...> to the next line

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86274829 — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java — @@ -137,7 +138,7 @@ public final synchronized ServiceDiscovered discoverService(String serviceName) @Override public final ListenableFuture<Set<String>> restartInstances(Map<String, ? extends Set<Integer>> runnableToInstanceIds) { + ? extends Set<Integer>> runnableToInstanceIds) { End diff – I agree the formatting is odd. Better move the entire Map<...> to the next line
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86273964

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +

              • End diff –

          It's not clear to me why these need to return a future (or anything at all). Is it to be able to communicate an error/exception in case the setting of the log level fails? So, when it returns it is guaranteed that all runnables have actually adapted the new log level?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86273964 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + End diff – It's not clear to me why these need to return a future (or anything at all). Is it to be able to communicate an error/exception in case the setting of the log level fails? So, when it returns it is guaranteed that all runnables have actually adapted the new log level?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275651

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -89,8 +91,7 @@ public String getRmSchedulerAddr()

          { return rmSchedulerAddr; }
          • @Nullable
          • public LogEntry.Level getLogLevel() {
          • return logLevel;
            + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; }
              • End diff –

          Same here, existing clients might break if you remove a method. Or is this internal?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275651 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -89,8 +91,7 @@ public String getRmSchedulerAddr() { return rmSchedulerAddr; } @Nullable public LogEntry.Level getLogLevel() { return logLevel; + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; } End diff – Same here, existing clients might break if you remove a method. Or is this internal?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86276224

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ
          if (src.getDebugPort() != null)

          { json.addProperty(DEBUG_PORT, src.getDebugPort()); }
          • if (src.getLogLevel() != null) {
          • json.addProperty(LOG_LEVEL, src.getLogLevel().toString());
            + if (src.getRootLogLevel() != null) {
              • End diff –

          and the deserialize is not reading it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86276224 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ if (src.getDebugPort() != null) { json.addProperty(DEBUG_PORT, src.getDebugPort()); } if (src.getLogLevel() != null) { json.addProperty(LOG_LEVEL, src.getLogLevel().toString()); + if (src.getRootLogLevel() != null) { End diff – and the deserialize is not reading it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86276109

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType()))

          { + return null; + }
          +
          + if (!(type.getType() instanceof ParameterizedType)) { + return null; + }

          +
          + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments()[0]);
          + return (TypeAdapter<T>) collectionAdapter(gson, valueType);
          + }
          +
          + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) {
          + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType);
          + return new TypeAdapter<Collection<V>>() {
          + @Override
          + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException {
          + if (collection == null)

          { + writer.nullValue(); + return; + }

          + writer.beginArray();
          + for (V value : collection)

          { + valueAdapter.write(writer, value); + }

          + writer.endArray();
          + }
          +
          + @Override
          + public Collection<V> read(JsonReader reader) throws IOException {
          — End diff –

          this is a @Nullable

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86276109 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; + } + + if (!(type.getType() instanceof ParameterizedType)) { + return null; + } + + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments() [0] ); + return (TypeAdapter<T>) collectionAdapter(gson, valueType); + } + + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) { + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType); + return new TypeAdapter<Collection<V>>() { + @Override + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException { + if (collection == null) { + writer.nullValue(); + return; + } + writer.beginArray(); + for (V value : collection) { + valueAdapter.write(writer, value); + } + writer.endArray(); + } + + @Override + public Collection<V> read(JsonReader reader) throws IOException { — End diff – this is a @Nullable
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86276202

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ
          if (src.getDebugPort() != null)

          { json.addProperty(DEBUG_PORT, src.getDebugPort()); }
          • if (src.getLogLevel() != null) {
          • json.addProperty(LOG_LEVEL, src.getLogLevel().toString());
            + if (src.getRootLogLevel() != null) {
              • End diff –

          is this needed? It appears to me that the root log level is already in the src.getLogLevels()

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86276202 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -55,9 +58,11 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ if (src.getDebugPort() != null) { json.addProperty(DEBUG_PORT, src.getDebugPort()); } if (src.getLogLevel() != null) { json.addProperty(LOG_LEVEL, src.getLogLevel().toString()); + if (src.getRootLogLevel() != null) { End diff – is this needed? It appears to me that the root log level is already in the src.getLogLevels()
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275966

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType())) {
          + return null;
          — End diff –

          this is an illegal argument? Or is this actually expected to happen? If so, you should annotate the method as @Nullable. If not, you should throw an exception.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275966 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; — End diff – this is an illegal argument? Or is this actually expected to happen? If so, you should annotate the method as @Nullable. If not, you should throw an exception.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86274401

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);

              • End diff –

          question about race conditions. Suppose I have an app that has N runnables, and I set them all to DEBUG because there was a problem. Now I have identified which runnable is the culprit, so I want to reset all others to WARN and the suspicious one to TRACE. It looks like, for this, I need to make two calls:
          ```
          setRootLogLevel(WARN);
          setRootLogLevel("suspiciousRunnable", TRACE);
          ```
          The suspicious runnable will be affected by both calls. Is it guaranteed that after all, it will be on TRACE? That is, the order of calls is preserved?

          Or what would be the correct way to do this?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86274401 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); End diff – question about race conditions. Suppose I have an app that has N runnables, and I set them all to DEBUG because there was a problem. Now I have identified which runnable is the culprit, so I want to reset all others to WARN and the suspicious one to TRACE. It looks like, for this, I need to make two calls: ``` setRootLogLevel(WARN); setRootLogLevel("suspiciousRunnable", TRACE); ``` The suspicious runnable will be affected by both calls. Is it guaranteed that after all, it will be on TRACE? That is, the order of calls is preserved? Or what would be the correct way to do this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275606

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -72,7 +76,7 @@ public String getZkConnectStr()

          { return zkConnectStr; }
          • public RunId getTwillRunId() {
            + public RunId getTwillAppRunId() {
              • End diff –

          For compatibility, can you keep the existing method as deprecated?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275606 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -72,7 +76,7 @@ public String getZkConnectStr() { return zkConnectStr; } public RunId getTwillRunId() { + public RunId getTwillAppRunId() { End diff – For compatibility, can you keep the existing method as deprecated?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86274149

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          does this also allow to set the root log level? Or does that have to happen in a separate setRootLogLevel() call?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86274149 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – does this also allow to set the root log level? Or does that have to happen in a separate setRootLogLevel() call?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275989

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType()))

          { + return null; + }

          +
          + if (!(type.getType() instanceof ParameterizedType)) {
          + return null;
          — End diff –

          ditto

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275989 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; + } + + if (!(type.getType() instanceof ParameterizedType)) { + return null; — End diff – ditto
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86275201

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -158,4 +174,31 @@ public String toString()

          { ", services=" + services + '}

          ';
          }
          +
          + private TwillRunResources wrapTwillRunResources(final String runnableName, TwillRunResources resources) {
          — End diff –

          can you explain what this is for? It's not obvious to me. I think it deserves some commenting.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86275201 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -158,4 +174,31 @@ public String toString() { ", services=" + services + '} '; } + + private TwillRunResources wrapTwillRunResources(final String runnableName, TwillRunResources resources) { — End diff – can you explain what this is for? It's not obvious to me. I think it deserves some commenting.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86274641

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);

              • End diff –

          Is there a way to reset the log level for a particular logger name? That is, after I have set that logger to TRACE for some time, I have found all information I need and I want it to go back to the root log level (or whatever it would do if I had never set the log level for this logger). That also means that if I change the root log level again, it will be applied to this logger.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86274641 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); End diff – Is there a way to reset the log level for a particular logger name? That is, after I have set that logger to TRACE for some time, I have found all information I need and I want it to go back to the root log level (or whatever it would do if I had never set the log level for this logger). That also means that if I change the root log level again, it will be applied to this logger.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86282520

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +

              • End diff –

          We are sending messages using ZooKeeper. The future will be completed once the message has been processed. Or if there is problem sending the message, an exception will be added as the value of the future.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86282520 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + End diff – We are sending messages using ZooKeeper. The future will be completed once the message has been processed. Or if there is problem sending the message, an exception will be added as the value of the future.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86282861

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          Yes, you can set the root log level by calling like:
          `setLogLevels(ImmutableMap.of("ROOT", TRACE))`

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86282861 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – Yes, you can set the root log level by calling like: `setLogLevels(ImmutableMap.of("ROOT", TRACE))`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86282926

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);

              • End diff –

          For this PR, we do not have that functionality. You will have to change the log level of the particular logger name again to have the log level back.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86282926 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); End diff – For this PR, we do not have that functionality. You will have to change the log level of the particular logger name again to have the log level back.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86285363

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType())) {
          + return null;
          — End diff –

          This factory is specifically for serializing`Collection<TwillRunResources>` since Gson will not catch the inner class for it. So we only need to create the TypeAdapter when we encounter `Collection<TwillRunResources>`. Actually the implementation is simulated from `TwillRuntimeSpecificationAdapter`, which create factory to deal with similar situation, https://github.com/apache/twill/blob/master/twill-core/src/main/java/org/apache/twill/internal/json/TwillRuntimeSpecificationAdapter.java#L106

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86285363 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; — End diff – This factory is specifically for serializing`Collection<TwillRunResources>` since Gson will not catch the inner class for it. So we only need to create the TypeAdapter when we encounter `Collection<TwillRunResources>`. Actually the implementation is simulated from `TwillRuntimeSpecificationAdapter`, which create factory to deal with similar situation, https://github.com/apache/twill/blob/master/twill-core/src/main/java/org/apache/twill/internal/json/TwillRuntimeSpecificationAdapter.java#L106
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86286137

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -158,4 +174,31 @@ public String toString()

          { ", services=" + services + '}

          ';
          }
          +
          + private TwillRunResources wrapTwillRunResources(final String runnableName, TwillRunResources resources) {
          — End diff –

          The `logLevels` for all running containers are stored in `ResourceReport`, and `ResourceReport` have a map containing the `TwillRunResources` for each running runnable. We want to return the correct log levels for each runnable. So we wrap the `TwillRunResources` with `ForwardingTwillRunResources`, which has a delegation and we override the `getLogLevels` methods to get the log levels from the map in `ResourceReport`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86286137 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -158,4 +174,31 @@ public String toString() { ", services=" + services + '} '; } + + private TwillRunResources wrapTwillRunResources(final String runnableName, TwillRunResources resources) { — End diff – The `logLevels` for all running containers are stored in `ResourceReport`, and `ResourceReport` have a map containing the `TwillRunResources` for each running runnable. We want to return the correct log levels for each runnable. So we wrap the `TwillRunResources` with `ForwardingTwillRunResources`, which has a delegation and we override the `getLogLevels` methods to get the log levels from the map in `ResourceReport`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86286160

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java —
          @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) {
          public ResourceReport fromJson(Reader reader)

          { return gson.fromJson(reader, ResourceReport.class); }

          +
          + /**
          + * A type adapter for serializing

          {@code Collection<TwillRunResources>}

          correctly
          + */
          + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory {
          +
          + @SuppressWarnings("unchecked")
          + @Override
          + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) {
          + if (!Collection.class.isAssignableFrom(type.getRawType()))

          { + return null; + }
          +
          + if (!(type.getType() instanceof ParameterizedType)) { + return null; + }

          +
          + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments()[0]);
          + return (TypeAdapter<T>) collectionAdapter(gson, valueType);
          + }
          +
          + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) {
          + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType);
          + return new TypeAdapter<Collection<V>>() {
          + @Override
          + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException {
          + if (collection == null)

          { + writer.nullValue(); + return; + }

          + writer.beginArray();
          + for (V value : collection)

          { + valueAdapter.write(writer, value); + }

          + writer.endArray();
          + }
          +
          + @Override
          + public Collection<V> read(JsonReader reader) throws IOException {
          — End diff –

          Added

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86286160 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/ResourceReportAdapter.java — @@ -59,4 +72,61 @@ public ResourceReport fromJson(String json) { public ResourceReport fromJson(Reader reader) { return gson.fromJson(reader, ResourceReport.class); } + + /** + * A type adapter for serializing {@code Collection<TwillRunResources>} correctly + */ + private static final class ResourceReportTypeAdapterFactory implements TypeAdapterFactory { + + @SuppressWarnings("unchecked") + @Override + public <T> TypeAdapter<T> create(Gson gson, TypeToken<T> type) { + if (!Collection.class.isAssignableFrom(type.getRawType())) { + return null; + } + + if (!(type.getType() instanceof ParameterizedType)) { + return null; + } + + TypeToken<?> valueType = TypeToken.get(((ParameterizedType) type.getType()).getActualTypeArguments() [0] ); + return (TypeAdapter<T>) collectionAdapter(gson, valueType); + } + + private <V> TypeAdapter<Collection<V>> collectionAdapter(Gson gson, TypeToken<V> valueType) { + final TypeAdapter<V> valueAdapter = gson.getAdapter(valueType); + return new TypeAdapter<Collection<V>>() { + @Override + public void write(JsonWriter writer, @Nullable Collection<V> collection) throws IOException { + if (collection == null) { + writer.nullValue(); + return; + } + writer.beginArray(); + for (V value : collection) { + valueAdapter.write(writer, value); + } + writer.endArray(); + } + + @Override + public Collection<V> read(JsonReader reader) throws IOException { — End diff – Added
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287011

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -575,18 +601,74 @@ private boolean removeContainerInfo(String containerId)

          { return false; }

          + private Map<String, Map<String, LogEntry.Level>> copyLogLevels(Map<String, Map<String, LogEntry.Level>> originalMap) {
          + Map<String, Map<String, LogEntry.Level>> result = new TreeMap<>();
          + for (Map.Entry<String, Map<String, LogEntry.Level>> entry : originalMap.entrySet()) {
          + Map<String, LogEntry.Level> value = new TreeMap<>();
          + for (Map.Entry<String, LogEntry.Level> valueEntry : entry.getValue().entrySet())

          { + value.put(valueEntry.getKey(), valueEntry.getValue()); + }

          + result.put(entry.getKey(), value);
          + }
          + return result;
          + }
          +
          + private void checkAndSetLogLevels(Message message, String runnableName) {
          + if (message.getType() != Message.Type.SYSTEM
          + || !SystemMessages.LOG_LEVEL.equals(message.getCommand().getCommand()))

          { + return; + }

          +
          + Map<String, LogEntry.Level> runnableLogLevels = convertLogLevelValuesToLogEntry(message.getCommand().getOptions());
          + if (!logLevels.containsKey(runnableName))

          { + logLevels.put(runnableName, runnableLogLevels); + }

          else

          { + logLevels.get(runnableName).putAll(runnableLogLevels); + }

          + }
          +
          + private Location saveLogLevels() {
          — End diff –

          Can you explain why the log levels have to be saved to a file? It is not clear to me.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287011 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -575,18 +601,74 @@ private boolean removeContainerInfo(String containerId) { return false; } + private Map<String, Map<String, LogEntry.Level>> copyLogLevels(Map<String, Map<String, LogEntry.Level>> originalMap) { + Map<String, Map<String, LogEntry.Level>> result = new TreeMap<>(); + for (Map.Entry<String, Map<String, LogEntry.Level>> entry : originalMap.entrySet()) { + Map<String, LogEntry.Level> value = new TreeMap<>(); + for (Map.Entry<String, LogEntry.Level> valueEntry : entry.getValue().entrySet()) { + value.put(valueEntry.getKey(), valueEntry.getValue()); + } + result.put(entry.getKey(), value); + } + return result; + } + + private void checkAndSetLogLevels(Message message, String runnableName) { + if (message.getType() != Message.Type.SYSTEM + || !SystemMessages.LOG_LEVEL.equals(message.getCommand().getCommand())) { + return; + } + + Map<String, LogEntry.Level> runnableLogLevels = convertLogLevelValuesToLogEntry(message.getCommand().getOptions()); + if (!logLevels.containsKey(runnableName)) { + logLevels.put(runnableName, runnableLogLevels); + } else { + logLevels.get(runnableName).putAll(runnableLogLevels); + } + } + + private Location saveLogLevels() { — End diff – Can you explain why the log levels have to be saved to a file? It is not clear to me.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287245

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -296,8 +298,42 @@ public TwillPreparer addSecureStore(SecureStore secureStore) {

          @Override
          public TwillPreparer setLogLevel(LogEntry.Level logLevel)

          { + return setRootLogLevel(logLevel); + }

          +
          + @Override
          + public TwillPreparer setRootLogLevel(LogEntry.Level logLevel)

          { Preconditions.checkNotNull(logLevel); this.logLevel = logLevel; + saveLogLevels(logLevel); + return this; + }

          +
          + @Override
          + public TwillPreparer setRootLogLevel(String runnableName, LogEntry.Level logLevel)

          { + setLogLevels(runnableName, ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)); + return this; + }

          +
          + @Override
          + public TwillPreparer setLogLevels(Map<String, LogEntry.Level> logLevels) {
          + Preconditions.checkNotNull(logLevels);
          + if (logLevels.containsKey(Logger.ROOT_LOGGER_NAME))

          { + this.logLevel = logLevels.get(Logger.ROOT_LOGGER_NAME); + }

          + for (String runnableName : twillSpec.getRunnables().keySet())

          { + saveLogLevels(runnableName, logLevels); + }

          + return this;
          + }
          +
          + @Override
          + public TwillPreparer setLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLeves) {
          — End diff –

          typo: runnableLogLevels

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287245 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -296,8 +298,42 @@ public TwillPreparer addSecureStore(SecureStore secureStore) { @Override public TwillPreparer setLogLevel(LogEntry.Level logLevel) { + return setRootLogLevel(logLevel); + } + + @Override + public TwillPreparer setRootLogLevel(LogEntry.Level logLevel) { Preconditions.checkNotNull(logLevel); this.logLevel = logLevel; + saveLogLevels(logLevel); + return this; + } + + @Override + public TwillPreparer setRootLogLevel(String runnableName, LogEntry.Level logLevel) { + setLogLevels(runnableName, ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)); + return this; + } + + @Override + public TwillPreparer setLogLevels(Map<String, LogEntry.Level> logLevels) { + Preconditions.checkNotNull(logLevels); + if (logLevels.containsKey(Logger.ROOT_LOGGER_NAME)) { + this.logLevel = logLevels.get(Logger.ROOT_LOGGER_NAME); + } + for (String runnableName : twillSpec.getRunnables().keySet()) { + saveLogLevels(runnableName, logLevels); + } + return this; + } + + @Override + public TwillPreparer setLogLevels(String runnableName, Map<String, LogEntry.Level> runnableLogLeves) { — End diff – typo: runnableLogLevels
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86286972

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -475,6 +496,11 @@ private void sendMessage(final String runnableName, final Message message,
          Futures.addCallback(controller.sendMessage(message), new FutureCallback<Message>() {
          @Override
          public void onSuccess(Message result) {
          + //update log level information
          — End diff –

          onSuccess here really means that the message has been sent to the runnable successfully, right? But it does not reflect the actual state of the runnable itself? It that on purpose?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86286972 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -475,6 +496,11 @@ private void sendMessage(final String runnableName, final Message message, Futures.addCallback(controller.sendMessage(message), new FutureCallback<Message>() { @Override public void onSuccess(Message result) { + //update log level information — End diff – onSuccess here really means that the message has been sent to the runnable successfully, right? But it does not reflect the actual state of the runnable itself? It that on purpose?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287193

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -123,6 +142,9 @@ protected void doStart() throws Exception {

          runnable = Instances.newInstance((Class<TwillRunnable>) runnableClass);
          runnable.initialize(context);
          + if (!setLogLevels(Maps.transformValues(logLevels, Functions.toStringFunction()))) {
          + LOG.error("LoggerFactory is not a logback LoggerContext, cannot change log level");
          — End diff –

          seems that the error logging belongs into the setLogLevels() method.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287193 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -123,6 +142,9 @@ protected void doStart() throws Exception { runnable = Instances.newInstance((Class<TwillRunnable>) runnableClass); runnable.initialize(context); + if (!setLogLevels(Maps.transformValues(logLevels, Functions.toStringFunction()))) { + LOG.error("LoggerFactory is not a logback LoggerContext, cannot change log level"); — End diff – seems that the error logging belongs into the setLogLevels() method.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287397

          — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java —
          @@ -0,0 +1,249 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.twill.yarn;
          +
          +import com.google.common.base.Stopwatch;
          +import com.google.common.collect.ImmutableMap;
          +import org.apache.twill.api.AbstractTwillRunnable;
          +import org.apache.twill.api.ResourceReport;
          +import org.apache.twill.api.TwillApplication;
          +import org.apache.twill.api.TwillController;
          +import org.apache.twill.api.TwillRunResources;
          +import org.apache.twill.api.TwillSpecification;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.api.logging.PrinterLogHandler;
          +import org.apache.twill.common.Threads;
          +import org.junit.Assert;
          +import org.junit.Test;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.PrintWriter;
          +import java.util.Map;
          +import java.util.concurrent.CountDownLatch;
          +import java.util.concurrent.TimeUnit;
          +
          +/**
          + * Test changing log level for a twill runnable.
          + */
          +public class LogLevelChangeTestRun extends BaseYarnTest {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class);
          +
          + /**
          + * Twill runnable.
          + */
          + public static final class LogLevelTestRunnable extends AbstractTwillRunnable {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class);
          +
          + private volatile Thread runThread;
          +
          + @Override
          + public void run() {
          + this.runThread = Thread.currentThread();
          +
          + // check if the initial log level is DEBUG
          + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled());
          +
          + int i = 0;
          + while (!Thread.interrupted()) {
          + if (i == 0 && !LOG.isDebugEnabled())

          { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + }
          + if (i == 1 && !LOG.isInfoEnabled()) { + // check if the log level is changed to WARN + Assert.assertTrue(LOG.isWarnEnabled()); + i++; + }
          +
          + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + }
          + }
          + }
          +
          + @Override
          + public void stop() {
          + if (runThread != null) { + runThread.interrupt(); + }
          + }
          + }
          +
          + /**
          + * Second runnable.
          + */
          + public static final class LogLevelTestSecondRunnable extends AbstractTwillRunnable {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestSecondRunnable.class);
          +
          + private volatile Thread runThread;
          +
          + @Override
          + public void run() {
          + this.runThread = Thread.currentThread();
          +
          + // check if the initial log level is DEBUG
          + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled());
          +
          + int i = 0;
          + while (!Thread.interrupted()) {
          + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + }

          + if (i == 1 && LOG.isDebugEnabled())

          { + // check if the log level is changed to TRACE + Assert.assertTrue(LOG.isTraceEnabled()); + i++; + }

          +
          + try

          { + TimeUnit.MILLISECONDS.sleep(100); + }

          catch (InterruptedException e)

          { + break; + }

          + }
          + }
          +
          + @Override
          + public void stop() {
          + if (runThread != null)

          { + runThread.interrupt(); + }

          + }
          + }
          +
          + /**
          + * A test TwillApplication to test setting log level to DEBUG.
          + */
          + public static final class LogLevelTestApplication implements TwillApplication {
          +
          + @Override
          + public TwillSpecification configure()

          { + return TwillSpecification.Builder.with() + .setName("LogLevelChangeTest") + .withRunnable() + .add(LogLevelTestRunnable.class.getSimpleName(), new LogLevelTestRunnable()).noLocalFiles() + .add(LogLevelTestSecondRunnable.class.getSimpleName(), new LogLevelTestSecondRunnable()).noLocalFiles() + .anyOrder() + .build(); + }

          +
          + }
          +
          + @Test
          + public void testChangeLogLevel() throws Exception {
          + YarnTwillRunnerService runner = getTwillRunner();
          + runner.start();
          +
          + // Set log level to DEBUG
          + TwillController controller = runner.prepare(new LogLevelTestApplication())
          + .setLogLevel(LogEntry.Level.DEBUG)
          + .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))
          + .start();
          +
          + // Lets wait until the service is running
          + final CountDownLatch running = new CountDownLatch(1);
          + controller.onRunning(new Runnable() {
          + @Override
          + public void run()

          { + running.countDown(); + }

          + }, Threads.SAME_THREAD_EXECUTOR);
          + Assert.assertTrue(running.await(200, TimeUnit.SECONDS));
          +
          + // assert that log level is DEBUG
          + waitForLogLevel(controller, LogLevelTestRunnable.class.getSimpleName(),
          — End diff –

          it seems that all you are testing here is that the log level in the resources have changed. That, however, represents the log level that was sent to the runnable, but we don't know whether it was actually applied. To truly validate, the runnable should emit log messages, and the test should validate that after changing the log level, the level of messages that actually get collected has changed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287397 — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java — @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.twill.yarn; + +import com.google.common.base.Stopwatch; +import com.google.common.collect.ImmutableMap; +import org.apache.twill.api.AbstractTwillRunnable; +import org.apache.twill.api.ResourceReport; +import org.apache.twill.api.TwillApplication; +import org.apache.twill.api.TwillController; +import org.apache.twill.api.TwillRunResources; +import org.apache.twill.api.TwillSpecification; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.api.logging.PrinterLogHandler; +import org.apache.twill.common.Threads; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.PrintWriter; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +/** + * Test changing log level for a twill runnable. + */ +public class LogLevelChangeTestRun extends BaseYarnTest { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class); + + /** + * Twill runnable. + */ + public static final class LogLevelTestRunnable extends AbstractTwillRunnable { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class); + + private volatile Thread runThread; + + @Override + public void run() { + this.runThread = Thread.currentThread(); + + // check if the initial log level is DEBUG + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled()); + + int i = 0; + while (!Thread.interrupted()) { + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + } + if (i == 1 && !LOG.isInfoEnabled()) { + // check if the log level is changed to WARN + Assert.assertTrue(LOG.isWarnEnabled()); + i++; + } + + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + } + } + } + + @Override + public void stop() { + if (runThread != null) { + runThread.interrupt(); + } + } + } + + /** + * Second runnable. + */ + public static final class LogLevelTestSecondRunnable extends AbstractTwillRunnable { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestSecondRunnable.class); + + private volatile Thread runThread; + + @Override + public void run() { + this.runThread = Thread.currentThread(); + + // check if the initial log level is DEBUG + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled()); + + int i = 0; + while (!Thread.interrupted()) { + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + } + if (i == 1 && LOG.isDebugEnabled()) { + // check if the log level is changed to TRACE + Assert.assertTrue(LOG.isTraceEnabled()); + i++; + } + + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + } + } + } + + @Override + public void stop() { + if (runThread != null) { + runThread.interrupt(); + } + } + } + + /** + * A test TwillApplication to test setting log level to DEBUG. + */ + public static final class LogLevelTestApplication implements TwillApplication { + + @Override + public TwillSpecification configure() { + return TwillSpecification.Builder.with() + .setName("LogLevelChangeTest") + .withRunnable() + .add(LogLevelTestRunnable.class.getSimpleName(), new LogLevelTestRunnable()).noLocalFiles() + .add(LogLevelTestSecondRunnable.class.getSimpleName(), new LogLevelTestSecondRunnable()).noLocalFiles() + .anyOrder() + .build(); + } + + } + + @Test + public void testChangeLogLevel() throws Exception { + YarnTwillRunnerService runner = getTwillRunner(); + runner.start(); + + // Set log level to DEBUG + TwillController controller = runner.prepare(new LogLevelTestApplication()) + .setLogLevel(LogEntry.Level.DEBUG) + .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out))) + .start(); + + // Lets wait until the service is running + final CountDownLatch running = new CountDownLatch(1); + controller.onRunning(new Runnable() { + @Override + public void run() { + running.countDown(); + } + }, Threads.SAME_THREAD_EXECUTOR); + Assert.assertTrue(running.await(200, TimeUnit.SECONDS)); + + // assert that log level is DEBUG + waitForLogLevel(controller, LogLevelTestRunnable.class.getSimpleName(), — End diff – it seems that all you are testing here is that the log level in the resources have changed. That, however, represents the log level that was sent to the runnable, but we don't know whether it was actually applied. To truly validate, the runnable should emit log messages, and the test should validate that after changing the log level, the level of messages that actually get collected has changed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287548

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +

              • End diff –

          So again, does success mean the message was successfully sent? Or successfully processed?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287548 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + End diff – So again, does success mean the message was successfully sent? Or successfully processed?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86287618

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -89,8 +91,7 @@ public String getRmSchedulerAddr()

          { return rmSchedulerAddr; }
          • @Nullable
          • public LogEntry.Level getLogLevel() {
          • return logLevel;
            + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; }
              • End diff –

          ok

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86287618 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -89,8 +91,7 @@ public String getRmSchedulerAddr() { return rmSchedulerAddr; } @Nullable public LogEntry.Level getLogLevel() { return logLevel; + public Map<String, Map<String, LogEntry.Level>> getLogLevels() { + return logLevels; } End diff – ok
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86288445

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -575,18 +601,74 @@ private boolean removeContainerInfo(String containerId)

          { return false; }

          + private Map<String, Map<String, LogEntry.Level>> copyLogLevels(Map<String, Map<String, LogEntry.Level>> originalMap) {
          + Map<String, Map<String, LogEntry.Level>> result = new TreeMap<>();
          + for (Map.Entry<String, Map<String, LogEntry.Level>> entry : originalMap.entrySet()) {
          + Map<String, LogEntry.Level> value = new TreeMap<>();
          + for (Map.Entry<String, LogEntry.Level> valueEntry : entry.getValue().entrySet())

          { + value.put(valueEntry.getKey(), valueEntry.getValue()); + }

          + result.put(entry.getKey(), value);
          + }
          + return result;
          + }
          +
          + private void checkAndSetLogLevels(Message message, String runnableName) {
          + if (message.getType() != Message.Type.SYSTEM
          + || !SystemMessages.LOG_LEVEL.equals(message.getCommand().getCommand()))

          { + return; + }

          +
          + Map<String, LogEntry.Level> runnableLogLevels = convertLogLevelValuesToLogEntry(message.getCommand().getOptions());
          + if (!logLevels.containsKey(runnableName))

          { + logLevels.put(runnableName, runnableLogLevels); + }

          else

          { + logLevels.get(runnableName).putAll(runnableLogLevels); + }

          + }
          +
          + private Location saveLogLevels() {
          — End diff –

          We save the log levels to file because we have no way to know if there is message about log level change come before the launch of some containers(this can happen when we change instance count or restart the container). So when a message about log level change come, we will have it stored in the Map in `RunningContainers`. And if there is a container launch afterwards, we will check if there is update in the log levels, and localize it for the container if necessary.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86288445 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -575,18 +601,74 @@ private boolean removeContainerInfo(String containerId) { return false; } + private Map<String, Map<String, LogEntry.Level>> copyLogLevels(Map<String, Map<String, LogEntry.Level>> originalMap) { + Map<String, Map<String, LogEntry.Level>> result = new TreeMap<>(); + for (Map.Entry<String, Map<String, LogEntry.Level>> entry : originalMap.entrySet()) { + Map<String, LogEntry.Level> value = new TreeMap<>(); + for (Map.Entry<String, LogEntry.Level> valueEntry : entry.getValue().entrySet()) { + value.put(valueEntry.getKey(), valueEntry.getValue()); + } + result.put(entry.getKey(), value); + } + return result; + } + + private void checkAndSetLogLevels(Message message, String runnableName) { + if (message.getType() != Message.Type.SYSTEM + || !SystemMessages.LOG_LEVEL.equals(message.getCommand().getCommand())) { + return; + } + + Map<String, LogEntry.Level> runnableLogLevels = convertLogLevelValuesToLogEntry(message.getCommand().getOptions()); + if (!logLevels.containsKey(runnableName)) { + logLevels.put(runnableName, runnableLogLevels); + } else { + logLevels.get(runnableName).putAll(runnableLogLevels); + } + } + + private Location saveLogLevels() { — End diff – We save the log levels to file because we have no way to know if there is message about log level change come before the launch of some containers(this can happen when we change instance count or restart the container). So when a message about log level change come, we will have it stored in the Map in `RunningContainers`. And if there is a container launch afterwards, we will check if there is update in the log levels, and localize it for the container if necessary.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86288660

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -475,6 +496,11 @@ private void sendMessage(final String runnableName, final Message message,
          Futures.addCallback(controller.sendMessage(message), new FutureCallback<Message>() {
          @Override
          public void onSuccess(Message result) {
          + //update log level information
          — End diff –

          onSuccess here means that the message has been processed by the TwillContainerService, and the log levels have been set up. If there is some error setting up the log levels, an failed future with exception will be returned to callback, so it will go to onFailure() method

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86288660 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -475,6 +496,11 @@ private void sendMessage(final String runnableName, final Message message, Futures.addCallback(controller.sendMessage(message), new FutureCallback<Message>() { @Override public void onSuccess(Message result) { + //update log level information — End diff – onSuccess here means that the message has been processed by the TwillContainerService, and the log levels have been set up. If there is some error setting up the log levels, an failed future with exception will be returned to callback, so it will go to onFailure() method
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86289119

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -72,7 +76,7 @@ public String getZkConnectStr()

          { return zkConnectStr; }
          • public RunId getTwillRunId() {
            + public RunId getTwillAppRunId() {
              • End diff –

          We don't have to since it is not in twill-api

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86289119 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -72,7 +76,7 @@ public String getZkConnectStr() { return zkConnectStr; } public RunId getTwillRunId() { + public RunId getTwillAppRunId() { End diff – We don't have to since it is not in twill-api
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r86290796

          — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java —
          @@ -0,0 +1,249 @@
          +/*
          + * Licensed to the Apache Software Foundation (ASF) under one
          + * or more contributor license agreements. See the NOTICE file
          + * distributed with this work for additional information
          + * regarding copyright ownership. The ASF licenses this file
          + * to you under the Apache License, Version 2.0 (the
          + * "License"); you may not use this file except in compliance
          + * with the License. You may obtain a copy of the License at
          + *
          + * http://www.apache.org/licenses/LICENSE-2.0
          + *
          + * Unless required by applicable law or agreed to in writing, software
          + * distributed under the License is distributed on an "AS IS" BASIS,
          + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          + * See the License for the specific language governing permissions and
          + * limitations under the License.
          + */
          +
          +package org.apache.twill.yarn;
          +
          +import com.google.common.base.Stopwatch;
          +import com.google.common.collect.ImmutableMap;
          +import org.apache.twill.api.AbstractTwillRunnable;
          +import org.apache.twill.api.ResourceReport;
          +import org.apache.twill.api.TwillApplication;
          +import org.apache.twill.api.TwillController;
          +import org.apache.twill.api.TwillRunResources;
          +import org.apache.twill.api.TwillSpecification;
          +import org.apache.twill.api.logging.LogEntry;
          +import org.apache.twill.api.logging.PrinterLogHandler;
          +import org.apache.twill.common.Threads;
          +import org.junit.Assert;
          +import org.junit.Test;
          +import org.slf4j.Logger;
          +import org.slf4j.LoggerFactory;
          +
          +import java.io.PrintWriter;
          +import java.util.Map;
          +import java.util.concurrent.CountDownLatch;
          +import java.util.concurrent.TimeUnit;
          +
          +/**
          + * Test changing log level for a twill runnable.
          + */
          +public class LogLevelChangeTestRun extends BaseYarnTest {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class);
          +
          + /**
          + * Twill runnable.
          + */
          + public static final class LogLevelTestRunnable extends AbstractTwillRunnable {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class);
          +
          + private volatile Thread runThread;
          +
          + @Override
          + public void run() {
          + this.runThread = Thread.currentThread();
          +
          + // check if the initial log level is DEBUG
          + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled());
          +
          + int i = 0;
          + while (!Thread.interrupted()) {
          + if (i == 0 && !LOG.isDebugEnabled())

          { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + }
          + if (i == 1 && !LOG.isInfoEnabled()) { + // check if the log level is changed to WARN + Assert.assertTrue(LOG.isWarnEnabled()); + i++; + }
          +
          + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + }
          + }
          + }
          +
          + @Override
          + public void stop() {
          + if (runThread != null) { + runThread.interrupt(); + }
          + }
          + }
          +
          + /**
          + * Second runnable.
          + */
          + public static final class LogLevelTestSecondRunnable extends AbstractTwillRunnable {
          + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestSecondRunnable.class);
          +
          + private volatile Thread runThread;
          +
          + @Override
          + public void run() {
          + this.runThread = Thread.currentThread();
          +
          + // check if the initial log level is DEBUG
          + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled());
          +
          + int i = 0;
          + while (!Thread.interrupted()) {
          + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + }

          + if (i == 1 && LOG.isDebugEnabled())

          { + // check if the log level is changed to TRACE + Assert.assertTrue(LOG.isTraceEnabled()); + i++; + }

          +
          + try

          { + TimeUnit.MILLISECONDS.sleep(100); + }

          catch (InterruptedException e)

          { + break; + }

          + }
          + }
          +
          + @Override
          + public void stop() {
          + if (runThread != null)

          { + runThread.interrupt(); + }

          + }
          + }
          +
          + /**
          + * A test TwillApplication to test setting log level to DEBUG.
          + */
          + public static final class LogLevelTestApplication implements TwillApplication {
          +
          + @Override
          + public TwillSpecification configure()

          { + return TwillSpecification.Builder.with() + .setName("LogLevelChangeTest") + .withRunnable() + .add(LogLevelTestRunnable.class.getSimpleName(), new LogLevelTestRunnable()).noLocalFiles() + .add(LogLevelTestSecondRunnable.class.getSimpleName(), new LogLevelTestSecondRunnable()).noLocalFiles() + .anyOrder() + .build(); + }

          +
          + }
          +
          + @Test
          + public void testChangeLogLevel() throws Exception {
          + YarnTwillRunnerService runner = getTwillRunner();
          + runner.start();
          +
          + // Set log level to DEBUG
          + TwillController controller = runner.prepare(new LogLevelTestApplication())
          + .setLogLevel(LogEntry.Level.DEBUG)
          + .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out)))
          + .start();
          +
          + // Lets wait until the service is running
          + final CountDownLatch running = new CountDownLatch(1);
          + controller.onRunning(new Runnable() {
          + @Override
          + public void run()

          { + running.countDown(); + }

          + }, Threads.SAME_THREAD_EXECUTOR);
          + Assert.assertTrue(running.await(200, TimeUnit.SECONDS));
          +
          + // assert that log level is DEBUG
          + waitForLogLevel(controller, LogLevelTestRunnable.class.getSimpleName(),
          — End diff –

          Actually I am not quite sure how to collect logs emitted by runnable in unit test. So in runnable I have checked the log level change: https://github.com/apache/twill/pull/14/files#diff-e7b3cb437b83c0f34231004947a7ff7eR61, which is simulated from the `LogLevelTestRun.class`, https://github.com/apache/twill/blob/master/twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelTestRun.java#L60

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r86290796 — Diff: twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelChangeTestRun.java — @@ -0,0 +1,249 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.twill.yarn; + +import com.google.common.base.Stopwatch; +import com.google.common.collect.ImmutableMap; +import org.apache.twill.api.AbstractTwillRunnable; +import org.apache.twill.api.ResourceReport; +import org.apache.twill.api.TwillApplication; +import org.apache.twill.api.TwillController; +import org.apache.twill.api.TwillRunResources; +import org.apache.twill.api.TwillSpecification; +import org.apache.twill.api.logging.LogEntry; +import org.apache.twill.api.logging.PrinterLogHandler; +import org.apache.twill.common.Threads; +import org.junit.Assert; +import org.junit.Test; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.PrintWriter; +import java.util.Map; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; + +/** + * Test changing log level for a twill runnable. + */ +public class LogLevelChangeTestRun extends BaseYarnTest { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.class); + + /** + * Twill runnable. + */ + public static final class LogLevelTestRunnable extends AbstractTwillRunnable { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestRunnable.class); + + private volatile Thread runThread; + + @Override + public void run() { + this.runThread = Thread.currentThread(); + + // check if the initial log level is DEBUG + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled()); + + int i = 0; + while (!Thread.interrupted()) { + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + } + if (i == 1 && !LOG.isInfoEnabled()) { + // check if the log level is changed to WARN + Assert.assertTrue(LOG.isWarnEnabled()); + i++; + } + + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + } + } + } + + @Override + public void stop() { + if (runThread != null) { + runThread.interrupt(); + } + } + } + + /** + * Second runnable. + */ + public static final class LogLevelTestSecondRunnable extends AbstractTwillRunnable { + public static final Logger LOG = LoggerFactory.getLogger(LogLevelChangeTestRun.LogLevelTestSecondRunnable.class); + + private volatile Thread runThread; + + @Override + public void run() { + this.runThread = Thread.currentThread(); + + // check if the initial log level is DEBUG + Assert.assertTrue(LOG.isDebugEnabled() && !LOG.isTraceEnabled()); + + int i = 0; + while (!Thread.interrupted()) { + if (i == 0 && !LOG.isDebugEnabled()) { + // check if the log level is changed to INFO + Assert.assertTrue(LOG.isInfoEnabled()); + i++; + } + if (i == 1 && LOG.isDebugEnabled()) { + // check if the log level is changed to TRACE + Assert.assertTrue(LOG.isTraceEnabled()); + i++; + } + + try { + TimeUnit.MILLISECONDS.sleep(100); + } catch (InterruptedException e) { + break; + } + } + } + + @Override + public void stop() { + if (runThread != null) { + runThread.interrupt(); + } + } + } + + /** + * A test TwillApplication to test setting log level to DEBUG. + */ + public static final class LogLevelTestApplication implements TwillApplication { + + @Override + public TwillSpecification configure() { + return TwillSpecification.Builder.with() + .setName("LogLevelChangeTest") + .withRunnable() + .add(LogLevelTestRunnable.class.getSimpleName(), new LogLevelTestRunnable()).noLocalFiles() + .add(LogLevelTestSecondRunnable.class.getSimpleName(), new LogLevelTestSecondRunnable()).noLocalFiles() + .anyOrder() + .build(); + } + + } + + @Test + public void testChangeLogLevel() throws Exception { + YarnTwillRunnerService runner = getTwillRunner(); + runner.start(); + + // Set log level to DEBUG + TwillController controller = runner.prepare(new LogLevelTestApplication()) + .setLogLevel(LogEntry.Level.DEBUG) + .addLogHandler(new PrinterLogHandler(new PrintWriter(System.out))) + .start(); + + // Lets wait until the service is running + final CountDownLatch running = new CountDownLatch(1); + controller.onRunning(new Runnable() { + @Override + public void run() { + running.countDown(); + } + }, Threads.SAME_THREAD_EXECUTOR); + Assert.assertTrue(running.await(200, TimeUnit.SECONDS)); + + // assert that log level is DEBUG + waitForLogLevel(controller, LogLevelTestRunnable.class.getSimpleName(), — End diff – Actually I am not quite sure how to collect logs emitted by runnable in unit test. So in runnable I have checked the log level change: https://github.com/apache/twill/pull/14/files#diff-e7b3cb437b83c0f34231004947a7ff7eR61 , which is simulated from the `LogLevelTestRun.class`, https://github.com/apache/twill/blob/master/twill-yarn/src/test/java/org/apache/twill/yarn/LogLevelTestRun.java#L60
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87237171

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          In that case, why we have the `setRootLogLevel` methods above? I don't like have two ways of doing the same task.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87237171 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – In that case, why we have the `setRootLogLevel` methods above? I don't like have two ways of doing the same task.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87236703

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,59 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevelsToDefault();

              • End diff –

          Just call it `resetLogLevels`

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87236703 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,59 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevelsToDefault(); End diff – Just call it `resetLogLevels`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87242433

          — Diff: twill-core/src/main/java/org/apache/twill/internal/ContainerLiveNodeData.java —
          @@ -43,4 +51,15 @@ public String getHost() {
          public String getDebugPort()

          { return debugPort; }

          +
          + public Map<String, LogEntry.Level> getLogLevels()

          { + return logLevels; + }

          +
          + public void updateLogLevels(Map<String, LogEntry.Level> logLevels, boolean isReset) {
          + if (isReset) {
          — End diff –

          This is not a good way overloading the responsibility of this method. Better have a separate method for resetting (and that mirrors the actual functionality being exposed through the public API).

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87242433 — Diff: twill-core/src/main/java/org/apache/twill/internal/ContainerLiveNodeData.java — @@ -43,4 +51,15 @@ public String getHost() { public String getDebugPort() { return debugPort; } + + public Map<String, LogEntry.Level> getLogLevels() { + return logLevels; + } + + public void updateLogLevels(Map<String, LogEntry.Level> logLevels, boolean isReset) { + if (isReset) { — End diff – This is not a good way overloading the responsibility of this method. Better have a separate method for resetting (and that mirrors the actual functionality being exposed through the public API).
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87244879

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/ServiceMain.java —
          @@ -264,6 +262,10 @@ protected String getLoggerLevel(Logger logger)

          { return "OFF"; }

          + protected String getLogLevels() {
          + return "";
          — End diff –

          Why we can't set the custom log levels at launch in the same programmatic way as when a system message is received?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87244879 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/ServiceMain.java — @@ -264,6 +262,10 @@ protected String getLoggerLevel(Logger logger) { return "OFF"; } + protected String getLogLevels() { + return ""; — End diff – Why we can't set the custom log levels at launch in the same programmatic way as when a system message is received?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87236796

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,59 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevelsToDefault();
            +
            + /**
            + * Reset the log levels of the given runnable to the default log level,

            {@code LogEntry.Level.INFO}

            .
            + *
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + * runnable name as the result.
            + */
            + Future<String> resetRunnableLogLevelsToDefault(String runnableName);

              • End diff –

          Same here, call it `resetLogLevels(String runnableName)` to mirror the `set` methods.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87236796 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,59 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevelsToDefault(); + + /** + * Reset the log levels of the given runnable to the default log level, {@code LogEntry.Level.INFO} . + * + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * runnable name as the result. + */ + Future<String> resetRunnableLogLevelsToDefault(String runnableName); End diff – Same here, call it `resetLogLevels(String runnableName)` to mirror the `set` methods.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87237480

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,15 +227,52 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications in all containers.
            *
          • @param logLevel the {@link LogEntry.Level} that should be set.
            * The level match the {@code Logback} levels.
            * @return This {@link TwillPreparer}.
            + * @deprecated Use {@link #setRootLogLevel(LogEntry.Level)} instead.
            */
            + @Deprecated
            TwillPreparer setLogLevel(LogEntry.Level logLevel);

            /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel the {@link LogEntry.Level}

            that should be set.
            + * The level match the

            {@code Logback}

            levels.
            + * @return This

            {@link TwillPreparer}.
            + */
            + TwillPreparer setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel the {@link LogEntry.Level} that should be set.
            + * @return This {@link TwillPreparer}

            .
            + */
            + TwillPreparer setRootLogLevel(String runnableName, LogEntry.Level logLevel);

              • End diff –

          I prefer not to introduce the `setRootLogLevel` and the deprecated `setLogLevel` replaced with call to `setLogLevel(Map)` with logger name `Logger.ROOT_LOGGER_NAME` to set root logger log level.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87237480 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,15 +227,52 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications in all containers. * @param logLevel the {@link LogEntry.Level} that should be set. * The level match the {@code Logback} levels. * @return This {@link TwillPreparer}. + * @deprecated Use {@link #setRootLogLevel(LogEntry.Level)} instead. */ + @Deprecated TwillPreparer setLogLevel(LogEntry.Level logLevel); /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel the {@link LogEntry.Level} that should be set. + * The level match the {@code Logback} levels. + * @return This {@link TwillPreparer}. + */ + TwillPreparer setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel the {@link LogEntry.Level} that should be set. + * @return This {@link TwillPreparer} . + */ + TwillPreparer setRootLogLevel(String runnableName, LogEntry.Level logLevel); End diff – I prefer not to introduce the `setRootLogLevel` and the deprecated `setLogLevel` replaced with call to `setLogLevel(Map)` with logger name `Logger.ROOT_LOGGER_NAME` to set root logger log level.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87238505

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -129,7 +153,7 @@ public String toString() {
          ", memoryMB=" + memoryMB +
          ", host='" + host + '\'' +
          ", debugPort=" + debugPort +

          • ", logLevel=" + logLevel +
            + ", rootLogLevel=" + logLevels.get(Logger.ROOT_LOGGER_NAME) +
              • End diff –

          Why only has root log level? The `toString` should includes the whole `logLevels` map.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87238505 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -129,7 +153,7 @@ public String toString() { ", memoryMB=" + memoryMB + ", host='" + host + '\'' + ", debugPort=" + debugPort + ", logLevel=" + logLevel + + ", rootLogLevel=" + logLevels.get(Logger.ROOT_LOGGER_NAME) + End diff – Why only has root log level? The `toString` should includes the whole `logLevels` map.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87244253

          — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java —
          @@ -17,12 +17,14 @@
          */
          package org.apache.twill.internal;

          +import ch.qos.logback.classic.Logger;
          — End diff –

          Unused import?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87244253 — Diff: twill-core/src/main/java/org/apache/twill/internal/TwillRuntimeSpecification.java — @@ -17,12 +17,14 @@ */ package org.apache.twill.internal; +import ch.qos.logback.classic.Logger; — End diff – Unused import?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87242632

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -41,19 +46,15 @@
          private final AtomicReference<List<String>> services;

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources)

          { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - }

          -

          • public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          • Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); }

          public DefaultResourceReport(String applicationId, TwillRunResources masterResources,
          Map<String, Collection<TwillRunResources>> resources, List<String> services) {
          this.applicationId = applicationId;
          this.appMasterResources = masterResources;

          • this.usedResources = HashMultimap.create();
            + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create());
              • End diff –

          Is this change still necessary?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87242632 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -41,19 +46,15 @@ private final AtomicReference<List<String>> services; public DefaultResourceReport(String applicationId, TwillRunResources masterResources) { - this(applicationId, masterResources, ImmutableMap.<String, Collection<TwillRunResources>>of()); - } - public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources) { - this(applicationId, masterResources, resources, ImmutableList.<String>of()); + this(applicationId, masterResources, Collections.<String, Collection<TwillRunResources>>emptyMap(), + Collections.<String>emptyList()); } public DefaultResourceReport(String applicationId, TwillRunResources masterResources, Map<String, Collection<TwillRunResources>> resources, List<String> services) { this.applicationId = applicationId; this.appMasterResources = masterResources; this.usedResources = HashMultimap.create(); + this.usedResources = Multimaps.synchronizedSetMultimap(HashMultimap.<String, TwillRunResources>create()); End diff – Is this change still necessary?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87239353

          — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java —
          @@ -170,6 +171,37 @@ public String apply(Set<String> input) {
          });
          }

          + @Override
          + public Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel)

          { + return sendMessage(SystemMessages.setLogLevels(ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)), logLevel); + }

          +
          + @Override
          + public Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel)

          { + return sendMessage(SystemMessages.setLogLevels(runnableName, ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)), + logLevel); + }

          +
          + @Override
          + public Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels)

          { + return sendMessage(SystemMessages.setLogLevels(logLevels), logLevels); + }

          +
          + @Override
          + public Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
          + Map<String, LogEntry.Level> runnableLogLevels)

          { + return sendMessage(SystemMessages.setLogLevels(runnableName, runnableLogLevels), runnableLogLevels); + }

          +
          + @Override
          + public Future<String> resetLogLevelsToDefault() {
          + return sendMessage(SystemMessages.resetLogLevels(null), "all");
          — End diff –

          This is inconsistent with the `setLogLevels` method, which doesn't require explicitly passing in `null` for all runnables case.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87239353 — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillController.java — @@ -170,6 +171,37 @@ public String apply(Set<String> input) { }); } + @Override + public Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel) { + return sendMessage(SystemMessages.setLogLevels(ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)), logLevel); + } + + @Override + public Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel) { + return sendMessage(SystemMessages.setLogLevels(runnableName, ImmutableMap.of(Logger.ROOT_LOGGER_NAME, logLevel)), + logLevel); + } + + @Override + public Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels) { + return sendMessage(SystemMessages.setLogLevels(logLevels), logLevels); + } + + @Override + public Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> runnableLogLevels) { + return sendMessage(SystemMessages.setLogLevels(runnableName, runnableLogLevels), runnableLogLevels); + } + + @Override + public Future<String> resetLogLevelsToDefault() { + return sendMessage(SystemMessages.resetLogLevels(null), "all"); — End diff – This is inconsistent with the `setLogLevels` method, which doesn't require explicitly passing in `null` for all runnables case.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87239528

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,59 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevelsToDefault();
            +
            + /**
            + * Reset the log levels of the given runnable to the default log level,

            {@code LogEntry.Level.INFO}

            .
            + *
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + * runnable name as the result.
            + */
            + Future<String> resetRunnableLogLevelsToDefault(String runnableName);

              • End diff –

          Also, this reset everything? So there is no way to reset only a subset of loggers?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87239528 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,59 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevelsToDefault(); + + /** + * Reset the log levels of the given runnable to the default log level, {@code LogEntry.Level.INFO} . + * + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * runnable name as the result. + */ + Future<String> resetRunnableLogLevelsToDefault(String runnableName); End diff – Also, this reset everything? So there is no way to reset only a subset of loggers?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87238388

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -30,17 +35,25 @@
          private final int memoryMB;
          private final String host;
          private final Integer debugPort;

          • private final Level logLevel;
            + private final Map<String, LogEntry.Level> logLevels;
            +
            + /**
            + * Constructor to create an instance of {@link DefaultTwillRunResources}

            with empty log levels.
            + */
            + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB,
            + String host, Integer debugPort) {
            + this(instanceId, containerId, cores, memoryMB, host, debugPort, new HashMap<String, Level>());

              • End diff –

          Use `Collections.<String, Level>emptyMap()` instead.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87238388 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -30,17 +35,25 @@ private final int memoryMB; private final String host; private final Integer debugPort; private final Level logLevel; + private final Map<String, LogEntry.Level> logLevels; + + /** + * Constructor to create an instance of {@link DefaultTwillRunResources} with empty log levels. + */ + public DefaultTwillRunResources(int instanceId, String containerId, int cores, int memoryMB, + String host, Integer debugPort) { + this(instanceId, containerId, cores, memoryMB, host, debugPort, new HashMap<String, Level>()); End diff – Use `Collections.<String, Level>emptyMap()` instead.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87237581

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java —
          @@ -58,7 +60,18 @@

          /**

          • @return the enabled log level for the container where the runnable is running in.
            + * @deprecated Use {@link #getRootLogLevel()}

            instead.
            */
            + @Deprecated
            LogEntry.Level getLogLevel();

          + /**
          + * @return the enabled log level for the container where the runnable is running in.
          + */
          + LogEntry.Level getRootLogLevel();
          — End diff –

          Same reason as above, don't introduce this method just specialized for root logger.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87237581 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillRunResources.java — @@ -58,7 +60,18 @@ /** @return the enabled log level for the container where the runnable is running in. + * @deprecated Use {@link #getRootLogLevel()} instead. */ + @Deprecated LogEntry.Level getLogLevel(); + /** + * @return the enabled log level for the container where the runnable is running in. + */ + LogEntry.Level getRootLogLevel(); — End diff – Same reason as above, don't introduce this method just specialized for root logger.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87242662

          — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java —
          @@ -17,17 +17,22 @@
          */
          package org.apache.twill.internal;

          +import ch.qos.logback.classic.Logger;
          import com.google.common.collect.HashMultimap;
          import com.google.common.collect.ImmutableList;
          import com.google.common.collect.ImmutableMap;
          import com.google.common.collect.Multimaps;
          import com.google.common.collect.SetMultimap;
          import org.apache.twill.api.ResourceReport;
          import org.apache.twill.api.TwillRunResources;
          +import org.apache.twill.api.logging.LogEntry;

          import java.util.Collection;
          +import java.util.Collections;
          +import java.util.HashMap;
          import java.util.List;
          import java.util.Map;
          +import java.util.concurrent.ConcurrentHashMap;
          — End diff –

          unused import?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87242662 — Diff: twill-core/src/main/java/org/apache/twill/internal/DefaultResourceReport.java — @@ -17,17 +17,22 @@ */ package org.apache.twill.internal; +import ch.qos.logback.classic.Logger; import com.google.common.collect.HashMultimap; import com.google.common.collect.ImmutableList; import com.google.common.collect.ImmutableMap; import com.google.common.collect.Multimaps; import com.google.common.collect.SetMultimap; import org.apache.twill.api.ResourceReport; import org.apache.twill.api.TwillRunResources; +import org.apache.twill.api.logging.LogEntry; import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; — End diff – unused import?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87253067

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,59 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevelsToDefault();
            +
            + /**
            + * Reset the log levels of the given runnable to the default log level,

            {@code LogEntry.Level.INFO}

            .
            + *
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + * runnable name as the result.
            + */
            + Future<String> resetRunnableLogLevelsToDefault(String runnableName);

              • End diff –

          `resetLogLevels` will reset all log levels passed in runtime except the root to null. If user want to reset log level for a particular logger, he can use `setLogLevels(ImmutableMap.of(someloggerName, null))` to set the log level since null is actually a valid log level for log back

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87253067 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,59 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevelsToDefault(); + + /** + * Reset the log levels of the given runnable to the default log level, {@code LogEntry.Level.INFO} . + * + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * runnable name as the result. + */ + Future<String> resetRunnableLogLevelsToDefault(String runnableName); End diff – `resetLogLevels` will reset all log levels passed in runtime except the root to null. If user want to reset log level for a particular logger, he can use `setLogLevels(ImmutableMap.of(someloggerName, null))` to set the log level since null is actually a valid log level for log back
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87254498

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/ServiceMain.java —
          @@ -264,6 +262,10 @@ protected String getLoggerLevel(Logger logger)

          { return "OFF"; }

          + protected String getLogLevels() {
          + return "";
          — End diff –

          For a launch, the `TwillContainerService` is constructed before the run of `doMain` method, which means the we have not called `configureLogger()` method at that time so log level setting at constructor will not work.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87254498 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/ServiceMain.java — @@ -264,6 +262,10 @@ protected String getLoggerLevel(Logger logger) { return "OFF"; } + protected String getLogLevels() { + return ""; — End diff – For a launch, the `TwillContainerService` is constructed before the run of `doMain` method, which means the we have not called `configureLogger()` method at that time so log level setting at constructor will not work.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87864069

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -90,8 +104,14 @@ public Integer getDebugPort() {
          }

          @Override
          + @Deprecated
          public Level getLogLevel() {

          • return logLevel;
            + return getLogLevels().get(Logger.ROOT_LOGGER_NAME);
              • End diff –

          So this may return `null`, right? Or can we guarantee that it won't be `null` since we always know the root log level when a container starts?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87864069 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -90,8 +104,14 @@ public Integer getDebugPort() { } @Override + @Deprecated public Level getLogLevel() { return logLevel; + return getLogLevels().get(Logger.ROOT_LOGGER_NAME); End diff – So this may return `null`, right? Or can we guarantee that it won't be `null` since we always know the root log level when a container starts?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87863869

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,15 +227,34 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications in all containers.
            *
          • @param logLevel the {@link LogEntry.Level}

            that should be set.

          • The level match the {@code Logback}

            levels.

          • @return This {@link TwillPreparer}.
            + * @deprecated Use {@link #setLogLevels(Map)} with logger name Logger.ROOT_LOGGER_NAME instead.
            */
            + @Deprecated
            TwillPreparer setLogLevel(LogEntry.Level logLevel);

            /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return This {@link TwillPreparer}

            .
            + */
            + TwillPreparer setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          Will this map allows `null` value? Unlike the dynamic case, it seems like there is no point in allowing that, which means we should validate and document it?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87863869 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,15 +227,34 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications in all containers. * @param logLevel the {@link LogEntry.Level} that should be set. The level match the {@code Logback} levels. @return This {@link TwillPreparer}. + * @deprecated Use {@link #setLogLevels(Map)} with logger name Logger.ROOT_LOGGER_NAME instead. */ + @Deprecated TwillPreparer setLogLevel(LogEntry.Level logLevel); /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return This {@link TwillPreparer} . + */ + TwillPreparer setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – Will this map allows `null` value? Unlike the dynamic case, it seems like there is no point in allowing that, which means we should validate and document it?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87874961

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -605,5 +672,14 @@ public synchronized Integer getDebugPort() {
          }
          return dynamicDebugPort;
          }
          +
          + @Override
          + public synchronized Map<String, LogEntry.Level> getLogLevels() {
          + ContainerLiveNodeData liveData = controller.getLiveNodeData();
          + if (liveData != null)

          { + return liveData.getLogLevels(); + }

          + return new HashMap<>();
          — End diff –

          return `Collections.emptyMap()`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87874961 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -605,5 +672,14 @@ public synchronized Integer getDebugPort() { } return dynamicDebugPort; } + + @Override + public synchronized Map<String, LogEntry.Level> getLogLevels() { + ContainerLiveNodeData liveData = controller.getLiveNodeData(); + if (liveData != null) { + return liveData.getLogLevels(); + } + return new HashMap<>(); — End diff – return `Collections.emptyMap()`.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87874799

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java —
          @@ -575,18 +597,63 @@ private boolean removeContainerInfo(String containerId)

          { return false; }

          + private void checkAndSetLogLevels(Message message, String runnableName) {
          + String command = message.getCommand().getCommand();
          + if (message.getType() != Message.Type.SYSTEM || !SystemMessages.LOG_LEVEL.equals(command))

          { + return; + }

          +
          + Map<String, LogEntry.Level> runnableLogLevels = copyAndConvertLogLevelsMap(message.getCommand().getOptions());
          + if (!logLevels.containsKey(runnableName))

          { + logLevels.put(runnableName, runnableLogLevels); + }

          else

          { + logLevels.get(runnableName).putAll(runnableLogLevels); + }

          + }
          +
          + private Location saveLogLevels() {
          + LOG.debug("save the log level file");
          + try {
          + Gson gson = new GsonBuilder().serializeNulls().create();
          + String jsonStr = gson.toJson(logLevels);
          + String fileName = Hashing.md5().hashString(jsonStr) + "." + Constants.Files.LOG_LEVELS;
          + Location location = applicationLocation.append(fileName);
          + if (!location.exists()) {
          + try (Writer writer = new OutputStreamWriter(location.getOutputStream(), Charsets.UTF_8)) {
          + gson.toJson(logLevels, new TypeToken<Map<String, Map<String, LogEntry.Level>>>() { }.getType(), writer);
          — End diff –

          Why need to encode again? You can just write the `jsonStr` to the file.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87874799 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/appmaster/RunningContainers.java — @@ -575,18 +597,63 @@ private boolean removeContainerInfo(String containerId) { return false; } + private void checkAndSetLogLevels(Message message, String runnableName) { + String command = message.getCommand().getCommand(); + if (message.getType() != Message.Type.SYSTEM || !SystemMessages.LOG_LEVEL.equals(command)) { + return; + } + + Map<String, LogEntry.Level> runnableLogLevels = copyAndConvertLogLevelsMap(message.getCommand().getOptions()); + if (!logLevels.containsKey(runnableName)) { + logLevels.put(runnableName, runnableLogLevels); + } else { + logLevels.get(runnableName).putAll(runnableLogLevels); + } + } + + private Location saveLogLevels() { + LOG.debug("save the log level file"); + try { + Gson gson = new GsonBuilder().serializeNulls().create(); + String jsonStr = gson.toJson(logLevels); + String fileName = Hashing.md5().hashString(jsonStr) + "." + Constants.Files.LOG_LEVELS; + Location location = applicationLocation.append(fileName); + if (!location.exists()) { + try (Writer writer = new OutputStreamWriter(location.getOutputStream(), Charsets.UTF_8)) { + gson.toJson(logLevels, new TypeToken<Map<String, Map<String, LogEntry.Level>>>() { }.getType(), writer); — End diff – Why need to encode again? You can just write the `jsonStr` to the file.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87353645

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,15 +227,34 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications in all containers.
            *
          • @param logLevel the {@link LogEntry.Level}

            that should be set.

          • The level match the {@code Logback}

            levels.

          • @return This {@link TwillPreparer}

            .
            + * @deprecated Use

            {@link #setLogLevels(Map)}

            with logger name Logger.ROOT_LOGGER_NAME instead.

              • End diff –

          Should be `

          {@link Logger#ROOT_LOGGER_NAME}

          `

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87353645 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,15 +227,34 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications in all containers. * @param logLevel the {@link LogEntry.Level} that should be set. The level match the {@code Logback} levels. @return This {@link TwillPreparer} . + * @deprecated Use {@link #setLogLevels(Map)} with logger name Logger.ROOT_LOGGER_NAME instead. End diff – Should be ` {@link Logger#ROOT_LOGGER_NAME} `
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87872004

          — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java —
          @@ -66,13 +67,14 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ
          public TwillRunResources deserialize(JsonElement json, Type typeOfT,
          JsonDeserializationContext context) throws JsonParseException {
          JsonObject jsonObj = json.getAsJsonObject();
          + Map<String, LogEntry.Level> logLevels =
          + context.deserialize(jsonObj.get("logLevels"), new TypeToken<Map<String, LogEntry.Level>>() { }.getType());
          — End diff –

          Since constants are defined, we should use those constants in deserialization as well.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87872004 — Diff: twill-core/src/main/java/org/apache/twill/internal/json/TwillRunResourcesCodec.java — @@ -66,13 +67,14 @@ public JsonElement serialize(TwillRunResources src, Type typeOfSrc, JsonSerializ public TwillRunResources deserialize(JsonElement json, Type typeOfT, JsonDeserializationContext context) throws JsonParseException { JsonObject jsonObj = json.getAsJsonObject(); + Map<String, LogEntry.Level> logLevels = + context.deserialize(jsonObj.get("logLevels"), new TypeToken<Map<String, LogEntry.Level>>() { }.getType()); — End diff – Since constants are defined, we should use those constants in deserialization as well.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87353516

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,30 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be

            {@code null}

            except for the root logger, which will reset the log level for

              • End diff –

          What happen if the caller pass in `null` value for the root logger? Will it throw exception? Or will it get ignore?

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87353516 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,30 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for End diff – What happen if the caller pass in `null` value for the root logger? Will it throw exception? Or will it get ignore?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87875415

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -56,24 +66,27 @@

          public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient,
          RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader,

          • Location applicationLocation) {
            + Location applicationLocation, Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; - this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.containerLiveNodeData = createLiveNodeData(containerInfo, + isLoggerContext() + ? logLevels : Collections.<String, LogEntry.Level>emptyMap()); this.context = context; }
          • private ContainerLiveNodeData createLiveNodeData(ContainerInfo containerInfo) {
            + private ContainerLiveNodeData createLiveNodeData(ContainerInfo containerInfo,
            + Map<String, LogEntry.Level> logLevels) {
            // if debugging is enabled, log the port and register it in service discovery.
            String debugPort = System.getProperty("twill.debug.port");
            if (debugPort != null) {
            LOG.info("JVM is listening for debugger on port {}", debugPort);
            }
            return new ContainerLiveNodeData(containerInfo.getId(),
            containerInfo.getHost().getCanonicalHostName(),
          • debugPort);
            + debugPort, logLevels);
              • End diff –

          I think we should filter out the one that has `null` value, since those are representing reset.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87875415 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -56,24 +66,27 @@ public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient, RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader, Location applicationLocation) { + Location applicationLocation, Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; - this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.containerLiveNodeData = createLiveNodeData(containerInfo, + isLoggerContext() + ? logLevels : Collections.<String, LogEntry.Level>emptyMap()); this.context = context; } private ContainerLiveNodeData createLiveNodeData(ContainerInfo containerInfo) { + private ContainerLiveNodeData createLiveNodeData(ContainerInfo containerInfo, + Map<String, LogEntry.Level> logLevels) { // if debugging is enabled, log the port and register it in service discovery. String debugPort = System.getProperty("twill.debug.port"); if (debugPort != null) { LOG.info("JVM is listening for debugger on port {}", debugPort); } return new ContainerLiveNodeData(containerInfo.getId(), containerInfo.getHost().getCanonicalHostName(), debugPort); + debugPort, logLevels); End diff – I think we should filter out the one that has `null` value, since those are representing reset.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87877029

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,30 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be

            {@code null}

            except for the root logger, which will reset the log level for

              • End diff –

          We check it using `Preconditions.checkArgument`, so an exception will be thrown for null log level for root logger.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87877029 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,30 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for End diff – We check it using `Preconditions.checkArgument`, so an exception will be thrown for null log level for root logger.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r87879825

          — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java —
          @@ -90,8 +104,14 @@ public Integer getDebugPort() {
          }

          @Override
          + @Deprecated
          public Level getLogLevel() {

          • return logLevel;
            + return getLogLevels().get(Logger.ROOT_LOGGER_NAME);
              • End diff –

          This will return `null` if the `ContainerLiveNodeData` not getting created or updated. We want it to be like that since it will reflect the actual state of the container, right?

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r87879825 — Diff: twill-api/src/main/java/org/apache/twill/internal/DefaultTwillRunResources.java — @@ -90,8 +104,14 @@ public Integer getDebugPort() { } @Override + @Deprecated public Level getLogLevel() { return logLevel; + return getLogLevels().get(Logger.ROOT_LOGGER_NAME); End diff – This will return `null` if the `ContainerLiveNodeData` not getting created or updated. We want it to be like that since it will reflect the actual state of the container, right?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hsaputra commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r89718330

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          where did you see the `setRootLogLevel` method definition?

          Show
          githubbot ASF GitHub Bot added a comment - Github user hsaputra commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r89718330 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – where did you see the `setRootLogLevel` method definition?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767209

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevels();
            +
            + /**
            + * Reset the log levels of the given runnable to the log levels when .

              • End diff –

          this sentence is incomplete. Does this mean restore the log level to what it was before the last setLogLevel()? Or before setLogLevel was called for the first time? Or to what it was in the original spec?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767209 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevels(); + + /** + * Reset the log levels of the given runnable to the log levels when . End diff – this sentence is incomplete. Does this mean restore the log level to what it was before the last setLogLevel()? Or before setLogLevel was called for the first time? Or to what it was in the original spec?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767079

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future}

            that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +

              • End diff –

          Still, the way I understand this is that it means, the message was received by the app master and passed on to the runnables. We have no idea whether the runnables have received or processed it. That can only be determined by examining the live node info for each runnable.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767079 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + End diff – Still, the way I understand this is that it means, the message was received by the app master and passed on to the runnables. We have no idea whether the runnables have received or processed it. That can only be determined by examining the live node info for each runnable.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767247

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -51,29 +63,36 @@
          private final ClassLoader classLoader;
          private final BasicTwillContext context;
          private final ContainerLiveNodeData containerLiveNodeData;
          + private final Map<String, LogEntry.Level> oldLogLevels;
          + private final Map<String, LogEntry.Level> defaultLogLevels;
          private ExecutorService commandExecutor;
          private TwillRunnable runnable;

          public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient,
          RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader,

          • Location applicationLocation) {
            + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels,
            + Map<String, LogEntry.Level> logLevels) {
            super(zkClient, runId, applicationLocation);

          this.specification = specification;
          this.classLoader = classLoader;

          • this.containerLiveNodeData = createLiveNodeData(containerInfo);
            + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels);
            + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels));
              • End diff –

          can you explain what oldLogLevels is for? My understanding is this:

          • when a log level is updated, you save previous level in oldLogLevels
          • when the log level is reset, you restore the log level from oldLogLevels
            Is that correct?

          If so, then consider this scenario:

          • root log level is INFO
          • I debug a problem, so I set the root log level to DEBUG
          • I set the log level for one logger to TRACE
          • now I found the problem.
          • I reset the log level for that particular logger
          • then I set the root log level back to INFO

          what will be the log level for that particular logger after all this? INFO or DEBUG?

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767247 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -51,29 +63,36 @@ private final ClassLoader classLoader; private final BasicTwillContext context; private final ContainerLiveNodeData containerLiveNodeData; + private final Map<String, LogEntry.Level> oldLogLevels; + private final Map<String, LogEntry.Level> defaultLogLevels; private ExecutorService commandExecutor; private TwillRunnable runnable; public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient, RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader, Location applicationLocation) { + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels, + Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels); + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels)); End diff – can you explain what oldLogLevels is for? My understanding is this: when a log level is updated, you save previous level in oldLogLevels when the log level is reset, you restore the log level from oldLogLevels Is that correct? If so, then consider this scenario: root log level is INFO I debug a problem, so I set the root log level to DEBUG I set the log level for one logger to TRACE now I found the problem. I reset the log level for that particular logger then I set the root log level back to INFO what will be the log level for that particular logger after all this? INFO or DEBUG?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767395

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevels();
            +
            + /**
            + * Reset the log levels of the given runnable to the log levels when .

              • End diff –

          Yes. The future is completed when the message is processed. We are not sure about whether the request is successful or not. Now the live node will be updated only if the log level set is completed.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767395 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevels(); + + /** + * Reset the log levels of the given runnable to the log levels when . End diff – Yes. The future is completed when the message is processed. We are not sure about whether the request is successful or not. Now the live node will be updated only if the log level set is completed.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767420

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevels();
            +
            + /**
            + * Reset the log levels of the given runnable to the log levels when .

              • End diff –

          My bad for not updating the comments. The `resetLogLevels` will reset the log level of a runnable to the time when it starts up. This means it will reset the log level to what `logback.xml` is + the log level requested from `TwillPreparer`

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767420 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevels(); + + /** + * Reset the log levels of the given runnable to the log levels when . End diff – My bad for not updating the comments. The `resetLogLevels` will reset the log level of a runnable to the time when it starts up. This means it will reset the log level to what `logback.xml` is + the log level requested from `TwillPreparer`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767535

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -51,29 +63,36 @@
          private final ClassLoader classLoader;
          private final BasicTwillContext context;
          private final ContainerLiveNodeData containerLiveNodeData;
          + private final Map<String, LogEntry.Level> oldLogLevels;
          + private final Map<String, LogEntry.Level> defaultLogLevels;
          private ExecutorService commandExecutor;
          private TwillRunnable runnable;

          public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient,
          RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader,

          • Location applicationLocation) {
            + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels,
            + Map<String, LogEntry.Level> logLevels) {
            super(zkClient, runId, applicationLocation);

          this.specification = specification;
          this.classLoader = classLoader;

          • this.containerLiveNodeData = createLiveNodeData(containerInfo);
            + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels);
            + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels));
              • End diff –

          The `defaultLogLevels` are the log levels passed with `TwillPreparer`, when resetting the log levels, we will need it to update the live node info to indicate the log levels. The `oldLogLevels` is the same as the `defaultLogLevels` when the runnable starts up. We will not update it for each log level change. It is used to save old log levels for loggers for the first time change as we do not know what is in the `logback.xml`. When reset, we reset based on this map.

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767535 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -51,29 +63,36 @@ private final ClassLoader classLoader; private final BasicTwillContext context; private final ContainerLiveNodeData containerLiveNodeData; + private final Map<String, LogEntry.Level> oldLogLevels; + private final Map<String, LogEntry.Level> defaultLogLevels; private ExecutorService commandExecutor; private TwillRunnable runnable; public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient, RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader, Location applicationLocation) { + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels, + Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels); + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels)); End diff – The `defaultLogLevels` are the log levels passed with `TwillPreparer`, when resetting the log levels, we will need it to update the live node info to indicate the log levels. The `oldLogLevels` is the same as the `defaultLogLevels` when the runnable starts up. We will not update it for each log level change. It is used to save old log levels for loggers for the first time change as we do not know what is in the `logback.xml`. When reset, we reset based on this map.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90767592

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -51,29 +63,36 @@
          private final ClassLoader classLoader;
          private final BasicTwillContext context;
          private final ContainerLiveNodeData containerLiveNodeData;
          + private final Map<String, LogEntry.Level> oldLogLevels;
          + private final Map<String, LogEntry.Level> defaultLogLevels;
          private ExecutorService commandExecutor;
          private TwillRunnable runnable;

          public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient,
          RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader,

          • Location applicationLocation) {
            + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels,
            + Map<String, LogEntry.Level> logLevels) {
            super(zkClient, runId, applicationLocation);

          this.specification = specification;
          this.classLoader = classLoader;

          • this.containerLiveNodeData = createLiveNodeData(containerInfo);
            + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels);
            + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels));
              • End diff –

          When you reset, you cannot reset the log level for a particular logger. It will reset the log levels for all loggers which are changed during runtime. So if you call reset, it will reset the log level of ROOT to INFO, and the particular logger to the log level when the runnable starts. If it was null initially, its log level will be following the ROOT log level, INFO

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90767592 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -51,29 +63,36 @@ private final ClassLoader classLoader; private final BasicTwillContext context; private final ContainerLiveNodeData containerLiveNodeData; + private final Map<String, LogEntry.Level> oldLogLevels; + private final Map<String, LogEntry.Level> defaultLogLevels; private ExecutorService commandExecutor; private TwillRunnable runnable; public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient, RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader, Location applicationLocation) { + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels, + Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels); + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels)); End diff – When you reset, you cannot reset the log level for a particular logger. It will reset the log levels for all loggers which are changed during runtime. So if you call reset, it will reset the log level of ROOT to INFO, and the particular logger to the log level when the runnable starts. If it was null initially, its log level will be following the ROOT log level, INFO
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90768827

          — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java —
          @@ -51,29 +63,36 @@
          private final ClassLoader classLoader;
          private final BasicTwillContext context;
          private final ContainerLiveNodeData containerLiveNodeData;
          + private final Map<String, LogEntry.Level> oldLogLevels;
          + private final Map<String, LogEntry.Level> defaultLogLevels;
          private ExecutorService commandExecutor;
          private TwillRunnable runnable;

          public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient,
          RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader,

          • Location applicationLocation) {
            + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels,
            + Map<String, LogEntry.Level> logLevels) {
            super(zkClient, runId, applicationLocation);

          this.specification = specification;
          this.classLoader = classLoader;

          • this.containerLiveNodeData = createLiveNodeData(containerInfo);
            + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels);
            + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels));
              • End diff –

          I see. Seems ok.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90768827 — Diff: twill-yarn/src/main/java/org/apache/twill/internal/container/TwillContainerService.java — @@ -51,29 +63,36 @@ private final ClassLoader classLoader; private final BasicTwillContext context; private final ContainerLiveNodeData containerLiveNodeData; + private final Map<String, LogEntry.Level> oldLogLevels; + private final Map<String, LogEntry.Level> defaultLogLevels; private ExecutorService commandExecutor; private TwillRunnable runnable; public TwillContainerService(BasicTwillContext context, ContainerInfo containerInfo, ZKClient zkClient, RunId runId, TwillRunnableSpecification specification, ClassLoader classLoader, Location applicationLocation) { + Location applicationLocation, Map<String, LogEntry.Level> defaultLogLevels, + Map<String, LogEntry.Level> logLevels) { super(zkClient, runId, applicationLocation); this.specification = specification; this.classLoader = classLoader; this.containerLiveNodeData = createLiveNodeData(containerInfo); + this.defaultLogLevels = ImmutableMap.copyOf(defaultLogLevels); + this.oldLogLevels = Collections.synchronizedMap(new HashMap<>(defaultLogLevels)); End diff – I see. Seems ok.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user anew commented on the issue:

          https://github.com/apache/twill/pull/14

          OK, my concerns are addressed and it LGTM. Please wait for @chtyim to do another pass before this gets merged.

          Show
          githubbot ASF GitHub Bot added a comment - Github user anew commented on the issue: https://github.com/apache/twill/pull/14 OK, my concerns are addressed and it LGTM. Please wait for @chtyim to do another pass before this gets merged.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90134323

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level,

            {@code LogEntry.Level.INFO}

            .

              • End diff –

          Is the default always `INFO`? It's better not to say that.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90134323 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO} . End diff – Is the default always `INFO`? It's better not to say that.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844855

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr
          }
          }

          + private void saveLogLevels(LogEntry.Level level) {
          + level = level == null ? LogEntry.Level.INFO : level;
          + Map<String, String> appLogLevels = new HashMap<>();
          + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name());
          + for (String runnableName : twillSpec.getRunnables().keySet())

          { + this.logLevels.put(runnableName, appLogLevels); + }

          + }
          +
          + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, String> newLevels = new HashMap<>();
          + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) {
          + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey());
          + newLevels.put(entry.getKey(), entry.getValue().name());
          + }
          + this.logLevels.get(runnableName).putAll(newLevels);
          — End diff –

          We should always create a new map instead of calling `putAll`. The reason is the exposed method is call `setLogLevels`, not `addLogLevels` or `appendLogLevels`, which implies a overwriting instead of appending.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844855 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr } } + private void saveLogLevels(LogEntry.Level level) { + level = level == null ? LogEntry.Level.INFO : level; + Map<String, String> appLogLevels = new HashMap<>(); + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name()); + for (String runnableName : twillSpec.getRunnables().keySet()) { + this.logLevels.put(runnableName, appLogLevels); + } + } + + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, String> newLevels = new HashMap<>(); + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) { + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey()); + newLevels.put(entry.getKey(), entry.getValue().name()); + } + this.logLevels.get(runnableName).putAll(newLevels); — End diff – We should always create a new map instead of calling `putAll`. The reason is the exposed method is call `setLogLevels`, not `addLogLevels` or `appendLogLevels`, which implies a overwriting instead of appending.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844834

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -151,8 +152,9 @@
          this.reservedMemory = yarnConfig.getInt(Configs.Keys.JAVA_RESERVED_MEMORY_MB,
          Configs.Defaults.JAVA_RESERVED_MEMORY_MB);
          this.extraOptions = extraOptions;

          • this.logLevel = logLevel;
            this.classAcceptor = new ClassAcceptor();
            + this.logLevel = logLevel;
              • End diff –

          Shouldn't have this field anymore. Every log level info should be in the `logLevels` map

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844834 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -151,8 +152,9 @@ this.reservedMemory = yarnConfig.getInt(Configs.Keys.JAVA_RESERVED_MEMORY_MB, Configs.Defaults.JAVA_RESERVED_MEMORY_MB); this.extraOptions = extraOptions; this.logLevel = logLevel; this.classAcceptor = new ClassAcceptor(); + this.logLevel = logLevel; End diff – Shouldn't have this field anymore. Every log level info should be in the `logLevels` map
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844862

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java —
          @@ -227,15 +227,36 @@
          TwillPreparer addSecureStore(SecureStore secureStore);

          /**

          • * Set the log level for Twill applications running in a container.
            + * Set the root log level for Twill applications in all containers.
            *
          • @param logLevel the {@link LogEntry.Level}

            that should be set.

          • The level match the {@code Logback}

            levels.

          • @return This {@link TwillPreparer}

            .
            + * @deprecated Use

            {@link #setLogLevels(Map)}

            with key

            {@link org.slf4j.Logger#ROOT_LOGGER_NAME}

            instead.
            */
            + @Deprecated
            TwillPreparer setLogLevel(LogEntry.Level logLevel);

          /**
          + * Set the log levels for requested logger names for Twill applications running in a container. The log level whose
          + * value is null will be ignored
          — End diff –

          I think the implementation actually throws `IllegalArgumentException`, which is better than silently ignored. Please update the doc here accordingly.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844862 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillPreparer.java — @@ -227,15 +227,36 @@ TwillPreparer addSecureStore(SecureStore secureStore); /** * Set the log level for Twill applications running in a container. + * Set the root log level for Twill applications in all containers. * @param logLevel the {@link LogEntry.Level} that should be set. The level match the {@code Logback} levels. @return This {@link TwillPreparer} . + * @deprecated Use {@link #setLogLevels(Map)} with key {@link org.slf4j.Logger#ROOT_LOGGER_NAME} instead. */ + @Deprecated TwillPreparer setLogLevel(LogEntry.Level logLevel); /** + * Set the log levels for requested logger names for Twill applications running in a container. The log level whose + * value is null will be ignored — End diff – I think the implementation actually throws `IllegalArgumentException`, which is better than silently ignored. Please update the doc here accordingly.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844732

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,48 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables.
            + * The log levels will be the same as when the runnables start up.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. The future result
            + * is the logger names provided in the parameter.
            + */
            + Future<String[]> resetLogLevels(String...loggerNames);
            +
            + /**
            + * Reset the log levels of the given runnable.
            + * The log levels will be same as when the runnable starts up.
            + *

              • End diff –

          Missed the `@param loggerNames` doc.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844732 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,48 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables. + * The log levels will be the same as when the runnables start up. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. The future result + * is the logger names provided in the parameter. + */ + Future<String[]> resetLogLevels(String...loggerNames); + + /** + * Reset the log levels of the given runnable. + * The log levels will be same as when the runnable starts up. + * End diff – Missed the `@param loggerNames` doc.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844825

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr
          }
          }

          + private void saveLogLevels(LogEntry.Level level) {
          + level = level == null ? LogEntry.Level.INFO : level;
          — End diff –

          Shouldn't default it to `INFO` at all. It should be treated the same as `null` is provided as the log level value as in the other `saveLogLevels` method, which will throw `IllegalArgumentException`. Also, rename the `level` parameter to `rootLogLevel` to be clear.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844825 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr } } + private void saveLogLevels(LogEntry.Level level) { + level = level == null ? LogEntry.Level.INFO : level; — End diff – Shouldn't default it to `INFO` at all. It should be treated the same as `null` is provided as the log level value as in the other `saveLogLevels` method, which will throw `IllegalArgumentException`. Also, rename the `level` parameter to `rootLogLevel` to be clear.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90134835

          — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java —
          @@ -197,16 +198,16 @@ protected final void shutDown() throws Exception {
          }
          }

          + protected final OperationFuture<?> updateLiveNode() {
          — End diff –

          Add a javadoc for protected method as it is meant to be called by the child class.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90134835 — Diff: twill-core/src/main/java/org/apache/twill/internal/AbstractTwillService.java — @@ -197,16 +198,16 @@ protected final void shutDown() throws Exception { } } + protected final OperationFuture<?> updateLiveNode() { — End diff – Add a javadoc for protected method as it is meant to be called by the child class.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r90134468

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevels();

              • End diff –

          What's the `String` inside the `Future` represents? If there is nothing actually returned from the `Future.get()`, you can specify the return type as `Future<?>` and returns `null` when `Future.get()` is called.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r90134468 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevels(); End diff – What's the `String` inside the `Future` represents? If there is nothing actually returned from the `Future.get()`, you can specify the return type as `Future<?>` and returns `null` when `Future.get()` is called.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844710

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,44 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the root log level for Twill applications in all containers.
            + *
            + * @param logLevel The log level for the root logger to change.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel);
            +
            + /**
            + * Set the root log level for a particular runnable.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevel The log level for the root logger to change.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry
            + * the log level as a result.
            + */
            + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);

              • End diff –

          @hsaputra I think it get removed after this comment, but this comment sticks.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844710 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,44 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the root log level for Twill applications in all containers. + * + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(LogEntry.Level logLevel); + + /** + * Set the root log level for a particular runnable. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevel The log level for the root logger to change. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry + * the log level as a result. + */ + Future<LogEntry.Level> setRootLogLevel(String runnableName, LogEntry.Level logLevel); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); End diff – @hsaputra I think it get removed after this comment, but this comment sticks.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91844729

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,48 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map}

            of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables.
            + * The log levels will be the same as when the runnables start up.
            + *

              • End diff –

          Missed the `@param loggerNames` doc.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91844729 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,48 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables. + * The log levels will be the same as when the runnables start up. + * End diff – Missed the `@param loggerNames` doc.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91845224

          — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java —
          @@ -89,4 +90,45 @@

          • @return A {@link Future} that will be completed when the restart operation has been done.
            */
            Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds);
            +
            + /**
            + * Set the log levels for requested logger names for Twill applications running in a container.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set.
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done. It will carry the
            + *

            {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels);
            +
            + /**
            + * Set the log levels for requested logger names for a {@link TwillRunnable}.
            + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for
            + * the specified logger.
            + *
            + * @param runnableName The name of the runnable to set the log level.
            + * @param logLevelsForRunnable The {@link Map}

            contains the requested logger name and log level that
            + * need to be changed.
            + * @return A

            {@link Future} that will be completed when the set log level operation has been done. It will carry the
            + * {@link Map} of log levels as the result.
            + */
            + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName,
            + Map<String, LogEntry.Level> logLevelsForRunnable);
            +
            + /**
            + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}.
            + *
            + * @return A {@link Future}

            that will be completed when the set log level operation has been done.
            + */
            + Future<String> resetLogLevels();

              • End diff –

          Changed the value in `Future` to be the loggerNames passed through this methods. Empty means resetting for all loggers

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91845224 — Diff: twill-api/src/main/java/org/apache/twill/api/TwillController.java — @@ -89,4 +90,45 @@ @return A {@link Future} that will be completed when the restart operation has been done. */ Future<String> restartInstances(String runnable, int instanceId, int... moreInstanceIds); + + /** + * Set the log levels for requested logger names for Twill applications running in a container. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param logLevels The {@link Map} contains the requested logger names and log levels that need to be set. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(Map<String, LogEntry.Level> logLevels); + + /** + * Set the log levels for requested logger names for a {@link TwillRunnable}. + * The log level for a logger name can be {@code null} except for the root logger, which will reset the log level for + * the specified logger. + * + * @param runnableName The name of the runnable to set the log level. + * @param logLevelsForRunnable The {@link Map} contains the requested logger name and log level that + * need to be changed. + * @return A {@link Future} that will be completed when the set log level operation has been done. It will carry the + * {@link Map} of log levels as the result. + */ + Future<Map<String, LogEntry.Level>> setLogLevels(String runnableName, + Map<String, LogEntry.Level> logLevelsForRunnable); + + /** + * Reset the log levels of all runnables to the default log level, {@code LogEntry.Level.INFO}. + * + * @return A {@link Future} that will be completed when the set log level operation has been done. + */ + Future<String> resetLogLevels(); End diff – Changed the value in `Future` to be the loggerNames passed through this methods. Empty means resetting for all loggers
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91845583

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr
          }
          }

          + private void saveLogLevels(LogEntry.Level level) {
          + level = level == null ? LogEntry.Level.INFO : level;
          + Map<String, String> appLogLevels = new HashMap<>();
          + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name());
          + for (String runnableName : twillSpec.getRunnables().keySet())

          { + this.logLevels.put(runnableName, appLogLevels); + }

          + }
          +
          + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, String> newLevels = new HashMap<>();
          + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) {
          + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey());
          + newLevels.put(entry.getKey(), entry.getValue().name());
          + }
          + this.logLevels.get(runnableName).putAll(newLevels);
          — End diff –

          Will this make the `setLogLevels` complicated for the user? For example, if user wants to change several log levels for the entire application. He also wants to set a particular logger for the runnable. He will have to have to pass in the entire map to `setLogLevels(runnableName, logLevelsMap)`. Is it good to do so?

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91845583 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr } } + private void saveLogLevels(LogEntry.Level level) { + level = level == null ? LogEntry.Level.INFO : level; + Map<String, String> appLogLevels = new HashMap<>(); + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name()); + for (String runnableName : twillSpec.getRunnables().keySet()) { + this.logLevels.put(runnableName, appLogLevels); + } + } + + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, String> newLevels = new HashMap<>(); + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) { + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey()); + newLevels.put(entry.getKey(), entry.getValue().name()); + } + this.logLevels.get(runnableName).putAll(newLevels); — End diff – Will this make the `setLogLevels` complicated for the user? For example, if user wants to change several log levels for the entire application. He also wants to set a particular logger for the runnable. He will have to have to pass in the entire map to `setLogLevels(runnableName, logLevelsMap)`. Is it good to do so?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on a diff in the pull request:

          https://github.com/apache/twill/pull/14#discussion_r91846095

          — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java —
          @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr
          }
          }

          + private void saveLogLevels(LogEntry.Level level) {
          + level = level == null ? LogEntry.Level.INFO : level;
          + Map<String, String> appLogLevels = new HashMap<>();
          + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name());
          + for (String runnableName : twillSpec.getRunnables().keySet())

          { + this.logLevels.put(runnableName, appLogLevels); + }

          + }
          +
          + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) {
          + Map<String, String> newLevels = new HashMap<>();
          + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) {
          + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey());
          + newLevels.put(entry.getKey(), entry.getValue().name());
          + }
          + this.logLevels.get(runnableName).putAll(newLevels);
          — End diff –

          What I meant was, since the method is "set", it should be setting / overwriting, not appending. And I think it's ok for the user to gather and pass in the map.

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on a diff in the pull request: https://github.com/apache/twill/pull/14#discussion_r91846095 — Diff: twill-yarn/src/main/java/org/apache/twill/yarn/YarnTwillPreparer.java — @@ -384,6 +408,24 @@ private void setEnv(String runnableName, Map<String, String> env, boolean overwr } } + private void saveLogLevels(LogEntry.Level level) { + level = level == null ? LogEntry.Level.INFO : level; + Map<String, String> appLogLevels = new HashMap<>(); + appLogLevels.put(Logger.ROOT_LOGGER_NAME, level.name()); + for (String runnableName : twillSpec.getRunnables().keySet()) { + this.logLevels.put(runnableName, appLogLevels); + } + } + + private void saveLogLevels(String runnableName, Map<String, LogEntry.Level> logLevels) { + Map<String, String> newLevels = new HashMap<>(); + for (Map.Entry<String, LogEntry.Level> entry : logLevels.entrySet()) { + Preconditions.checkArgument(entry.getValue() != null, "Log level cannot be null for logger {}", entry.getKey()); + newLevels.put(entry.getKey(), entry.getValue().name()); + } + this.logLevels.get(runnableName).putAll(newLevels); — End diff – What I meant was, since the method is "set", it should be setting / overwriting, not appending. And I think it's ok for the user to gather and pass in the map.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user chtyim commented on the issue:

          https://github.com/apache/twill/pull/14

          LGTM

          Show
          githubbot ASF GitHub Bot added a comment - Github user chtyim commented on the issue: https://github.com/apache/twill/pull/14 LGTM
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/twill/pull/14

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/twill/pull/14
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hsaputra commented on the issue:

          https://github.com/apache/twill/pull/14

          Congrats @yaojiefeng ! =)

          Show
          githubbot ASF GitHub Bot added a comment - Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/14 Congrats @yaojiefeng ! =)
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user yaojiefeng commented on the issue:

          https://github.com/apache/twill/pull/14

          Thanks! @hsaputra

          Show
          githubbot ASF GitHub Bot added a comment - Github user yaojiefeng commented on the issue: https://github.com/apache/twill/pull/14 Thanks! @hsaputra

            People

            • Assignee:
              yaojie Yaojie Feng
              Reporter:
              hsaputra Henry Saputra
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development