Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.2.0
    • Fix Version/s: v1.2.0
    • Component/s: Configuration
    • Labels:
      None
    • Environment:

      [CentOS 6.2 64-bit]

      Description

      [Marked as Major because it's highly visible in the logs, it's important to see pre-post options especially when dealing with reconfigutations during troubleshooting]
      [Tested with the sample config template]

      1. WARN msg is printed in the logs, when it shouldn't be a warning... maybe just INFO:
      2012-04-24 19:56:59,370 (conf-file-poller-0) [WARN - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:591)] Sink: loggerSink

      2. The initial (prevalidation) config should show the props that were read from the config file before validation, rather than showing an empty list.
      2012-04-24 19:56:59,353 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:284)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent]
      SOURCES: {}
      CHANNELS: {}
      SINKS: {}

      3. The final (post-validation) config should be pretty-printed, like the initial config. Currently it looks like this:
      2012-04-24 19:56:59,389 (conf-file-poller-0) [INFO - org.apache.flume.node.nodemanager.DefaultLogicalNodeManager.onNodeConfigurationChanged(DefaultLogicalNodeManager.java:54)] Node configuration change:{ sourceRunners:{seqGenSrc=PollableSourceRunner: { source:org.apache.flume.source.SequenceGeneratorSource@450e790c counterGroup:{ name:null counters:{} } }} sinkRunners:{loggerSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@392d263f counterGroup:{ name:null counters:{} } }} channels:

      {memoryChannel=org.apache.flume.channel.MemoryChannel@432a0f6c}

      }

      1. FLUME-1148.patch
        9 kB
        Hari Shreedharan

        Activity

        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4870/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Changed the way we log during configuration validation.

        This addresses bug FLUME-1148.
        https://issues.apache.org/jira/browse/FLUME-1148

        Diffs


        flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java 97ac302
        flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java 881c0fa
        flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac4d3a3
        flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 24b4e1c

        Diff: https://reviews.apache.org/r/4870/diff

        Testing
        -------

        Ran an agent with template configuration. In the following log, lines between <-- and --> were added here(they arent part of the log):

        2012-04-24 23:14:20,840 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:284)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent]
        SOURCES: {seqGenSrc={ parameters:

        {channels=memoryChannel, type=seq} }}
        CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }}
        SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }}

        <-This stuff is what we read from the conf file. It has not yet been validated, therefore we say initial configuration(this is the same as before, just format changed)->



        2012-04-24 23:14:20,846 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:438)] Created channel memoryChannel
        2012-04-24 23:14:20,860 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:617)] Creating sink: loggerSink using LOGGER

        <-The following is post validation->

        2012-04-24 23:14:20,864 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:342)] Post validation configuration for agent
        AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[agent]
        SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq}

        }}
        CHANNELS: {memoryChannel={ parameters:

        {capacity=100, type=memory}

        }}

        <--This stuff - we couldn't find config stubs for these. We did basic syntactic validation, and found them to be ok. Since no sinks belong to this category(because we could do component specific validation), there is no sinks section. If there was a sink, whcih we could do only Syntactic validation, even that would appear here -->

        AgentConfiguration created with Configuration stubs for which full validation was performed[agent]
        SINKS:

        {loggerSink=ComponentConfiguration[loggerSink] CONFIG: CHANNEL:memoryChannel }

        <--We could do a more thorough component specific validation of this stuff. Since we couldn't do component specific validation of any channels, or sources, they aren't listed. If we could, then there would be a section for that and it would be listed like SOURCES: <blah> -->

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4870/ ----------------------------------------------------------- Review request for Flume. Summary ------- Changed the way we log during configuration validation. This addresses bug FLUME-1148 . https://issues.apache.org/jira/browse/FLUME-1148 Diffs flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java 97ac302 flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java 881c0fa flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac4d3a3 flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 24b4e1c Diff: https://reviews.apache.org/r/4870/diff Testing ------- Ran an agent with template configuration. In the following log, lines between <-- and --> were added here(they arent part of the log): 2012-04-24 23:14:20,840 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:284)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration [agent] SOURCES: {seqGenSrc={ parameters: {channels=memoryChannel, type=seq} }} CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }} SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }} <- This stuff is what we read from the conf file. It has not yet been validated, therefore we say initial configuration(this is the same as before, just format changed) -> 2012-04-24 23:14:20,846 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:438)] Created channel memoryChannel 2012-04-24 23:14:20,860 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:617)] Creating sink: loggerSink using LOGGER <- The following is post validation -> 2012-04-24 23:14:20,864 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:342)] Post validation configuration for agent AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed [agent] SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }} CHANNELS: {memoryChannel={ parameters: {capacity=100, type=memory} }} <--This stuff - we couldn't find config stubs for these. We did basic syntactic validation, and found them to be ok. Since no sinks belong to this category(because we could do component specific validation), there is no sinks section. If there was a sink, whcih we could do only Syntactic validation, even that would appear here --> AgentConfiguration created with Configuration stubs for which full validation was performed [agent] SINKS: {loggerSink=ComponentConfiguration[loggerSink] CONFIG: CHANNEL:memoryChannel } <--We could do a more thorough component specific validation of this stuff. Since we couldn't do component specific validation of any channels, or sources, they aren't listed. If we could, then there would be a section for that and it would be listed like SOURCES: <blah> --> Thanks, Hari
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4870/#review7211
        -----------------------------------------------------------

        Ship it!

        +1

        • Arvind

        On 2012-04-25 06:47:11, Hari Shreedharan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4870/

        -----------------------------------------------------------

        (Updated 2012-04-25 06:47:11)

        Review request for Flume.

        Summary

        -------

        Changed the way we log during configuration validation.

        This addresses bug FLUME-1148.

        https://issues.apache.org/jira/browse/FLUME-1148

        Diffs

        -----

        flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java 97ac302

        flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java 881c0fa

        flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac4d3a3

        flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 24b4e1c

        Diff: https://reviews.apache.org/r/4870/diff

        Testing

        -------

        Ran an agent with template configuration. In the following log, lines between <-- and --> were added here(they arent part of the log):

        2012-04-24 23:14:20,840 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:284)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent]

        SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }}
        bq. CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }}
        bq. SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }}
        bq.
        bq. <-This stuff is what we read from the conf file. It has not yet been validated, therefore we say initial configuration(this is the same as before, just format changed)->
        bq.
        bq.
        bq.
        bq. 2012-04-24 23:14:20,846 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:438)] Created channel memoryChannel
        bq. 2012-04-24 23:14:20,860 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:617)] Creating sink: loggerSink using LOGGER
        bq.
        bq. <-The following is post validation->
        bq.
        bq. 2012-04-24 23:14:20,864 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:342)] Post validation configuration for agent
        bq. AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[agent]
        bq. SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }}

        CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }}

        <--This stuff - we couldn't find config stubs for these. We did basic syntactic validation, and found them to be ok. Since no sinks belong to this category(because we could do component specific validation), there is no sinks section. If there was a sink, whcih we could do only Syntactic validation, even that would appear here -->

        AgentConfiguration created with Configuration stubs for which full validation was performed[agent]

        SINKS: {loggerSink=ComponentConfiguration[loggerSink] bq. CONFIG: bq. CHANNEL:memoryChannel bq. }

        <--We could do a more thorough component specific validation of this stuff. Since we couldn't do component specific validation of any channels, or sources, they aren't listed. If we could, then there would be a section for that and it would be listed like SOURCES: <blah> -->

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4870/#review7211 ----------------------------------------------------------- Ship it! +1 Arvind On 2012-04-25 06:47:11, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4870/ ----------------------------------------------------------- (Updated 2012-04-25 06:47:11) Review request for Flume. Summary ------- Changed the way we log during configuration validation. This addresses bug FLUME-1148 . https://issues.apache.org/jira/browse/FLUME-1148 Diffs ----- flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java 97ac302 flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java 881c0fa flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java ac4d3a3 flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java 24b4e1c Diff: https://reviews.apache.org/r/4870/diff Testing ------- Ran an agent with template configuration. In the following log, lines between <-- and --> were added here(they arent part of the log): 2012-04-24 23:14:20,840 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:284)] Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration [agent] SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }} bq. CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }} bq. SINKS: {loggerSink={ parameters:{type=logger, channel=memoryChannel} }} bq. bq. <- This stuff is what we read from the conf file. It has not yet been validated, therefore we say initial configuration(this is the same as before, just format changed) -> bq. bq. bq. bq. 2012-04-24 23:14:20,846 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateChannels(FlumeConfiguration.java:438)] Created channel memoryChannel bq. 2012-04-24 23:14:20,860 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.validateSinks(FlumeConfiguration.java:617)] Creating sink: loggerSink using LOGGER bq. bq. <- The following is post validation -> bq. bq. 2012-04-24 23:14:20,864 (conf-file-poller-0) [DEBUG - org.apache.flume.conf.FlumeConfiguration$AgentConfiguration.isValid(FlumeConfiguration.java:342)] Post validation configuration for agent bq. AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed [agent] bq. SOURCES: {seqGenSrc={ parameters:{channels=memoryChannel, type=seq} }} CHANNELS: {memoryChannel={ parameters:{capacity=100, type=memory} }} <--This stuff - we couldn't find config stubs for these. We did basic syntactic validation, and found them to be ok. Since no sinks belong to this category(because we could do component specific validation), there is no sinks section. If there was a sink, whcih we could do only Syntactic validation, even that would appear here --> AgentConfiguration created with Configuration stubs for which full validation was performed [agent] SINKS: {loggerSink=ComponentConfiguration[loggerSink] bq. CONFIG: bq. CHANNEL:memoryChannel bq. } <--We could do a more thorough component specific validation of this stuff. Since we couldn't do component specific validation of any channels, or sources, they aren't listed. If we could, then there would be a section for that and it would be listed like SOURCES: <blah> --> Thanks, Hari
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Hari!

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Hari!
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #185 (See https://builds.apache.org/job/flume-trunk/185/)
        FLUME-1148. Verbose logging for configuration validation.

        (Hari Shreedharan via Arvind Prabhakar) (Revision 1330231)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1330231
        Files :

        • /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java
        • /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java
        • /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java
        • /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java
        Show
        Hudson added a comment - Integrated in flume-trunk #185 (See https://builds.apache.org/job/flume-trunk/185/ ) FLUME-1148 . Verbose logging for configuration validation. (Hari Shreedharan via Arvind Prabhakar) (Revision 1330231) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1330231 Files : /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/ComponentConfiguration.java /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/FlumeConfiguration.java /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/sink/SinkConfiguration.java /incubator/flume/trunk/flume-ng-configuration/src/main/java/org/apache/flume/conf/source/SourceConfiguration.java

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Will McQueen
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development