Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: NG alpha 1
    • Fix Version/s: v1.2.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      bin/flume-ng agent -n syslog-agent -f conf/flume-conf.properties.template
      Warning: No configuration directory set! Use --conf <dir> to override.
      12/03/03 09:57:34 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
      12/03/03 09:57:34 INFO node.FlumeNode: Flume node starting - syslog-agent
      12/03/03 09:57:34 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
      12/03/03 09:57:34 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
      12/03/03 09:57:34 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
      12/03/03 09:57:34 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:conf/flume-conf.properties.template
      12/03/03 09:57:34 WARN properties.FlumeConfiguration: Agent configuration for 'exec-agent' source 'tail' has no valid channels. Removing.
      12/03/03 09:57:34 WARN properties.FlumeConfiguration: Agent configuration for 'exec-agent': source 'ExecTail' does not have 'type' specified.
      12/03/03 09:57:34 WARN properties.FlumeConfiguration: Agent configuration for 'exec-agent': source 'ExecTail' has some required attributes missing. Removing.
      12/03/03 09:57:34 WARN properties.FlumeConfiguration: Agent configuration for 'exec-agent': source 'ExecTail' is not configured. Removing.
      12/03/03 09:57:34 INFO properties.FlumeConfiguration: Post-validation flume configuration contains configuation for agents: [foo, exec-agent, syslog-agent]
      12/03/03 09:57:34 ERROR properties.PropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
      org.apache.flume.FlumeException: Unable to load source type: syslogTcp, class: syslogTcp
      at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:92)
      at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSources(PropertiesFileConfigurationProvider.java:265)
      at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:213)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:124)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:203)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.ClassNotFoundException: syslogTcp
      at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:169)
      at org.apache.flume.source.DefaultSourceFactory.create(DefaultSourceFactory.java:90)

      1. web.conf
        0.7 kB
        Rafael Nunes
      2. FLUME-1017-1.patch
        0.7 kB
        Hari Shreedharan
      3. flume.log
        7 kB
        Rafael Nunes

        Activity

        Hide
        Arvind Prabhakar added a comment -

        @Alex - do you still see this happening on the trunk? If so, please provide a sample configuration that will reproduce this.

        Show
        Arvind Prabhakar added a comment - @Alex - do you still see this happening on the trunk? If so, please provide a sample configuration that will reproduce this.
        Hide
        Rafael Nunes added a comment - - edited

        @Arvind, hi.

        I'm facing the same issue.
        I took a look at the source code and find that in the package org.apache.flume.source exists an ENUM called SourceType that doesn't have the syslogTcp type in it.

        I'll try to implement this, but I'm too newbie as developer, so I thought that would be better, at least, point this out.

        Here is the link to this ENUM at the trunk:
        https://svn.apache.org/repos/asf/incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java

        Attached is the conf used by my agent and its log.

        Rgs.

        Show
        Rafael Nunes added a comment - - edited @Arvind, hi. I'm facing the same issue. I took a look at the source code and find that in the package org.apache.flume.source exists an ENUM called SourceType that doesn't have the syslogTcp type in it. I'll try to implement this, but I'm too newbie as developer, so I thought that would be better, at least, point this out. Here is the link to this ENUM at the trunk: https://svn.apache.org/repos/asf/incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java Attached is the conf used by my agent and its log. Rgs.
        Hide
        Rafael Nunes added a comment -

        Hi. Here are some points that I had to deal while trying to use a Syslog Source:

        1. When specifying the source's type, use the full class name, as mentioned in the Getting Started wiki page of Flume-NG. ie:

        web.conf
        # Define a memory channel called ch1 on web agent
        web.channels.ch1.type = memory
        
        # Define an SyslogTCP source called accesslog on web agent and
        # tell it to bind to 0.0.0.0:10514. Connect it to channel ch1.
        web.sources.accesslog.type = org.apache.flume.source.SyslogTcpSource
        web.sources.accesslog.host = 0.0.0.0
        web.sources.accesslog.port = 10514
        web.sources.accesslog.channels = ch1
        
        # Define a logger sink that simply logs all events it receives
        # and connect it to the other end of the same channel.
        web.sinks.log-sink1.type = logger
        web.sinks.log-sink1.channel = ch1
        
        # Finally, now that we've defined all of our components, tell
        # web which ones we want to activate.
        web.sources = accesslog
        web.sinks = log-sink1
        web.channels = ch1
        

        2. When using a RSYSLOG 4.6.3 (as in Fedora 14) to redirect remote messages to Flume's Syslog Source, either TCP or UDP works great;

        3. When using a SYSLOG 1.4.1 (as in RedHat 5.2), there are some issues:
        a) SYSLOG 1.4.1 supports only UDP;
        b) SYSLOG 1.4.1 only tries to connect on remote's host UDP port 514, so your SyslogUPDSource have to have this port configured (or use any sort of port redirection/translation).
        c) The SyslogUtils.extractEvent method behaves different when used against RSYSLOG UPD and SYSLOG UDP. In the excerpt of the SyslogUtils class bellow, I try to explain what I saw until here:

        SyslogUtils.java
          public static Event extractEvent(ChannelBuffer in) 
                throws IOException {
        
          //omitted code
        
            try {
              while (!doneReading) {
        
                // The line bellow generate a IndexOutOfBoundsException when using 
                // RSYSLOG, forcing the try/catch block execute the statement under
                // the catch and then "return e". But when using against SYSLOG 1.4.1
                // the Exception doesn't happen and the try block executes the 
                // "return null" line.
                b = in.readByte();
                switch (m) {
                case START:
                  if (b == '<') {
                    m = Mode.PRIO;
                  } else {
                    m = Mode.ERR;
                  }
                  break;
                case PRIO:
                  if (b == '>') {
                    m = Mode.DATA;
                  } else {
                    char ch = (char) b;
                    if (Character.isDigit(ch)) {
                      prio.append(ch); // stay in PRIO mode
                    } else {
                      m = Mode.ERR;
                    }
                  }
                  break;
                case DATA:
                  // TCP syslog entries are separated by '\n'
                  if (b == '\n') {
                    e = buildEvent(prio, baos);
                    doneReading = true;
                  }
                  baos.write(b);
                  break;
                case ERR:
                  if (b == '<') {
                    // check if its start of new event
                    m = Mode.PRIO;
                  }
                  // otherwise stay in Mode.ERR;
                  break;
                }
              }
              return null;
            } catch (IndexOutOfBoundsException eF) {
                e = buildEvent(prio, baos);
            }
            return e;
          }
        

        Well, after removing the "return null" from the try block, I was able to get my SyslogUDP Sources working with my RSYSLOG and SYSLOG servers.

        Any comments?

        Thanks for the good work on this project.

        Show
        Rafael Nunes added a comment - Hi. Here are some points that I had to deal while trying to use a Syslog Source: 1. When specifying the source's type, use the full class name, as mentioned in the Getting Started wiki page of Flume-NG. ie: web.conf # Define a memory channel called ch1 on web agent web.channels.ch1.type = memory # Define an SyslogTCP source called accesslog on web agent and # tell it to bind to 0.0.0.0:10514. Connect it to channel ch1. web.sources.accesslog.type = org.apache.flume.source.SyslogTcpSource web.sources.accesslog.host = 0.0.0.0 web.sources.accesslog.port = 10514 web.sources.accesslog.channels = ch1 # Define a logger sink that simply logs all events it receives # and connect it to the other end of the same channel. web.sinks.log-sink1.type = logger web.sinks.log-sink1.channel = ch1 # Finally, now that we've defined all of our components, tell # web which ones we want to activate. web.sources = accesslog web.sinks = log-sink1 web.channels = ch1 2. When using a RSYSLOG 4.6.3 (as in Fedora 14) to redirect remote messages to Flume's Syslog Source, either TCP or UDP works great; 3. When using a SYSLOG 1.4.1 (as in RedHat 5.2), there are some issues: a) SYSLOG 1.4.1 supports only UDP; b) SYSLOG 1.4.1 only tries to connect on remote's host UDP port 514, so your SyslogUPDSource have to have this port configured (or use any sort of port redirection/translation). c) The SyslogUtils.extractEvent method behaves different when used against RSYSLOG UPD and SYSLOG UDP. In the excerpt of the SyslogUtils class bellow, I try to explain what I saw until here: SyslogUtils.java public static Event extractEvent(ChannelBuffer in) throws IOException { //omitted code try { while (!doneReading) { // The line bellow generate a IndexOutOfBoundsException when using // RSYSLOG, forcing the try / catch block execute the statement under // the catch and then " return e" . But when using against SYSLOG 1.4.1 // the Exception doesn't happen and the try block executes the // " return null " line. b = in.readByte(); switch (m) { case START: if (b == '<') { m = Mode.PRIO; } else { m = Mode.ERR; } break ; case PRIO: if (b == '>') { m = Mode.DATA; } else { char ch = ( char ) b; if ( Character .isDigit(ch)) { prio.append(ch); // stay in PRIO mode } else { m = Mode.ERR; } } break ; case DATA: // TCP syslog entries are separated by '\n' if (b == '\n') { e = buildEvent(prio, baos); doneReading = true ; } baos.write(b); break ; case ERR: if (b == '<') { // check if its start of new event m = Mode.PRIO; } // otherwise stay in Mode.ERR; break ; } } return null ; } catch (IndexOutOfBoundsException eF) { e = buildEvent(prio, baos); } return e; } Well, after removing the "return null" from the try block, I was able to get my SyslogUDP Sources working with my RSYSLOG and SYSLOG servers. Any comments? Thanks for the good work on this project.
        Hide
        Arvind Prabhakar added a comment -

        Thanks for the details on this Rafael. We are working towards testing and fixing the Syslog source and will have an update available soon. Among the things that this update will address are the issues with SyslogUtils implementation along with concurrent connection handling details.

        Show
        Arvind Prabhakar added a comment - Thanks for the details on this Rafael. We are working towards testing and fixing the Syslog source and will have an update available soon. Among the things that this update will address are the issues with SyslogUtils implementation along with concurrent connection handling details.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4663/
        -----------------------------------------------------------

        Review request for Flume.

        Summary
        -------

        Adding SyslogTcpSource and SyslogUDPSource to the enum.

        This addresses bug FLUME-1017.
        https://issues.apache.org/jira/browse/FLUME-1017

        Diffs


        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470

        Diff: https://reviews.apache.org/r/4663/diff

        Testing
        -------

        Builds fine, current unit tests are ok.

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4663/ ----------------------------------------------------------- Review request for Flume. Summary ------- Adding SyslogTcpSource and SyslogUDPSource to the enum. This addresses bug FLUME-1017 . https://issues.apache.org/jira/browse/FLUME-1017 Diffs flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470 Diff: https://reviews.apache.org/r/4663/diff Testing ------- Builds fine, current unit tests are ok. Thanks, Hari
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4663/#review6724
        -----------------------------------------------------------

        Ship it!

        +1

        Please attach the patch to the Jira.

        • Arvind

        On 2012-04-06 01:07:45, Hari Shreedharan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4663/

        -----------------------------------------------------------

        (Updated 2012-04-06 01:07:45)

        Review request for Flume.

        Summary

        -------

        Adding SyslogTcpSource and SyslogUDPSource to the enum.

        This addresses bug FLUME-1017.

        https://issues.apache.org/jira/browse/FLUME-1017

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470

        Diff: https://reviews.apache.org/r/4663/diff

        Testing

        -------

        Builds fine, current unit tests are ok.

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4663/#review6724 ----------------------------------------------------------- Ship it! +1 Please attach the patch to the Jira. Arvind On 2012-04-06 01:07:45, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4663/ ----------------------------------------------------------- (Updated 2012-04-06 01:07:45) Review request for Flume. Summary ------- Adding SyslogTcpSource and SyslogUDPSource to the enum. This addresses bug FLUME-1017 . https://issues.apache.org/jira/browse/FLUME-1017 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470 Diff: https://reviews.apache.org/r/4663/diff Testing ------- Builds fine, current unit tests are ok. Thanks, Hari
        Hide
        Arvind Prabhakar added a comment -

        Patch committed. Thanks Hari!

        @Rafael - let us move the discussion on syslog source dropping events to FLUME-1103 which is filed to track this problem.

        Show
        Arvind Prabhakar added a comment - Patch committed. Thanks Hari! @Rafael - let us move the discussion on syslog source dropping events to FLUME-1103 which is filed to track this problem.
        Hide
        Hudson added a comment -

        Integrated in flume-trunk #160 (See https://builds.apache.org/job/flume-trunk/160/)
        FLUME-1017. Syslog Source does not have a short name.

        (Hari Shreedharan via Arvind Prabhakar) (Revision 1310149)

        Result = SUCCESS
        arvind : http://svn.apache.org/viewvc/?view=rev&rev=1310149
        Files :

        • /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java
        Show
        Hudson added a comment - Integrated in flume-trunk #160 (See https://builds.apache.org/job/flume-trunk/160/ ) FLUME-1017 . Syslog Source does not have a short name. (Hari Shreedharan via Arvind Prabhakar) (Revision 1310149) Result = SUCCESS arvind : http://svn.apache.org/viewvc/?view=rev&rev=1310149 Files : /incubator/flume/trunk/flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java
        Hide
        Will McQueen added a comment -

        I would have liked to see the SYSLOG_TCP and SYSLOG_UDP instead, to match the current convention of separating words with underscores like we currently do with FILE_ROLL.

        Show
        Will McQueen added a comment - I would have liked to see the SYSLOG_TCP and SYSLOG_UDP instead, to match the current convention of separating words with underscores like we currently do with FILE_ROLL.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/4663/#review6772
        -----------------------------------------------------------

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java
        <https://reviews.apache.org/r/4663/#comment14930>

        SYSLOGTCP => SYSLOG_TCP
        (to match current convention of underscore-separated words, like with FILE_ROLL)

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java
        <https://reviews.apache.org/r/4663/#comment14931>

        SYSLOGUDP => SYSLOG_UDP
        (to match current convention of underscore-separated words, like with FILE_ROLL)

        • Will

        On 2012-04-06 01:07:45, Hari Shreedharan wrote:

        -----------------------------------------------------------

        This is an automatically generated e-mail. To reply, visit:

        https://reviews.apache.org/r/4663/

        -----------------------------------------------------------

        (Updated 2012-04-06 01:07:45)

        Review request for Flume.

        Summary

        -------

        Adding SyslogTcpSource and SyslogUDPSource to the enum.

        This addresses bug FLUME-1017.

        https://issues.apache.org/jira/browse/FLUME-1017

        Diffs

        -----

        flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470

        Diff: https://reviews.apache.org/r/4663/diff

        Testing

        -------

        Builds fine, current unit tests are ok.

        Thanks,

        Hari

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4663/#review6772 ----------------------------------------------------------- flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java < https://reviews.apache.org/r/4663/#comment14930 > SYSLOGTCP => SYSLOG_TCP (to match current convention of underscore-separated words, like with FILE_ROLL) flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java < https://reviews.apache.org/r/4663/#comment14931 > SYSLOGUDP => SYSLOG_UDP (to match current convention of underscore-separated words, like with FILE_ROLL) Will On 2012-04-06 01:07:45, Hari Shreedharan wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4663/ ----------------------------------------------------------- (Updated 2012-04-06 01:07:45) Review request for Flume. Summary ------- Adding SyslogTcpSource and SyslogUDPSource to the enum. This addresses bug FLUME-1017 . https://issues.apache.org/jira/browse/FLUME-1017 Diffs ----- flume-ng-core/src/main/java/org/apache/flume/source/SourceType.java 9082470 Diff: https://reviews.apache.org/r/4663/diff Testing ------- Builds fine, current unit tests are ok. Thanks, Hari

          People

          • Assignee:
            Hari Shreedharan
            Reporter:
            Alexander Alten-Lorenz
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development