Details

    • Type: New Feature New Feature
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.4
    • Fix Version/s: None
    • Component/s: Easy, Sinks+Sources
    • Labels:
      None

      Description

      It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

      This is probably just a simplification of syslogudp.

        Issue Links

          Activity

          Hide
          Jonathan Hsieh added a comment -

          Similar but 127 is tcp.

          Show
          Jonathan Hsieh added a comment - Similar but 127 is tcp.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2380/
          -----------------------------------------------------------

          Review request for Flume.

          Summary
          -------

          It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

          This addresses bug Flume-209.
          https://issues.apache.org/jira/browse/Flume-209

          Diffs


          flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111
          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION
          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2380/diff

          Testing
          -------

          TestUdp is added and passed locally.

          Thanks,

          Mingjie

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/ ----------------------------------------------------------- Review request for Flume. Summary ------- It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network. This addresses bug Flume-209. https://issues.apache.org/jira/browse/Flume-209 Diffs flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111 flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION Diff: https://reviews.apache.org/r/2380/diff Testing ------- TestUdp is added and passed locally. Thanks, Mingjie
          Hide
          Mingjie Lai added a comment -

          I know current trunk might be replace by the flume-ng in near future. However we're using this udp source, and I think it makes sense to push to upstream. Please help to review and commit. Thanks.

          Show
          Mingjie Lai added a comment - I know current trunk might be replace by the flume-ng in near future. However we're using this udp source, and I think it makes sense to push to upstream. Please help to review and commit. Thanks.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2380/#review2553
          -----------------------------------------------------------

          Mingjie,

          Look pretty good.

          I noticed a few nits that should be addressed.

          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java
          <https://reviews.apache.org/r/2380/#comment5729>

          change to apache

          /*

          • Licensed to the Apache Software Foundation (ASF) under one
          • or more contributor license agreements. See the NOTICE file
          • distributed with this work for additional information
          • regarding copyright ownership. The ASF licenses this file
          • to you under the Apache License, Version 2.0 (the
          • "License"); you may not use this file except in compliance
          • with the License. You may obtain a copy of the License at
            *
          • http://www.apache.org/licenses/LICENSE-2.0
            *
          • Unless required by applicable law or agreed to in writing, software
          • distributed under the License is distributed on an "AS IS" BASIS,
          • WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
          • See the License for the specific language governing permissions and
          • limitations under the License.
            */

          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java
          <https://reviews.apache.org/r/2380/#comment5726>

          needs ')'

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java
          <https://reviews.apache.org/r/2380/#comment5730>

          change to apache

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java
          <https://reviews.apache.org/r/2380/#comment5727>

          final?

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java
          <https://reviews.apache.org/r/2380/#comment5728>

          Maybe put in finally so that we don't have to hang 30s if this fails or funny exceptions thrown?

          • jmhsieh

          On 2011-10-13 07:51:02, Mingjie Lai wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2380/

          -----------------------------------------------------------

          (Updated 2011-10-13 07:51:02)

          Review request for Flume.

          Summary

          -------

          It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

          This addresses bug Flume-209.

          https://issues.apache.org/jira/browse/Flume-209

          Diffs

          -----

          flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111

          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2380/diff

          Testing

          -------

          TestUdp is added and passed locally.

          Thanks,

          Mingjie

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/#review2553 ----------------------------------------------------------- Mingjie, Look pretty good. I noticed a few nits that should be addressed. flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java < https://reviews.apache.org/r/2380/#comment5729 > change to apache /* Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements. See the NOTICE file distributed with this work for additional information regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at * http://www.apache.org/licenses/LICENSE-2.0 * Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. */ flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java < https://reviews.apache.org/r/2380/#comment5726 > needs ')' flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java < https://reviews.apache.org/r/2380/#comment5730 > change to apache flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java < https://reviews.apache.org/r/2380/#comment5727 > final? flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java < https://reviews.apache.org/r/2380/#comment5728 > Maybe put in finally so that we don't have to hang 30s if this fails or funny exceptions thrown? jmhsieh On 2011-10-13 07:51:02, Mingjie Lai wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/ ----------------------------------------------------------- (Updated 2011-10-13 07:51:02) Review request for Flume. Summary ------- It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network. This addresses bug Flume-209. https://issues.apache.org/jira/browse/Flume-209 Diffs ----- flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111 flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION Diff: https://reviews.apache.org/r/2380/diff Testing ------- TestUdp is added and passed locally. Thanks, Mingjie
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2380/
          -----------------------------------------------------------

          (Updated 2011-10-13 18:07:50.050517)

          Review request for Flume.

          Changes
          -------

          Updated to fix the issues raised by Jon's reviews. Thanks, Jon.

          Summary
          -------

          It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

          This addresses bug Flume-209.
          https://issues.apache.org/jira/browse/Flume-209

          Diffs (updated)


          flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111
          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION
          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2380/diff

          Testing
          -------

          TestUdp is added and passed locally.

          Thanks,

          Mingjie

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/ ----------------------------------------------------------- (Updated 2011-10-13 18:07:50.050517) Review request for Flume. Changes ------- Updated to fix the issues raised by Jon's reviews. Thanks, Jon. Summary ------- It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network. This addresses bug Flume-209. https://issues.apache.org/jira/browse/Flume-209 Diffs (updated) flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111 flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION Diff: https://reviews.apache.org/r/2380/diff Testing ------- TestUdp is added and passed locally. Thanks, Mingjie
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2380/#review2649
          -----------------------------------------------------------

          Ship it!

          lgtm.

          • jmhsieh

          On 2011-10-13 18:07:50, Mingjie Lai wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2380/

          -----------------------------------------------------------

          (Updated 2011-10-13 18:07:50)

          Review request for Flume.

          Summary

          -------

          It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

          This addresses bug Flume-209.

          https://issues.apache.org/jira/browse/Flume-209

          Diffs

          -----

          flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111

          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2380/diff

          Testing

          -------

          TestUdp is added and passed locally.

          Thanks,

          Mingjie

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/#review2649 ----------------------------------------------------------- Ship it! lgtm. jmhsieh On 2011-10-13 18:07:50, Mingjie Lai wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/ ----------------------------------------------------------- (Updated 2011-10-13 18:07:50) Review request for Flume. Summary ------- It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network. This addresses bug Flume-209. https://issues.apache.org/jira/browse/Flume-209 Diffs ----- flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111 flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION Diff: https://reviews.apache.org/r/2380/diff Testing ------- TestUdp is added and passed locally. Thanks, Mingjie
          Hide
          Jonathan Hsieh added a comment -

          Currently testing precommit.

          Show
          Jonathan Hsieh added a comment - Currently testing precommit.
          Hide
          Mingjie Lai added a comment -

          I also have a threaded version to maximize the throughput. Want me to post it?

          Show
          Mingjie Lai added a comment - I also have a threaded version to maximize the throughput. Want me to post it?
          Hide
          Jonathan Hsieh added a comment -

          @Mingjie,

          If the threaded version is separate (a separate source), put it up for
          review.

          If it is built off of the current on, let's get this version working first!
          I've posted some comments on the review – apparently there is a flakyness
          problem with the test.

          Jon.

          On Thu, Oct 20, 2011 at 4:54 PM, Mingjie Lai (Commented) (JIRA) <


          // Jonathan Hsieh (shay)
          // Software Engineer, Cloudera
          // jon@cloudera.com

          Show
          Jonathan Hsieh added a comment - @Mingjie, If the threaded version is separate (a separate source), put it up for review. If it is built off of the current on, let's get this version working first! I've posted some comments on the review – apparently there is a flakyness problem with the test. Jon. On Thu, Oct 20, 2011 at 4:54 PM, Mingjie Lai (Commented) (JIRA) < – // Jonathan Hsieh (shay) // Software Engineer, Cloudera // jon@cloudera.com
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2380/#review2731
          -----------------------------------------------------------

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java
          <https://reviews.apache.org/r/2380/#comment6147>

          mvn test -Dtest=TestUdpSource

          This line seems to fail due to timeout. On a good run it seems to finish in 5-10s but on a bad run it fails.

          My guess is that there is udp loss due to a full udp buffer. Maybe it has the delay in the sink side is the problem? (why not process right away, and put delays on the send side?)

          It would great to figure out why.

          • jmhsieh

          On 2011-10-13 18:07:50, Mingjie Lai wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2380/

          -----------------------------------------------------------

          (Updated 2011-10-13 18:07:50)

          Review request for Flume.

          Summary

          -------

          It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network.

          This addresses bug Flume-209.

          https://issues.apache.org/jira/browse/Flume-209

          Diffs

          -----

          flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111

          flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION

          flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION

          Diff: https://reviews.apache.org/r/2380/diff

          Testing

          -------

          TestUdp is added and passed locally.

          Thanks,

          Mingjie

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/#review2731 ----------------------------------------------------------- flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java < https://reviews.apache.org/r/2380/#comment6147 > mvn test -Dtest=TestUdpSource This line seems to fail due to timeout. On a good run it seems to finish in 5-10s but on a bad run it fails. My guess is that there is udp loss due to a full udp buffer. Maybe it has the delay in the sink side is the problem? (why not process right away, and put delays on the send side?) It would great to figure out why. jmhsieh On 2011-10-13 18:07:50, Mingjie Lai wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2380/ ----------------------------------------------------------- (Updated 2011-10-13 18:07:50) Review request for Flume. Summary ------- It would be nice to have something listen on a specified udp port and convert each packet them into flume events. This could be useful for "sensors" or other simple devices on the network. This addresses bug Flume-209. https://issues.apache.org/jira/browse/Flume-209 Diffs ----- flume-core/src/main/java/com/cloudera/flume/conf/SourceFactoryImpl.java ed7b111 flume-core/src/main/java/com/cloudera/flume/handlers/socket/UdpSource.java PRE-CREATION flume-core/src/test/java/com/cloudera/flume/handlers/socket/TestUdpSource.java PRE-CREATION Diff: https://reviews.apache.org/r/2380/diff Testing ------- TestUdp is added and passed locally. Thanks, Mingjie

            People

            • Assignee:
              Unassigned
              Reporter:
              Jonathan Hsieh
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:

                Development