Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-2863

How to install Flume on Windows server (best practices)

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Trivial
    • Resolution: Unresolved
    • Affects Version/s: 1.6.0
    • Fix Version/s: None
    • Component/s: Configuration
    • Environment:

      Windows Server 2008 R2 Datacenter
      Intel Xeon CPU e5 - 4Gb ram
      Flume1.6.0.
      Flume0.9.4 (exe)

    • Flags:
      Important

      Description

      Hi guys,
      I have to install flume on Windows environment and, have some issues..

      1st attempt - I used a file flume-node0.9.4.exe from cloudera git website (https://github.com/cloudera/flume/downloads) than, the log files on flume give that error:
      WARN agent.MultiMasterRPC: Could not connect to any master nodes (tried 1: [localhost:35872])
      INFO agent.MultiMasterRPC: No active master RPC connection

      2nd attempt - I downloaded flume from apache website (https://flume.apache.org/download.html) file: apache-flume-1.6.0-bin.tar.gz, extract all files and configure files: flume.conf, flume-env.sh and, executed flume-ng.cmd (command: flume-ng shaman) I don't receive any error message, just simple shutdown of flume-ng.cmd window..

      Anyone has any experience with Flume on Windows sending .log files to HDFS in another environment?

      Attach my file flume.conf to all see what I wanna do..

      – define source, channel and sink
      shaman.sources = tail_source1
      shaman.channels = ch1
      shaman.sinks = hdfs_sink1
      – define tail source
      shaman.sources.tail_source1.type = exec
      shaman.sources.tail_source1.channels = ch1
      shaman.sources.tail_source1.shell = /bin/bash -c
      shaman.sources.tail_source1.command = tail -F F:/LogFiles/w3r54
      shaman.sources.tail_source1.interceptors = ts
      shaman.sources.tail_source1.interceptors.ts.type = timestamp
      – define in-memory channel
      shaman.channels.ch1.type = memory
      shaman.channels.ch1.capacity = 100000
      shaman.channels.ch1.transactionCapacity = 1000
      – define HDFS sink properties
      shaman.sinks.hdfs_sink1.type = hdfs
      shaman.sinks.hdfs_sink1.hdfs.path = hdfs://xxxx:8020/user/admin/fromflume/%y%m%d/%H%M%S
      shaman.sinks.hdfs_sink1.hdfs.fileType = DataStream
      shaman.sinks.hdfs_sink1.channel = ch1

      Thanks!

      Alan..

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              alan.paiva Alan Paiva
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Time Tracking

                Estimated:
                Original Estimate - 24h
                24h
                Remaining:
                Remaining Estimate - 24h
                24h
                Logged:
                Time Spent - Not Specified
                Not Specified