Flume
  1. Flume
  2. FLUME-1222

rolling file sink "Directory may not be null" (Preconditions check) error when a directory is set

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Blocker Blocker
    • Resolution: Not A Problem
    • Affects Version/s: v1.2.0
    • Fix Version/s: None
    • Component/s: Docs, Sinks+Sources
    • Labels:
      None

      Description

      12/05/22 08:24:21 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 1
      12/05/22 08:24:21 INFO node.FlumeNode: Flume node starting - agent1
      12/05/22 08:24:21 INFO nodemanager.DefaultLogicalNodeManager: Node manager starting
      12/05/22 08:24:21 INFO lifecycle.LifecycleSupervisor: Starting lifecycle supervisor 10
      12/05/22 08:24:21 INFO properties.PropertiesFileConfigurationProvider: Configuration provider starting
      12/05/22 08:24:21 INFO properties.PropertiesFileConfigurationProvider: Reloading configuration file:/flumeNG/test1.cnf
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:sink
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:log-sink1
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:sink
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:log-sink1
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:sink
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Processing:sink
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Added sinks: sink Agent: agent1
      12/05/22 08:24:22 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [agent1]
      12/05/22 08:24:22 INFO properties.PropertiesFileConfigurationProvider: Creating channels
      12/05/22 08:24:23 INFO properties.PropertiesFileConfigurationProvider: created channel ch1
      12/05/22 08:24:23 INFO sink.DefaultSinkFactory: Creating instance of sink sink typefile_roll
      >>>>>>>> content of directory:null <<<<<<<<<<
      12/05/22 08:24:23 ERROR properties.PropertiesFileConfigurationProvider: Failed to load configuration data. Exception follows.
      java.lang.IllegalArgumentException: Directory may not be nullnull
      at com.google.common.base.Preconditions.checkArgument(Preconditions.java:88)
      at org.apache.flume.sink.RollingFileSink.configure(RollingFileSink.java:85)
      at org.apache.flume.conf.Configurables.configure(Configurables.java:41)
      at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.loadSinks(PropertiesFileConfigurationProvider.java:350)
      at org.apache.flume.conf.properties.PropertiesFileConfigurationProvider.load(PropertiesFileConfigurationProvider.java:223)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider.doLoad(AbstractFileConfigurationProvider.java:123)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider.access$300(AbstractFileConfigurationProvider.java:38)
      at org.apache.flume.conf.file.AbstractFileConfigurationProvider$FileWatcherRunnable.run(AbstractFileConfigurationProvider.java:202)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)

      Config:
      agent1.channels.ch1.type = memory

      agent1.sources.avro-source1.channels = ch1
      agent1.sources.avro-source1.type = avro
      agent1.sources.avro-source1.bind = 0.0.0.0
      agent1.sources.avro-source1.port = 41414

      agent1.sinks.log-sink1.channel = ch1
      agent1.sinks.log-sink1.type = logger

      agent1.sinks.sink.channel = ch1
      agent1.sinks.sink.directory = /home/alex/test1/
      agent1.sinks.sink.rollInterval=0
      agent1.sinks.sink.type = file_roll

      agent1.channels = ch1
      agent1.sources = avro-source1
      agent1.sinks = sink

      I added System.out.print ("content of directory:" + directory +"\n"); indeed the variable is empty.

        Issue Links

          Activity

          Hide
          Alexander Alten-Lorenz added a comment -

          RollingFileSink.java:

          String directory = context.getString("sink.directory");
          System.out.print ("content of directory:" + directory +"\n");

          ...

          Preconditions.checkArgument(directory != null, "Directory may not be null");

          Show
          Alexander Alten-Lorenz added a comment - RollingFileSink.java: String directory = context.getString("sink.directory"); System.out.print ("content of directory:" + directory +"\n"); ... Preconditions.checkArgument(directory != null, "Directory may not be null");
          Hide
          Mike Percy added a comment -

          Some of the flume NG config options are a little strange. This is one of them.

          You have to specify agent.sinks.file-sink-1.sink.directory = /some/dir

          Note the extra ".sink" before ".directory". It's for historical reasons.

          Show
          Mike Percy added a comment - Some of the flume NG config options are a little strange. This is one of them. You have to specify agent.sinks.file-sink-1.sink.directory = /some/dir Note the extra ".sink" before ".directory". It's for historical reasons.
          Hide
          Alexander Alten-Lorenz added a comment - - edited

          Ah, thanks Mike. We should that add into the user guide.

          Show
          Alexander Alten-Lorenz added a comment - - edited Ah, thanks Mike. We should that add into the user guide.
          Hide
          Alexander Alten-Lorenz added a comment -

          Doesn't work

          Show
          Alexander Alten-Lorenz added a comment - Doesn't work
          Hide
          Alexander Alten-Lorenz added a comment -

          The whole config section has to be extended with a additional identifier:

          agent1.sinks.file-sink-1.sink.channel = ch1
          agent1.sinks.file-sink-1.sink.directory = /foo/bar
          agent1.sinks.file-sink-1.sink.rollInterval=0
          agent1.sinks.file-sink-1.sink.type = file_roll

          Reopened and marked as DOC

          Show
          Alexander Alten-Lorenz added a comment - The whole config section has to be extended with a additional identifier: agent1.sinks.file-sink-1.sink.channel = ch1 agent1.sinks.file-sink-1.sink.directory = /foo/bar agent1.sinks.file-sink-1.sink.rollInterval=0 agent1.sinks.file-sink-1.sink.type = file_roll Reopened and marked as DOC
          Hide
          Alexander Alten-Lorenz added a comment -

          Moved into DOC

          Show
          Alexander Alten-Lorenz added a comment - Moved into DOC
          Hide
          Mike Percy added a comment -

          Not everything. Should be like this:

          agent1.sinks.file-sink-1.type = file_roll
          agent1.sinks.file-sink-1.channel = ch1
          agent1.sinks.file-sink-1.sink.directory = /foo/bar
          agent1.sinks.file-sink-1.sink.rollInterval=0

          Show
          Mike Percy added a comment - Not everything. Should be like this: agent1.sinks.file-sink-1.type = file_roll agent1.sinks.file-sink-1.channel = ch1 agent1.sinks.file-sink-1.sink.directory = /foo/bar agent1.sinks.file-sink-1.sink.rollInterval=0
          Hide
          Alexander Alten-Lorenz added a comment -

          Added into FLUME-1223, thanks again

          Show
          Alexander Alten-Lorenz added a comment - Added into FLUME-1223 , thanks again
          Hide
          Mike Percy added a comment -

          Oh, gotcha. Thanks Alex!!

          Show
          Mike Percy added a comment - Oh, gotcha. Thanks Alex!!

            People

            • Assignee:
              Unassigned
              Reporter:
              Alexander Alten-Lorenz
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development