Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-1934

Spoolingdir source exception when reading multiple zero size files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.3.1
    • Fix Version/s: 1.6.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      windows 7, flume 1.3.1, spooling dir source.

      Description

      move more than one files to the spool dir, and each file size is 0, then flume agent will throw IllegalStateException forever, and never work again,its' main cause is commited flag will not set to true.

      logs:

      08 三月 2013 08:00:14,406 ERROR [pool-5-thread-1] (org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run:148) - Uncaught exception in Runnable
      java.lang.IllegalStateException: File should not roll when commit is outstanding.
      at org.apache.flume.client.avro.SpoolingFileLineReader.readLines(SpoolingFileLineReader.java:164)
      at org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:135)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
      at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:619)

        Issue Links

          Activity

          Hide
          mpercy Mike Percy added a comment -

          I think this is fixed. Does this still affect you on Flume 1.4.0?

          Show
          mpercy Mike Percy added a comment - I think this is fixed. Does this still affect you on Flume 1.4.0?
          Hide
          mpercy Mike Percy added a comment -

          I believe the underlying issue could actually be FLUME-2157 ... please reopen if you still see this issue on trunk.

          Show
          mpercy Mike Percy added a comment - I believe the underlying issue could actually be FLUME-2157 ... please reopen if you still see this issue on trunk.
          Hide
          mpercy Mike Percy added a comment -

          Another possible issue could be that you were running multiple SpoolDir sources pointing to the same spool directory. That will not work and is not supported.

          Show
          mpercy Mike Percy added a comment - Another possible issue could be that you were running multiple SpoolDir sources pointing to the same spool directory. That will not work and is not supported.
          Hide
          granthenke Grant Henke added a comment -

          This is still an issue. It can be quickly reproduced with the following command:

          1. Create empty files in a spooling directory
            touch flumeSpool/zero1; touch flumeSpool/zero2; touch flumeSpool/zero3;

          I will attach a patch with a failing test and code to make it succeed. If there is a better way to write the test please provide feedback.

          Show
          granthenke Grant Henke added a comment - This is still an issue. It can be quickly reproduced with the following command: Create empty files in a spooling directory touch flumeSpool/zero1; touch flumeSpool/zero2; touch flumeSpool/zero3; I will attach a patch with a failing test and code to make it succeed. If there is a better way to write the test please provide feedback.
          Hide
          gwenshap Gwen Shapira added a comment -

          Clicking for Grant Henke who submitted the patch but apparently doesn't have access to change status

          Show
          gwenshap Gwen Shapira added a comment - Clicking for Grant Henke who submitted the patch but apparently doesn't have access to change status
          Hide
          jarcec Jarek Jarcec Cecho added a comment -

          I've granted you "contributor" role on JIRA Grant Henke. You should be able to assign the JIRA to yourself.

          Show
          jarcec Jarek Jarcec Cecho added a comment - I've granted you "contributor" role on JIRA Grant Henke . You should be able to assign the JIRA to yourself.
          Hide
          gwenshap Gwen Shapira added a comment -

          +1 - simple and clean fix, nice tests and nice log message.

          Show
          gwenshap Gwen Shapira added a comment - +1 - simple and clean fix, nice tests and nice log message.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          +1. Looks good, running tests and committing.

          Show
          hshreedharan Hari Shreedharan added a comment - +1. Looks good, running tests and committing.
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Holding off until FLUME-2657 is committed, as full builds are currently failing due to hive depending on a snapshot of calcite that is now missing from snapshots repos

          Show
          hshreedharan Hari Shreedharan added a comment - Holding off until FLUME-2657 is committed, as full builds are currently failing due to hive depending on a snapshot of calcite that is now missing from snapshots repos
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 4d2a34e931554baa1c1b255d95540a46354a521f in flume's branch refs/heads/trunk from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4d2a34e ]

          FLUME-1934. Spooling Directory Source dies on encountering zero-byte files.

          (Grant Henke via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 4d2a34e931554baa1c1b255d95540a46354a521f in flume's branch refs/heads/trunk from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=4d2a34e ] FLUME-1934 . Spooling Directory Source dies on encountering zero-byte files. (Grant Henke via Hari)
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 04f384576c1a84677d11a693812def519ac57baa in flume's branch refs/heads/flume-1.6 from Hari Shreedharan
          [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=04f3845 ]

          FLUME-1934. Spooling Directory Source dies on encountering zero-byte files.

          (Grant Henke via Hari)

          Show
          jira-bot ASF subversion and git services added a comment - Commit 04f384576c1a84677d11a693812def519ac57baa in flume's branch refs/heads/flume-1.6 from Hari Shreedharan [ https://git-wip-us.apache.org/repos/asf?p=flume.git;h=04f3845 ] FLUME-1934 . Spooling Directory Source dies on encountering zero-byte files. (Grant Henke via Hari)
          Hide
          hshreedharan Hari Shreedharan added a comment -

          Committed! Thanks Grant!

          Show
          hshreedharan Hari Shreedharan added a comment - Committed! Thanks Grant!
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in flume-trunk #726 (See https://builds.apache.org/job/flume-trunk/726/)
          FLUME-1934. Spooling Directory Source dies on encountering zero-byte files. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4d2a34e931554baa1c1b255d95540a46354a521f)

          • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in flume-trunk #726 (See https://builds.apache.org/job/flume-trunk/726/ ) FLUME-1934 . Spooling Directory Source dies on encountering zero-byte files. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4d2a34e931554baa1c1b255d95540a46354a521f ) flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Hide
          hudson Hudson added a comment -

          UNSTABLE: Integrated in Flume-trunk-hbase-98 #83 (See https://builds.apache.org/job/Flume-trunk-hbase-98/83/)
          FLUME-1934. Spooling Directory Source dies on encountering zero-byte files. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4d2a34e931554baa1c1b255d95540a46354a521f)

          • flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java
          • flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java
          Show
          hudson Hudson added a comment - UNSTABLE: Integrated in Flume-trunk-hbase-98 #83 (See https://builds.apache.org/job/Flume-trunk-hbase-98/83/ ) FLUME-1934 . Spooling Directory Source dies on encountering zero-byte files. (hshreedharan: http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=4d2a34e931554baa1c1b255d95540a46354a521f ) flume-ng-core/src/test/java/org/apache/flume/source/TestSpoolDirectorySource.java flume-ng-core/src/main/java/org/apache/flume/client/avro/ReliableSpoolingFileEventReader.java

            People

            • Assignee:
              granthenke Grant Henke
              Reporter:
              ablozhou andy zhou
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development