Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4777

ContinuousFileMonitoringFunction may throw IOException when files are moved

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2.0, 1.1.2
    • Fix Version/s: 1.2.0, 1.1.3
    • Component/s: Streaming Connectors
    • Labels:
      None

      Description

      FileSystem.listStatus(path) may throw an IOException when it lists files and then retrieves their file status. This is quite common, e.g. editors which create temporary files and move them. The ContinuousFileMonitoringFunction can only apply a file path filter afterwards. Thus, this problem can't easily be avoided and needs to be taken care of by the function itself.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user mxm opened a pull request:

          https://github.com/apache/flink/pull/2610

          FLINK-4777 catch IOException in ContinuousFileMonitoringFunction

          FileSystem.listStatus(path) may throw an IOException when it lists files
          and then retrieves their file status. This is quite common, e.g. editors
          which create temporary files and move them. The
          ContinuousFileMonitoringFunction can only apply a file path filter
          afterwards.

          The solution is to defer file checks until no exception is caught anymore.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/mxm/flink release-1.1

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2610.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2610


          commit 7cb492410ab083dcc56515a0c653936fe32c2c14
          Author: Maximilian Michels <mxm@apache.org>
          Date: 2016-10-07T18:06:18Z

          FLINK-4777 catch IOException in ContinuousFileMonitoringFunction

          FileSystem.listStatus(path) may throw an IOException when it lists files
          and then retrieves their file status. This is quite common, e.g. editors
          which create temporary files and move them. The
          ContinuousFileMonitoringFunction can only apply a file path filter
          afterwards.

          The solution is to defer file checks until no exception is caught anymore.


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user mxm opened a pull request: https://github.com/apache/flink/pull/2610 FLINK-4777 catch IOException in ContinuousFileMonitoringFunction FileSystem.listStatus(path) may throw an IOException when it lists files and then retrieves their file status. This is quite common, e.g. editors which create temporary files and move them. The ContinuousFileMonitoringFunction can only apply a file path filter afterwards. The solution is to defer file checks until no exception is caught anymore. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mxm/flink release-1.1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2610.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2610 commit 7cb492410ab083dcc56515a0c653936fe32c2c14 Author: Maximilian Michels <mxm@apache.org> Date: 2016-10-07T18:06:18Z FLINK-4777 catch IOException in ContinuousFileMonitoringFunction FileSystem.listStatus(path) may throw an IOException when it lists files and then retrieves their file status. This is quite common, e.g. editors which create temporary files and move them. The ContinuousFileMonitoringFunction can only apply a file path filter afterwards. The solution is to defer file checks until no exception is caught anymore.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/2610

          Looks good to me...

          +1 to merge this

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/2610 Looks good to me... +1 to merge this
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/2610

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2610
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user mxm commented on the issue:

          https://github.com/apache/flink/pull/2610

          Thanks, merged to `release-1.1` and `master`.

          Show
          githubbot ASF GitHub Bot added a comment - Github user mxm commented on the issue: https://github.com/apache/flink/pull/2610 Thanks, merged to `release-1.1` and `master`.
          Hide
          mxm Maximilian Michels added a comment -

          master: b949d42d9f16fe52126f03a960fdf23ad71e6e63
          release-1.1: 7267562bb8110e3f7300007e996ce96355d37c59

          Show
          mxm Maximilian Michels added a comment - master: b949d42d9f16fe52126f03a960fdf23ad71e6e63 release-1.1: 7267562bb8110e3f7300007e996ce96355d37c59

            People

            • Assignee:
              mxm Maximilian Michels
              Reporter:
              mxm Maximilian Michels
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development