Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4870

ContinuousFileMonitoringFunction does not properly handle absolut Windows paths

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 1.1.2
    • Fix Version/s: 1.2.0, 1.3.0
    • Component/s: Streaming
    • Labels:
      None

      Description

      The ContinuousFileMonitoringFunction fails for absolute windows paths without a dedicated scheme (e.g "C:\\tmp
      test.csv"), since the String path is directly fed into the URI constructor (which doesn't handle it properly) instead of first creating a flink Path and converting that into an URI.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user zentol opened a pull request:

          https://github.com/apache/flink/pull/2887

          FLINK-4870 FLINK-5162 Fix issues on windows related to Path/URI generation

          This PR fixes the FileSystemBlocStore and ContinuousFileMonitoringFunction on Windows.

          In both classes an URI is generated from a String and given to a Flink FileSystem instance. However, on Windows it is a hard requirement that all URI's that are generated first go through the Path class.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/zentol/flink 5160_4870_test_uri_generation

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/2887.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #2887


          commit 2942b871ad1a0369e37aa72eae17ca6a126ee212
          Author: zentol <chesnay@apache.org>
          Date: 2016-11-25T12:23:56Z

          FLINK-5162 Fix BlobRecoveryITCase#testBlobServerRecovery on Windows

          commit 935943ad520e253d62da22f3320e15b619f11325
          Author: zentol <chesnay@apache.org>
          Date: 2016-11-25T12:27:43Z

          FLINK-4870 Fix path handling in ContinuousFileMonitoringFunction


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user zentol opened a pull request: https://github.com/apache/flink/pull/2887 FLINK-4870 FLINK-5162 Fix issues on windows related to Path/URI generation This PR fixes the FileSystemBlocStore and ContinuousFileMonitoringFunction on Windows. In both classes an URI is generated from a String and given to a Flink FileSystem instance. However, on Windows it is a hard requirement that all URI's that are generated first go through the Path class. You can merge this pull request into a Git repository by running: $ git pull https://github.com/zentol/flink 5160_4870_test_uri_generation Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/2887.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2887 commit 2942b871ad1a0369e37aa72eae17ca6a126ee212 Author: zentol <chesnay@apache.org> Date: 2016-11-25T12:23:56Z FLINK-5162 Fix BlobRecoveryITCase#testBlobServerRecovery on Windows commit 935943ad520e253d62da22f3320e15b619f11325 Author: zentol <chesnay@apache.org> Date: 2016-11-25T12:27:43Z FLINK-4870 Fix path handling in ContinuousFileMonitoringFunction
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on a diff in the pull request:

          https://github.com/apache/flink/pull/2887#discussion_r92373183

          — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java —
          @@ -81,7 +81,7 @@ public void put(File localFile, JobID jobId, String key) throws Exception {
          }

          private void put(File fromFile, String toBlobPath) throws Exception {

          • try (OutputStream os = FileSystem.get(new URI(toBlobPath))
            + try (OutputStream os = FileSystem.get(new Path(toBlobPath).toUri())
              • End diff –

          Would be good to avoid creating the Path twice and do `Path p = new Path(toBlobPath); p.getFileSystem().create(p, true)`

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/2887#discussion_r92373183 — Diff: flink-runtime/src/main/java/org/apache/flink/runtime/blob/FileSystemBlobStore.java — @@ -81,7 +81,7 @@ public void put(File localFile, JobID jobId, String key) throws Exception { } private void put(File fromFile, String toBlobPath) throws Exception { try (OutputStream os = FileSystem.get(new URI(toBlobPath)) + try (OutputStream os = FileSystem.get(new Path(toBlobPath).toUri()) End diff – Would be good to avoid creating the Path twice and do `Path p = new Path(toBlobPath); p.getFileSystem().create(p, true)`
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/2887

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/2887
          Hide
          Zentol Chesnay Schepler added a comment -

          1.2: 24109cb2692f1f0dd2b9f8c9c8dcc02e55148bab
          1.3: ed83b5b89c61cac5d49e690279dab0ea94c35bb3

          Show
          Zentol Chesnay Schepler added a comment - 1.2: 24109cb2692f1f0dd2b9f8c9c8dcc02e55148bab 1.3: ed83b5b89c61cac5d49e690279dab0ea94c35bb3

            People

            • Assignee:
              Zentol Chesnay Schepler
              Reporter:
              Zentol Chesnay Schepler
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development