Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20804

Add SwiftFS end-to-end test

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: FileSystems, Tests
    • Labels:
      None

      Description

      Our swift filesystem is closely aligned with the S3 filesystems in that it follows the same packaging scheme as flink-fs-hadoop-shaded (used by the S3 filesystems), and thus ideally is kept in sync with these filesystems, for example in regards to dependencies.

      As it stands, this is not the case. While the S3 modules work against Hadoop 3, swift is still sitting on 2.8.1 . Presumable this difference is due to us having no infrastructure for actually testing our swift filesystem.

      Unfortunately it appears that Switft is rather annoying to setup, but I was able find a project that tries to provide a docker image containing everything you need. Maybe we could use this to bootstrap some end-to-end-test, allowing us to finally touch this module again without fear of completely breaking it.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              chesnay Chesnay Schepler

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment