Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-11266

Only release hadoop-free Flink

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.8.0
    • 1.8.0
    • Release System
    • None
    • Hide
      Convenience binaries that include hadoop are no longer released.

      If a deployment relies on flink-shaded-hadoop2 being included in flink-dist, then it must be manually downloaded and copied into the /lib directory.
      Alternatively, a Flink distribution that includes hadoop can be built by packaging flink-dist and activating the "include-hadoop" maven profile.

      As hadoop is no longer included in flink-dist by default, specifying "-DwithoutHadoop" when packaging flink-dist no longer impacts the build.
      Show
      Convenience binaries that include hadoop are no longer released. If a deployment relies on flink-shaded-hadoop2 being included in flink-dist, then it must be manually downloaded and copied into the /lib directory. Alternatively, a Flink distribution that includes hadoop can be built by packaging flink-dist and activating the "include-hadoop" maven profile. As hadoop is no longer included in flink-dist by default, specifying "-DwithoutHadoop" when packaging flink-dist no longer impacts the build.

    Description

      Currently we release 10 different binary releases (2 scala versions * (4 hadoop version + hadoop-free)), which has increased the size of our release to more than 2 GB.
      Naturally, building Flink 10 times also takes a while, slowing down the release process.

      However, the only difference between the hadoop versions is the bundled flink-shaded-hadoop2 jar; the rest is completely identical.

      I propose to stop releasing hadoop-specific distributions, and instead have us release multiple versions of flink-shaded-hadoop2 that users copy into the hadoop-free distribution if required.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            chesnay Chesnay Schepler
            chesnay Chesnay Schepler
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h 20m
                1h 20m

                Slack

                  Issue deployment