Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-4813

Having flink-test-utils as a dependency outside Flink fails the build

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.2.0
    • Fix Version/s: 1.3.0, 1.2.1
    • Component/s: Build System
    • Labels:
      None

      Description

      The flink-test-utils depend on hadoop-minikdc, which has a dependency, which is only resolvable, if the maven-bundle-plugin is loaded.

      This is the error message

      [ERROR] Failed to execute goal on project quickstart-1.2-tests: Could not resolve dependencies for project com.dataartisans:quickstart-1.2-tests:jar:1.0-SNAPSHOT: Failure to find org.apache.directory.jdbm:apacheds-jdbm1:bundle:2.0.0-M2 in https://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
      

      flink-parent loads that plugin, so all "internal" dependencies to the test utils can resolve the plugin.
      Right now, users have to use the maven bundle plugin to use our test utils externally.

      By making the hadoop minikdc dependency optional, we can probably resolve the issues. Then, only users who want to use the security-related tools in the test utils need to manually add the hadoop minikdc dependency + the plugin.

        Issue Links

          Activity

          Hide
          mxm Maximilian Michels added a comment -

          +1 That sounds like a good solution.

          Show
          mxm Maximilian Michels added a comment - +1 That sounds like a good solution.
          Hide
          StephanEwen Stephan Ewen added a comment -

          +1

          Show
          StephanEwen Stephan Ewen added a comment - +1
          Hide
          NicoK Nico Kruber added a comment -

          for documentation: until the minikdc dependency is made optional, users are required to add the following to their build plugins:

          <!--
            https://issues.apache.org/jira/browse/DIRSHARED-134
            Required to pull the Mini-KDC transitive dependency
          -->
          <plugin>
          	<groupId>org.apache.felix</groupId>
          	<artifactId>maven-bundle-plugin</artifactId>
          	<version>3.0.1</version>
          	<inherited>true</inherited>
          	<extensions>true</extensions>
          </plugin>
          
          Show
          NicoK Nico Kruber added a comment - for documentation: until the minikdc dependency is made optional, users are required to add the following to their build plugins: <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Required to pull the Mini-KDC transitive dependency --> <plugin> <groupId> org.apache.felix </groupId> <artifactId> maven-bundle-plugin </artifactId> <version> 3.0.1 </version> <inherited> true </inherited> <extensions> true </extensions> </plugin>
          Hide
          StephanEwen Stephan Ewen added a comment -

          Nico Kruber Do you want to go ahead and patch this?

          Show
          StephanEwen Stephan Ewen added a comment - Nico Kruber Do you want to go ahead and patch this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user NicoK opened a pull request:

          https://github.com/apache/flink/pull/3322

          FLINK-4813[flink-test-utils] make the hadoop-minikdc dependency optional

          This removes the need to add the `maven-bundle-plugin`plugin for most projects using `flink-test-utils`.

          Instead, any project using `flink-test-utils` that also requires
          `SecureTestEnvironment` must add a dependency to `hadoop-minikdc` itself, e.g. in
          `pom.xml`:

          ```xml
          ...
          <dependencies>
          <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-minikdc</artifactId>
          <version>$

          {minikdc.version}</version>
          <scope>compile</scope>
          </dependency>
          ...
          </dependencies>
          ...

          <build>
          <plugins>
          <!--
          https://issues.apache.org/jira/browse/DIRSHARED-134
          Required to pull the Mini-KDC transitive dependency
          -->
          <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>3.0.1</version>
          <inherited>true</inherited>
          <extensions>true</extensions>
          </plugin>
          ...
          ```

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/NicoK/flink flink-4813

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3322.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3322

          ----
          commit db3da8bbdcee1b1a72bd7638864b7d052c7f1c83
          Author: Nico Kruber <nico@data-artisans.com>
          Date: 2017-02-15T13:24:32Z

          FLINK-4813[flink-test-utils] make the hadoop-minikdc dependency optional

          With this change, any project using flink-test-utils which also requires
          SecureTestEnvironment must add a dependency to hadoop-minikdc itself, e.g. in
          pom.xml:

          ...
          <dependencies>
          <dependency>
          <groupId>org.apache.hadoop</groupId>
          <artifactId>hadoop-minikdc</artifactId>
          <version>${minikdc.version}

          </version>
          <scope>compile</scope>
          </dependency>
          ...
          </dependencies>
          ...

          <build>
          <plugins>
          <!--
          https://issues.apache.org/jira/browse/DIRSHARED-134
          Required to pull the Mini-KDC transitive dependency
          -->
          <plugin>
          <groupId>org.apache.felix</groupId>
          <artifactId>maven-bundle-plugin</artifactId>
          <version>3.0.1</version>
          <inherited>true</inherited>
          <extensions>true</extensions>
          </plugin>
          ...


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user NicoK opened a pull request: https://github.com/apache/flink/pull/3322 FLINK-4813 [flink-test-utils] make the hadoop-minikdc dependency optional This removes the need to add the `maven-bundle-plugin`plugin for most projects using `flink-test-utils`. Instead, any project using `flink-test-utils` that also requires `SecureTestEnvironment` must add a dependency to `hadoop-minikdc` itself, e.g. in `pom.xml`: ```xml ... <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>$ {minikdc.version}</version> <scope>compile</scope> </dependency> ... </dependencies> ... <build> <plugins> <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Required to pull the Mini-KDC transitive dependency --> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <version>3.0.1</version> <inherited>true</inherited> <extensions>true</extensions> </plugin> ... ``` You can merge this pull request into a Git repository by running: $ git pull https://github.com/NicoK/flink flink-4813 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3322.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3322 ---- commit db3da8bbdcee1b1a72bd7638864b7d052c7f1c83 Author: Nico Kruber <nico@data-artisans.com> Date: 2017-02-15T13:24:32Z FLINK-4813 [flink-test-utils] make the hadoop-minikdc dependency optional With this change, any project using flink-test-utils which also requires SecureTestEnvironment must add a dependency to hadoop-minikdc itself, e.g. in pom.xml: ... <dependencies> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-minikdc</artifactId> <version>${minikdc.version} </version> <scope>compile</scope> </dependency> ... </dependencies> ... <build> <plugins> <!-- https://issues.apache.org/jira/browse/DIRSHARED-134 Required to pull the Mini-KDC transitive dependency --> <plugin> <groupId>org.apache.felix</groupId> <artifactId>maven-bundle-plugin</artifactId> <version>3.0.1</version> <inherited>true</inherited> <extensions>true</extensions> </plugin> ...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3322

          Very good, I had stumbled over that problem as well.

          Could you also remove the bundle plugin from the root pom (this should speed up dependency resolutions in IDEs by a lot) and instead add the above described dependency to the test projects that use the `SecureTestEnvironment`?

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3322 Very good, I had stumbled over that problem as well. Could you also remove the bundle plugin from the root pom (this should speed up dependency resolutions in IDEs by a lot) and instead add the above described dependency to the test projects that use the `SecureTestEnvironment`?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user NicoK commented on the issue:

          https://github.com/apache/flink/pull/3322

          sure, that makes sense
          actually, I only had to add it to the flink-test-utils sub-project since all the others already included the bundler

          Show
          githubbot ASF GitHub Bot added a comment - Github user NicoK commented on the issue: https://github.com/apache/flink/pull/3322 sure, that makes sense actually, I only had to add it to the flink-test-utils sub-project since all the others already included the bundler
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on a diff in the pull request:

          https://github.com/apache/flink/pull/3322#discussion_r101565321

          — Diff: flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/SecureTestEnvironment.java —
          @@ -37,9 +37,39 @@
          /**

          • Helper {@link SecureTestEnvironment}

            to handle MiniKDC lifecycle.

          • This class can be used to start/stop MiniKDC and create secure configurations for MiniDFSCluster
          • * and MiniYarn
            + * and MiniYarn.
            + *
            + * If you use this class in your project, please make sure to add a dependency to
            + * <tt>hadoop-minikdc</tt>, e.g. in your <tt>pom.xml</tt>:
            + * <pre>
            + * ...
            + * <dependencies>
              • End diff –

          If you do the example code as follows, you can use the `<` and make the sample more readable.
          ```
          <pre>

          {@code <dependency> <groupId>org.apache.hadoop</groupId> ... </dependency> }

          </pre>
          ```

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on a diff in the pull request: https://github.com/apache/flink/pull/3322#discussion_r101565321 — Diff: flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/SecureTestEnvironment.java — @@ -37,9 +37,39 @@ /** Helper {@link SecureTestEnvironment} to handle MiniKDC lifecycle. This class can be used to start/stop MiniKDC and create secure configurations for MiniDFSCluster * and MiniYarn + * and MiniYarn. + * + * If you use this class in your project, please make sure to add a dependency to + * <tt>hadoop-minikdc</tt>, e.g. in your <tt>pom.xml</tt>: + * <pre> + * ... + * <dependencies> End diff – If you do the example code as follows, you can use the `<` and make the sample more readable. ``` <pre> {@code <dependency> <groupId>org.apache.hadoop</groupId> ... </dependency> } </pre> ```
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3322

          The change looks good, thank you!
          Merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3322 The change looks good, thank you! Merging this...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3322

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3322
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in

          • 1.2.1 via 2c08f7548ad8388ab081f82cd6f6c916c8900ee0
          • 1.3.0 via 391efd35bffe3f5796cd655ae1598f102a2e8fb7
          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 1.2.1 via 2c08f7548ad8388ab081f82cd6f6c916c8900ee0 1.3.0 via 391efd35bffe3f5796cd655ae1598f102a2e8fb7

            People

            • Assignee:
              NicoK Nico Kruber
              Reporter:
              rmetzger Robert Metzger
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development