Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5749

unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.3.0, 1.2.1
    • Component/s: None
    • Labels:
      None

      Description

      Currently when we are trying to build flink on a machine with HADOOP_HOME environment variable set, Test data will be written to HDFS, instead of local tmp dir which is expected. This will cause tests failed.
      I suggest unset HADOOP_HOME and HADOOP_CONF_DIR environment variable in pom, to make sure maven run the test cases in the exactly way we want.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user wenlong88 opened a pull request:

          https://github.com/apache/flink/pull/3288

          FLINK-5749unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT

          Currently when we are trying to build flink on a machine with HADOOP_HOME environment variable set, Test data will be written to HDFS, instead of local tmp dir which is expected. This will cause tests failed.
          This PR unsets HADOOP_HOME and HADOOP_CONF_DIR environment variable in pom, to make sure maven run the test cases in the exactly way we want.

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/wenlong88/flink jira-5749

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/flink/pull/3288.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #3288


          commit d0e6464f37e2057c508a0e7858d3b42778610637
          Author: wenlong.lwl <wenlong.lwl@alibaba-inc.com>
          Date: 2017-01-13T06:12:17Z

          to #9896513 Unset HADOOP_HOME and HADOOP_CONF_DIR in pom

          Summary:
          unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT

          fix T9896513

          Test Plan: UT/IT

          Reviewers: 黎钢

          Reviewed By: 黎钢

          Subscribers: P577102

          Differential Revision: https://aone.alibaba-inc.com/code/D96861


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user wenlong88 opened a pull request: https://github.com/apache/flink/pull/3288 FLINK-5749 unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT Currently when we are trying to build flink on a machine with HADOOP_HOME environment variable set, Test data will be written to HDFS, instead of local tmp dir which is expected. This will cause tests failed. This PR unsets HADOOP_HOME and HADOOP_CONF_DIR environment variable in pom, to make sure maven run the test cases in the exactly way we want. You can merge this pull request into a Git repository by running: $ git pull https://github.com/wenlong88/flink jira-5749 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/3288.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3288 commit d0e6464f37e2057c508a0e7858d3b42778610637 Author: wenlong.lwl <wenlong.lwl@alibaba-inc.com> Date: 2017-01-13T06:12:17Z to #9896513 Unset HADOOP_HOME and HADOOP_CONF_DIR in pom Summary: unset HADOOP_HOME and HADOOP_CONF_DIR to avoid env in build machine failing the UT and IT fix T9896513 Test Plan: UT/IT Reviewers: 黎钢 Reviewed By: 黎钢 Subscribers: P577102 Differential Revision: https://aone.alibaba-inc.com/code/D96861
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3288

          I think this is a good addition.

          We could try and make this change more targeted to the Maven project that has the affected tests. That would help us keeping the root pom.xml more compact. For example if this affects only tests in `flink-fs-tests` then we can for example make this change in `flink-fs-tests`.

          Can you share what tests are affected by this?

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3288 I think this is a good addition. We could try and make this change more targeted to the Maven project that has the affected tests. That would help us keeping the root pom.xml more compact. For example if this affects only tests in `flink-fs-tests` then we can for example make this change in `flink-fs-tests`. Can you share what tests are affected by this?
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wenlong88 commented on the issue:

          https://github.com/apache/flink/pull/3288

          hi @StephanEwen, the cases in flink-filesystem-connector will fail first in our building machine. I have go through the cases in flink-fs-tests, I think they will fail too if I remove the change from the root pom. I think it is better to set in the root pom, because the HadoopUtils is in the package flink-java, and many modules have dependency on flink-java and users of the utils may have no idea that the HadoopUtils have dependency on the environment variables that will cause wired failures and it is hard to trace.

          Show
          githubbot ASF GitHub Bot added a comment - Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/3288 hi @StephanEwen, the cases in flink-filesystem-connector will fail first in our building machine. I have go through the cases in flink-fs-tests, I think they will fail too if I remove the change from the root pom. I think it is better to set in the root pom, because the HadoopUtils is in the package flink-java, and many modules have dependency on flink-java and users of the utils may have no idea that the HadoopUtils have dependency on the environment variables that will cause wired failures and it is hard to trace.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3288

          For the `HadoopUtils` in `flink-java` - I think those need to depend on the variables and are also not in the test scope.

          The reason why I am asking also do make the change more to `filesystem-connector` and `fs-tests` is that we want to gradually make Flink depend less and less on Hadoop, so to move all Hadoop dependencies into separate optional modules. More targeted environment variable exclusion would fit that design better.

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3288 For the `HadoopUtils` in `flink-java` - I think those need to depend on the variables and are also not in the test scope. The reason why I am asking also do make the change more to `filesystem-connector` and `fs-tests` is that we want to gradually make Flink depend less and less on Hadoop, so to move all Hadoop dependencies into separate optional modules. More targeted environment variable exclusion would fit that design better.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wenlong88 commented on the issue:

          https://github.com/apache/flink/pull/3288

          ok, I will move the changes to filesystem-connector and fs-tests poms.

          Show
          githubbot ASF GitHub Bot added a comment - Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/3288 ok, I will move the changes to filesystem-connector and fs-tests poms.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user wenlong88 commented on the issue:

          https://github.com/apache/flink/pull/3288

          hi @StephanEwen , I have move the changes to child module pom, `flink-tests` and `flink-hadoop-compatibility` need to be changed too, because the mapreduce related cases will load output compaction setting from the cluster configuration.

          Show
          githubbot ASF GitHub Bot added a comment - Github user wenlong88 commented on the issue: https://github.com/apache/flink/pull/3288 hi @StephanEwen , I have move the changes to child module pom, `flink-tests` and `flink-hadoop-compatibility` need to be changed too, because the mapreduce related cases will load output compaction setting from the cluster configuration.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user StephanEwen commented on the issue:

          https://github.com/apache/flink/pull/3288

          Looks good, thank you!
          Merging this...

          Show
          githubbot ASF GitHub Bot added a comment - Github user StephanEwen commented on the issue: https://github.com/apache/flink/pull/3288 Looks good, thank you! Merging this...
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/flink/pull/3288

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/3288
          Hide
          StephanEwen Stephan Ewen added a comment -

          Fixed in

          • 1.2.1 via 24aff09ce51a6e90cd6c39d89a3f2c5b7d6f4423
          • 1.3.0 via 48109378bc337736c7469de0fe08ec12f7a8f76f
          Show
          StephanEwen Stephan Ewen added a comment - Fixed in 1.2.1 via 24aff09ce51a6e90cd6c39d89a3f2c5b7d6f4423 1.3.0 via 48109378bc337736c7469de0fe08ec12f7a8f76f

            People

            • Assignee:
              wenlong.lwl Wenlong Lyu
              Reporter:
              wenlong.lwl Wenlong Lyu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development