Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-14306

flink-python build fails with No module named pkg_resources

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Benchmark builds started to fail with

      [INFO] Adding generated sources (java): /home/jenkins/workspace/flink-master-benchmarks/flink/flink-python/target/generated-sources
      [INFO] 
      [INFO] --- exec-maven-plugin:1.5.0:exec (Protos Generation) @ flink-python_2.11 ---
      Traceback (most recent call last):
        File "/home/jenkins/workspace/flink-master-benchmarks/flink/flink-python/pyflink/gen_protos.py", line 33, in <module>
          import pkg_resources
      ImportError: No module named pkg_resources
      [ERROR] Command execution failed.
      (...)
      [INFO] flink-state-processor-api .......................... SUCCESS [  0.299 s]
      [INFO] flink-python ....................................... FAILURE [  0.434 s]
      [INFO] flink-scala-shell .................................. SKIPPED
      

      because of this ticket: https://issues.apache.org/jira/browse/FLINK-14018

      I think I can solve the benchmark builds failing quite easily by installing setuptools python package, so this ticket is not about this, but about deciding how should we treat such kind of external dependencies. I don't see this dependency being mentioned anywhere in the documentation (for example here).

      Probably at the very least those external dependencies should be documented, but also I fear about such kind of manual steps to do before building the Flink can become a problem if grow out of control. Some questions:

      1. Do we really need this dependency?
      2. Could this dependency be resolve automatically? By installing into a local python virtual environment?
      3. Should we document those dependencies somewhere?
      4. Maybe we should not build flink-python by default?
      5. Maybe we should add a pre-build script for flink-python to verify the dependencies and to throw an easy to understand error with hint how to fix it?

      CC Hequn Cheng Dian Fu Till Rohrmann Jincheng Sun

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              dianfu Dian Fu
              Reporter:
              pnowojski Piotr Nowojski

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m

                  Issue deployment