Ambari
  1. Ambari
  2. AMBARI-1779

Remove hard references to /usr/lib/pythonX.Y/site-packages from all files (shell scripts, java, poms, python)

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: infra
    • Labels:
      None
    • Environment:

      Fedora

      Description

      Hard references to /usr/lib/pythonX.Y/site-packages should be eliminated from the build process and java/runtime scripts for portability.

      Any python modules in site-packages should be importable (ie, a proper package) and will be found on the standard python path. Therefore, any code that needs to be run from those modules can be run with python wrapper scripts located in /usr/bin or /usr/sbin (for example) without an explicit path reference.

      Python code in site-packages that is not importable should be made so and run from a wrapper, or it should be relocated (to /usr/lib/exec/ambari, for example)

      1. AMBARI-1779.patch
        11 kB
        Trevor McKay

        Issue Links

          Activity

          Hide
          Trevor McKay added a comment -

          This is a patch on branches/branch-1.2 to address hard deps on /usr/lib/python*/site-packages.

          Show
          Trevor McKay added a comment - This is a patch on branches/branch-1.2 to address hard deps on /usr/lib/python*/site-packages.
          Hide
          Trevor McKay added a comment -

          A summary of changes by file in ambari_build.patch

          ambari-agent/src/main/python/ambari-machine

          Add wrapper for ambari-agent/machine.py and place in /usr/sbin/ to eliminate site-packages reference.

          ambari-agent/src/main/python/ambari-agent.py

          Add wrapper for ambari-agent/main.py and place in /usr/sbin to eliminate site-packages reference.

          ambari-agent/conf/unix/ambari-agent

          Removed python version checks, the rpm will check the version. Stop calling the main.py script out of
          site-packages and call new wrapper /usr/sbin/ambari-agent.py instead. Invoke the script directly instead
          of using "python blah".

          ambari-server/sbin/ambari-server

          Removed python version checks, the rpm will check the version. Invoke the /usr/sbin/ambari-server.py script
          directly instead of using "python blah".

          ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java

          Change default location of bootstrap.py and setupAgent.py to /usr/lib/exec/ambari-server/

          ambari-server/src/main/python/setupAgent.py

          Invoke /usr/sbin/ambari-machine wrapper instead of referencing machine.py out of site-packages.

          ambari-server/conf/unix/ambari.properties

          Set "bootstrap.script" and "bootstrap.setup_agent.script" to reference /usr/lib/exec/ambari_server/*
          instead of referencing site-packages.

          ambari-agent/pom.xml

          Add python.ver (new default), python.exec, and python.requires properties.
          Change install.dir to agent.install.dir and make the default dependent on python.ver.
          Use python.exec for maven commands, defaulted to python2
          Set location of ambari-agent.py and ambari-machine to /usr/sbin.

          ambari-server/pom.xml

          Change python.ver to be just the version number (not the requires clause)
          Add python.exec and python.requires.
          Use python.exec for maven commands, defaulted to python2
          Move bootstrap.py and setupAgent.py out of site-packages to /usr/lib/exec/ambari-server/

          ambari-web/pom.xml

          Add python.exec property to be consistent and default to python2.
          Not sure this even matters, it is an environment variable to npm.... ??

          Changes to flags:

          -Dpython.ver – Sets the python version. This value is used in the
          "requires" clause in generated rpms.

          For ambari-agent, it is also used in construction of the python lib path
          (/usr/lib/python$

          {python.ver} by default. A default was added to
          ambari-agent/pom.xml of 2.7 since other things already touched 2.7.

          In ambari-server/pom.xml it is defaulted to 2.6 but since the
          requires clause is >= it doesn't really matter. That's the only thing
          it is currently used for in ambari-server.

          So, in general when building from the top level this can be set to
          the python version for the agent host. When building from a subdir
          obviously it can be controlled individually.

          -Dpython.exec – Defaults to python2. This is the command that maven
          uses to run python scripts during the build. Python2 gets rid of the
          python2.6/2.7 problem.

          -Dpython.requires – The requires clause for rpms. Defaults to >= python.ver

          -Dagent.install.dir – This used to be install.dir but I added "agent." to make
          it clear when building from the top level. It is only used in agent.

          Default is /usr/lib/python${python.ver}

          /site-packages/ambari_agent

          Show
          Trevor McKay added a comment - A summary of changes by file in ambari_build.patch ambari-agent/src/main/python/ambari-machine Add wrapper for ambari-agent/machine.py and place in /usr/sbin/ to eliminate site-packages reference. ambari-agent/src/main/python/ambari-agent.py Add wrapper for ambari-agent/main.py and place in /usr/sbin to eliminate site-packages reference. ambari-agent/conf/unix/ambari-agent Removed python version checks, the rpm will check the version. Stop calling the main.py script out of site-packages and call new wrapper /usr/sbin/ambari-agent.py instead. Invoke the script directly instead of using "python blah". ambari-server/sbin/ambari-server Removed python version checks, the rpm will check the version. Invoke the /usr/sbin/ambari-server.py script directly instead of using "python blah". ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java Change default location of bootstrap.py and setupAgent.py to /usr/lib/exec/ambari-server/ ambari-server/src/main/python/setupAgent.py Invoke /usr/sbin/ambari-machine wrapper instead of referencing machine.py out of site-packages. ambari-server/conf/unix/ambari.properties Set "bootstrap.script" and "bootstrap.setup_agent.script" to reference /usr/lib/exec/ambari_server/* instead of referencing site-packages. ambari-agent/pom.xml Add python.ver (new default), python.exec, and python.requires properties. Change install.dir to agent.install.dir and make the default dependent on python.ver. Use python.exec for maven commands, defaulted to python2 Set location of ambari-agent.py and ambari-machine to /usr/sbin. ambari-server/pom.xml Change python.ver to be just the version number (not the requires clause) Add python.exec and python.requires. Use python.exec for maven commands, defaulted to python2 Move bootstrap.py and setupAgent.py out of site-packages to /usr/lib/exec/ambari-server/ ambari-web/pom.xml Add python.exec property to be consistent and default to python2. Not sure this even matters, it is an environment variable to npm.... ?? Changes to flags: -Dpython.ver – Sets the python version. This value is used in the "requires" clause in generated rpms. For ambari-agent, it is also used in construction of the python lib path (/usr/lib/python$ {python.ver} by default. A default was added to ambari-agent/pom.xml of 2.7 since other things already touched 2.7. In ambari-server/pom.xml it is defaulted to 2.6 but since the requires clause is >= it doesn't really matter. That's the only thing it is currently used for in ambari-server. So, in general when building from the top level this can be set to the python version for the agent host. When building from a subdir obviously it can be controlled individually. -Dpython.exec – Defaults to python2. This is the command that maven uses to run python scripts during the build. Python2 gets rid of the python2.6/2.7 problem. -Dpython.requires – The requires clause for rpms. Defaults to >= python.ver -Dagent.install.dir – This used to be install.dir but I added "agent." to make it clear when building from the top level. It is only used in agent. Default is /usr/lib/python${python.ver} /site-packages/ambari_agent
          Hide
          Trevor McKay added a comment -

          See also https://issues.apache.org/jira/browse/AMBARI-1721

          I believe the problem in 1721 is fixed by the addition of python.exec with a default of "python2"

          Show
          Trevor McKay added a comment - See also https://issues.apache.org/jira/browse/AMBARI-1721 I believe the problem in 1721 is fixed by the addition of python.exec with a default of "python2"
          Hide
          Trevor McKay added a comment -

          Note, the one thing here I didn't find a solution for was the install location for ambari-agent. There doesn't seem to be a good way to deal with the python lib path dynamically with the rpm plugin.

          Show
          Trevor McKay added a comment - Note, the one thing here I didn't find a solution for was the install location for ambari-agent. There doesn't seem to be a good way to deal with the python lib path dynamically with the rpm plugin.
          Hide
          Mahadev konar added a comment -

          Trevor I am not sure I understand, why would a default of python2 work?

          Show
          Mahadev konar added a comment - Trevor I am not sure I understand, why would a default of python2 work?
          Hide
          Trevor McKay added a comment -

          Mahadev (is that right?) are you referring to my comment on AMBARI-1721 above?

          my understanding: that issue reported by Jeff occurs on Fedora 18 (for example) where the python2.6 executable does not exist. Instead, you've got /usr/bin/python2 linked to /usr/bin/python2.7. This results in the 'Cannot run program "python2.6"'

          If maven scripts invoke python2 instead of explicitly invoking python2.x to run tests, etc, it should work regardless of whether a platform has pyhon2.6 or python2.7 installed (or even some other python2.x but that's unlikely, I think). The /usr/bin/python2 link should be standard for python installs.

          I'll spin up an F18 box and confirm. Are we on the same page? Have I missed something? Thanks.

          Show
          Trevor McKay added a comment - Mahadev (is that right?) are you referring to my comment on AMBARI-1721 above? my understanding: that issue reported by Jeff occurs on Fedora 18 (for example) where the python2.6 executable does not exist. Instead, you've got /usr/bin/python2 linked to /usr/bin/python2.7. This results in the 'Cannot run program "python2.6"' If maven scripts invoke python2 instead of explicitly invoking python2.x to run tests, etc, it should work regardless of whether a platform has pyhon2.6 or python2.7 installed (or even some other python2.x but that's unlikely, I think). The /usr/bin/python2 link should be standard for python installs. I'll spin up an F18 box and confirm. Are we on the same page? Have I missed something? Thanks.
          Hide
          Trevor McKay added a comment -

          Glad I tried this. Patch was corrupted by bad svn diff. Remaking, will update ASAP.

          Show
          Trevor McKay added a comment - Glad I tried this. Patch was corrupted by bad svn diff. Remaking, will update ASAP.
          Hide
          Trevor McKay added a comment -

          Corrected patch.

          Show
          Trevor McKay added a comment - Corrected patch.
          Hide
          Mahadev konar added a comment -

          Trevor McKay the issue is that a pyton2 executable is not standard. It might work on fedora 18. Instead of python2 why dont we just use python and just make sure its > 2.0. The patch attached will cause issues on different platforms (lets say on a mac, there is no python2). Does that make sense?

          Also for changing the structure, we will also have to do some post install changes wherein the prior configs that have some paths in there that reference old 2.6/site packages are upgraded to new ones when you upgrade the agent.

          Show
          Mahadev konar added a comment - Trevor McKay the issue is that a pyton2 executable is not standard. It might work on fedora 18. Instead of python2 why dont we just use python and just make sure its > 2.0. The patch attached will cause issues on different platforms (lets say on a mac, there is no python2). Does that make sense? Also for changing the structure, we will also have to do some post install changes wherein the prior configs that have some paths in there that reference old 2.6/site packages are upgraded to new ones when you upgrade the agent.
          Hide
          Trevor McKay added a comment -

          "the issue is that a pyton2 executable is not standard"

          Ah, my mistake. It looked to me like it was from python wikis, etc. You're right, just "python" will do the trick.

          Show
          Trevor McKay added a comment - "the issue is that a pyton2 executable is not standard" Ah, my mistake. It looked to me like it was from python wikis, etc. You're right, just "python" will do the trick.
          Hide
          Trevor McKay added a comment -

          I suppose an alternative to moving the server side py files is to actually make them into an importable module and wrap them from /usr/sbin. That might be more backwards compatible. Kind of violates the spirit of site-packages imho but may be simpler.

          Show
          Trevor McKay added a comment - I suppose an alternative to moving the server side py files is to actually make them into an importable module and wrap them from /usr/sbin. That might be more backwards compatible. Kind of violates the spirit of site-packages imho but may be simpler.
          Hide
          Trevor McKay added a comment -

          Refresh the changes against current trunk.

          Mahadev's comment on post install changes is still outstanding...

          Show
          Trevor McKay added a comment - Refresh the changes against current trunk. Mahadev's comment on post install changes is still outstanding...
          Hide
          Chad Roberts added a comment -

          +1, assuming that the above referenced "post install changes" will be addressed as well in a separate JIRA.

          Show
          Chad Roberts added a comment - +1, assuming that the above referenced "post install changes" will be addressed as well in a separate JIRA.

            People

            • Assignee:
              Trevor McKay
              Reporter:
              Trevor McKay
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development