Bigtop
  1. Bigtop
  2. BIGTOP-688

improve hue packaging via making virtual env relocatable and moving DB files into /var/lib/hue

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.4.0
    • Fix Version/s: 0.5.0
    • Component/s: general
    • Labels:
      None

      Description

      Currently Hue packaging does unfortunate things with permissions and mutability of things under /usr/lib/hue. We should make sure our virtual env is relocatable and can be used in a R/O fashion and we should move mutable bits (desktop.db and app.reg) under /var/lib/hue.

      1. BIGTOP-688.patch.txt
        16 kB
        Roman Shaposhnik

        Activity

        Hide
        Roman Shaposhnik added a comment -

        This is somewhat sizable patch, but what it does is actually pretty simple:

        1. makes the python virtual env relocatable
        2. moves the desktop.db out of /usr/lib/hue/desktop into /var/lib/hue
        3. makes everything under /usr/lib/hue owned by root and immutable

        I do have one question for the python experts out there. The way that virtual env becomes relocatable includes, among other things, changing all the shebang lines inside of env/bin/<script> into:

        #!/usr/bin/env python2.4
        

        this, of course, makes it necessary to have the current location of the env (/usr/lib/hue/build/env/bin) on the PATH (see changes to the init.d scripts). I am wondering whether doing something like this:

        -  start_daemon -u $USER $DAEMON $DAEMON_OPTS
        +  PATH=/usr/lib/hue/build/env/bin:$PATH start_daemon -u $USER $DAEMON $DAEMON_OPTS
        

        is truly the best way to make sure that the virtual env gets used. Ideas?

        Show
        Roman Shaposhnik added a comment - This is somewhat sizable patch, but what it does is actually pretty simple: makes the python virtual env relocatable moves the desktop.db out of /usr/lib/hue/desktop into /var/lib/hue makes everything under /usr/lib/hue owned by root and immutable I do have one question for the python experts out there. The way that virtual env becomes relocatable includes, among other things, changing all the shebang lines inside of env/bin/<script> into: #!/usr/bin/env python2.4 this, of course, makes it necessary to have the current location of the env (/usr/lib/hue/build/env/bin) on the PATH (see changes to the init.d scripts). I am wondering whether doing something like this: - start_daemon -u $USER $DAEMON $DAEMON_OPTS + PATH=/usr/lib/hue/build/env/bin:$PATH start_daemon -u $USER $DAEMON $DAEMON_OPTS is truly the best way to make sure that the virtual env gets used. Ideas?
        Hide
        bc Wong added a comment -

        You don't need to change the PATH. The beauty of relocating the virtualenv is that you can run it with system python. Magic.

        Show
        bc Wong added a comment - You don't need to change the PATH. The beauty of relocating the virtualenv is that you can run it with system python. Magic.
        Hide
        Roman Shaposhnik added a comment -

        What if there's no python on the system?

        Show
        Roman Shaposhnik added a comment - What if there's no python on the system?
        Hide
        bc Wong added a comment -

        Can we not make that a prereq? Having Python is a very reasonable thing. Trying to make the Hue distribution self-contained is a slippery slope. What about sasl lib, libxml2, libxslt, etc?

        Show
        bc Wong added a comment - Can we not make that a prereq? Having Python is a very reasonable thing. Trying to make the Hue distribution self-contained is a slippery slope. What about sasl lib, libxml2, libxslt, etc?
        Hide
        Roman Shaposhnik added a comment -

        An additional complication is that even on systems with the default python if I don't do the PATH setting the actual executable that is used remains to be the system python, which clearly defeats the purpose of the relocatable virtual environment to begin with (the 3 PIDs are from the top level supervisor down to the rpc server):

        // remove the path setting
        # service hue start
        # ls -l /proc/19736/exe
        lrwxrwxrwx 1 root root 0 Aug 17 12:14 /proc/19736/exe -> /usr/bin/python2.4
        # ls -l /proc/19740/exe
        lrwxrwxrwx 1 root root 0 Aug 17 12:14 /proc/19740/exe -> /usr/bin/python2.4
        # ls -l /proc/19772/exe
        lrwxrwxrwx 1 hue hue 0 Aug 17 12:14 /proc/19772/exe -> /usr/bin/python2.4
        
        Show
        Roman Shaposhnik added a comment - An additional complication is that even on systems with the default python if I don't do the PATH setting the actual executable that is used remains to be the system python, which clearly defeats the purpose of the relocatable virtual environment to begin with (the 3 PIDs are from the top level supervisor down to the rpc server): // remove the path setting # service hue start # ls -l /proc/19736/exe lrwxrwxrwx 1 root root 0 Aug 17 12:14 /proc/19736/exe -> /usr/bin/python2.4 # ls -l /proc/19740/exe lrwxrwxrwx 1 root root 0 Aug 17 12:14 /proc/19740/exe -> /usr/bin/python2.4 # ls -l /proc/19772/exe lrwxrwxrwx 1 hue hue 0 Aug 17 12:14 /proc/19772/exe -> /usr/bin/python2.4
        Hide
        Anatoli Fomenko added a comment - - edited

        Some comments on the patch:

        1. In install_hue.sh there are 2 variables related to logs: LOG_DIR and DESKTOP_LOG_DIR. While they may be required by the environment, it may be cleaner to keep it to one.
        2. In hue-app-postinstall.tpl, hue-app.prerm.tpl and other app related files kept some variables that are removed in other files, such as ROOT=/usr/lib/hue
        3. In a few places user name "hue" is used without defining USER=hue
        4. Build successful on Precise
        5. Package install successful

        Tried a few commands. Met this problem:

         /usr/lib/hue/build/env/bin]$ sudo ./hue test
        Traceback (most recent call last):
          File "./hue", line 9, in <module>
            load_entry_point('desktop==2.0.1', 'console_scripts', 'hue')()
          File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line
        60, in entry
            execute_manager(settings)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 438, in execute_manager
            utility.execute()
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 379, in execute
            self.fetch_command(subcommand).run_from_argv(self.argv)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 261, in fetch_command
            klass = load_command_class(app_name, subcommand)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 67, in load_command_class
            module = import_module('%s.management.commands.%s' % (app_name, name))
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py",
        line 35, in import_module
            __import__(name)
          File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test.py",
        line 24, in <module>
            from django_nose import nose_runner
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/django_nose-0.5-py2.7.egg/django_nose/nose_runner.py",
        line 22, in <module>
            import nose
        ImportError: No module named nose
        
        

        After installing python-django-nose:

        sudo ./hue test
        Traceback (most recent call last):
          File "./hue", line 9, in <module>
            load_entry_point('desktop==2.0.1', 'console_scripts', 'hue')()
          File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line
        60, in entry
            execute_manager(settings)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 438, in execute_manager
            utility.execute()
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 379, in execute
            self.fetch_command(subcommand).run_from_argv(self.argv)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 261, in fetch_command
            klass = load_command_class(app_name, subcommand)
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py",
        line 67, in load_command_class
            module = import_module('%s.management.commands.%s' % (app_name, name))
          File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py",
        line 35, in import_module
            __import__(name)
          File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test.py",
        line 32, in <module>
            from desktop.management.commands import test_windmill
          File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test_windmill.py",
        line 27, in <module>
            from windmill.authoring import djangotest
        ImportError: No module named windmill.authoring
        
        

        Obviously, test_windmill also fails.

        Most other commands run fine.

        Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build`?

        Show
        Anatoli Fomenko added a comment - - edited Some comments on the patch: In install_hue.sh there are 2 variables related to logs: LOG_DIR and DESKTOP_LOG_DIR. While they may be required by the environment, it may be cleaner to keep it to one. In hue-app-postinstall.tpl, hue-app.prerm.tpl and other app related files kept some variables that are removed in other files, such as ROOT=/usr/lib/hue In a few places user name "hue" is used without defining USER=hue Build successful on Precise Package install successful Tried a few commands. Met this problem: /usr/lib/hue/build/env/bin]$ sudo ./hue test Traceback (most recent call last): File "./hue", line 9, in <module> load_entry_point('desktop==2.0.1', 'console_scripts', 'hue')() File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry execute_manager(settings) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 67, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py", line 35, in import_module __import__(name) File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test.py", line 24, in <module> from django_nose import nose_runner File "/usr/lib/hue/build/env/lib/python2.7/site-packages/django_nose-0.5-py2.7.egg/django_nose/nose_runner.py", line 22, in <module> import nose ImportError: No module named nose After installing python-django-nose: sudo ./hue test Traceback (most recent call last): File "./hue", line 9, in <module> load_entry_point('desktop==2.0.1', 'console_scripts', 'hue')() File "/usr/lib/hue/desktop/core/src/desktop/manage_entry.py", line 60, in entry execute_manager(settings) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 438, in execute_manager utility.execute() File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 379, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 261, in fetch_command klass = load_command_class(app_name, subcommand) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/core/management/__init__.py", line 67, in load_command_class module = import_module('%s.management.commands.%s' % (app_name, name)) File "/usr/lib/hue/build/env/lib/python2.7/site-packages/Django-1.2.3-py2.7.egg/django/utils/importlib.py", line 35, in import_module __import__(name) File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test.py", line 32, in <module> from desktop.management.commands import test_windmill File "/usr/lib/hue/desktop/core/src/desktop/management/commands/test_windmill.py", line 27, in <module> from windmill.authoring import djangotest ImportError: No module named windmill.authoring Obviously, test_windmill also fails. Most other commands run fine. Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build`?
        Hide
        Roman Shaposhnik added a comment -

        @Anatoli Fomenko

        In install_hue.sh there are 2 variables related to logs: LOG_DIR and DESKTOP_LOG_DIR. While they may be required by the environment, it may be cleaner to keep it to one.

        That's just a convention that we've got going in our _install.sh scripts. I agree that it is a bit confusing since Hue has the same var, but it is probably best to stay consistent within the _install.sh scripts.

        In hue-app-postinstall.tpl, hue-app.prerm.tpl and other app related files kept some variables that are removed in other files, such as ROOT=/usr/lib/hue

        That's on purpose. We need this functionality to remain for things that are plugins into Hue. The model of Hue integration in Bigtop is that we prepackage all the core bits together (and thus do not need to register them dynamically) but things like beeswax and future plugins still need to run registration code at package install time.

        In a few places user name "hue" is used without defining USER=hue

        Could you, please, be a bit more specific? I could only see that in chown lines which could arguably be changed.

        Tried a few commands. Met this problem: /usr/lib/hue/build/env/bin]$ sudo ./hue test

        In general Bigtop packaging tries to remove test code from packages. Hence any residual code is not expected to work. An argument could be had that we should do a better job of completely removing hue test code, but it looks pretty tightly integrated. Do you want to take a look at it?

        Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build`?

        Which build system are you referring to? The Bigtop itself or Hue? Can you, please, be more specific?

        Show
        Roman Shaposhnik added a comment - @ Anatoli Fomenko In install_hue.sh there are 2 variables related to logs: LOG_DIR and DESKTOP_LOG_DIR. While they may be required by the environment, it may be cleaner to keep it to one. That's just a convention that we've got going in our _install.sh scripts. I agree that it is a bit confusing since Hue has the same var, but it is probably best to stay consistent within the _install.sh scripts. In hue-app-postinstall.tpl, hue-app.prerm.tpl and other app related files kept some variables that are removed in other files, such as ROOT=/usr/lib/hue That's on purpose. We need this functionality to remain for things that are plugins into Hue. The model of Hue integration in Bigtop is that we prepackage all the core bits together (and thus do not need to register them dynamically) but things like beeswax and future plugins still need to run registration code at package install time. In a few places user name "hue" is used without defining USER=hue Could you, please, be a bit more specific? I could only see that in chown lines which could arguably be changed. Tried a few commands. Met this problem: /usr/lib/hue/build/env/bin]$ sudo ./hue test In general Bigtop packaging tries to remove test code from packages. Hence any residual code is not expected to work. An argument could be had that we should do a better job of completely removing hue test code, but it looks pretty tightly integrated. Do you want to take a look at it? Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build`? Which build system are you referring to? The Bigtop itself or Hue? Can you, please, be more specific?
        Hide
        Anatoli Fomenko added a comment -

        @Roman

        In general Bigtop packaging tries to remove test code from packages. Hence any residual code is not expected to work. An argument could be had that we should do a better job of completely removing hue test code, but it looks pretty tightly integrated. Do you want to take a look at it?

        Good idea. Thanks.

        Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build`?

        Which build system are you referring to? The Bigtop itself or Hue? Can you, please, be more specific?

        Bigtop build system. Perhaps I should move this question to a separate thread, for better coverage.

        Show
        Anatoli Fomenko added a comment - @Roman In general Bigtop packaging tries to remove test code from packages. Hence any residual code is not expected to work. An argument could be had that we should do a better job of completely removing hue test code, but it looks pretty tightly integrated. Do you want to take a look at it? Good idea. Thanks. Off-topic question: why the build system does not detect change in the src files, only `rm -rf <component>/build` ? Which build system are you referring to? The Bigtop itself or Hue? Can you, please, be more specific? Bigtop build system. Perhaps I should move this question to a separate thread, for better coverage.
        Hide
        Sean Mackrory added a comment -

        +1

        Show
        Sean Mackrory added a comment - +1

          People

          • Assignee:
            Roman Shaposhnik
            Reporter:
            Roman Shaposhnik
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development