Avro
  1. Avro
  2. AVRO-849

Python unittests don't work if avro is already installed

    Details

    • Type: Bug Bug
    • Status: Patch Available
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 1.4.1
    • Fix Version/s: None
    • Component/s: python
    • Labels:
      None
    • Tags:
      python, unittests

      Description

      I think there is a problem running the python unittests if avro is already installed. If avro is already installed then when we import avro or its subpackages in the unittests we can end up importing the installed avro instead of the avro version being compiled because the installed version might be higher on the path.

      One (slightly cumbersome way to fix this) would be to pass the the paths listed in test.path to our unittests (e.g by using an environment variable). The unittests could then adjust the path so these directories were highest on the path. The downside of this solution would be every unittest would need some boilerplate code to adjust the path.

      Does anyone have suggestions about how to pass the values in test.path to our unittests? I don't know anything about ant, so any help would be appreciated.

      Or does anyone have suggestions about a better solution which doesn't require adjusting the path in each unittest?

      1. AVRO-849.patch
        4 kB
        Jeremy Lewi
      2. AVRO-849.patch
        3 kB
        Jeremy Lewi

        Activity

        Hide
        Jeremy Lewi added a comment -

        I renamed the module to "set_avro_test_path.py" to avoid using the JIRA issue number. I also removed all references to the JIRA # in the comments.

        Show
        Jeremy Lewi added a comment - I renamed the module to "set_avro_test_path.py" to avoid using the JIRA issue number. I also removed all references to the JIRA # in the comments.
        Hide
        Doug Cutting added a comment -

        This seems to be a well-known Python problem with no easy solution.

        http://www.velocityreviews.com/forums/t716589-pythonpath-and-eggs.html
        http://stackoverflow.com/questions/897792/pythons-sys-path-value

        I'd be okay with the patch here if we change it to instead of 'avro_849_fix' to be something descriptive like 'avro-test-path' or somesuch.

        Show
        Doug Cutting added a comment - This seems to be a well-known Python problem with no easy solution. http://www.velocityreviews.com/forums/t716589-pythonpath-and-eggs.html http://stackoverflow.com/questions/897792/pythons-sys-path-value I'd be okay with the patch here if we change it to instead of 'avro_849_fix' to be something descriptive like 'avro-test-path' or somesuch.
        Hide
        Jeremy Lewi added a comment -

        I'm not sure exactly how the path precedence is determined from PYTHONPATH but on my system it looks like "eggs" appear higher on the path then directories obtained from PYTHONPATH.

        A simple test

        export PYTHONPATH=/some/dir
        python -c 'import sys; print "\n".join(sys.path)'
        

        In my case the output is

        /usr/lib/python2.7/site-packages/ipython-0.11-py2.7.egg
        /usr/lib/python2.7/site-packages/avro-1.6.0_SNAPSHOT-py2.7.egg
        /some/dir
        /usr/lib64/python27.zip
        /usr/lib64/python2.7
        ...
        

        So at least in my case, the installed avro egg will take precedence over (lang/py/

        {src,test,lib}

        ) directories added via PYTHONPATH.

        Show
        Jeremy Lewi added a comment - I'm not sure exactly how the path precedence is determined from PYTHONPATH but on my system it looks like "eggs" appear higher on the path then directories obtained from PYTHONPATH. A simple test export PYTHONPATH=/some/dir python -c ' import sys; print "\n" .join(sys.path)' In my case the output is /usr/lib/python2.7/site-packages/ipython-0.11-py2.7.egg /usr/lib/python2.7/site-packages/avro-1.6.0_SNAPSHOT-py2.7.egg /some/dir /usr/lib64/python27.zip /usr/lib64/python2.7 ... So at least in my case, the installed avro egg will take precedence over (lang/py/ {src,test,lib} ) directories added via PYTHONPATH.
        Hide
        Doug Cutting added a comment -

        I'm confused. The values of test.path (lang/py/

        {src,test,lib}

        ) become the value of PYTHONPATH, which should supercede anything installed, no?

        Show
        Doug Cutting added a comment - I'm confused. The values of test.path (lang/py/ {src,test,lib} ) become the value of PYTHONPATH, which should supercede anything installed, no?
        Hide
        Jeremy Lewi added a comment -

        Here is a patch.
        I created a module avro_849_fix.py which adjusts the path based on the location of the module. SO unittests just need to add the following at the top

        import avro_849_fix
        
        Show
        Jeremy Lewi added a comment - Here is a patch. I created a module avro_849_fix.py which adjusts the path based on the location of the module. SO unittests just need to add the following at the top import avro_849_fix
        Hide
        Jeremy Lewi added a comment -

        Sorry I thought submit patch was the way to attach a patch.

        Show
        Jeremy Lewi added a comment - Sorry I thought submit patch was the way to attach a patch.
        Hide
        Jeremy Lewi added a comment -

        Here is a patch to address this issue. At the start of each unittest you add the code

        import avro_fix_849
        

        This causes the path to be adjusted based on the location of the unittest

        Show
        Jeremy Lewi added a comment - Here is a patch to address this issue. At the start of each unittest you add the code import avro_fix_849 This causes the path to be adjusted based on the location of the unittest

          People

          • Assignee:
            Jeremy Lewi
            Reporter:
            Jeremy Lewi
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development