Issue 86251

Summary: Python systemPathToFileUrl does not like national characters - approved
Product: App Dev Reporter: bmarcelly <marcelly>
Component: scriptingAssignee: joergbudi
Status: CLOSED FIXED QA Contact: Unknown <non-migrated>
Severity: trivial    
Priority: P3 CC: darabos.daniel, frank.schoenheit, issues, joerg.skottke, maison.godard
Version: OOo 3.3 or older   
Target Milestone: ---   
Hardware: All   
OS: Windows XP   
Issue Type: DEFECT Latest Confirmation on: ---
Developer Difficulty: ---
Issue Depends on:    
Issue Blocks: 88258    
Attachments:
Description Flags
patch for pythonscript.py
none
Patched version for pythonscript.py for OOo-2.4.0
none
Patched version pythonloader.py in case you installed OOo to a path with non-ascii charcters for OOo-2.4.0
none
function "checkForPythonPathBesideComponent" also needs encfile
none
version with "checkForPythonPathBesideComponent" patched none

Description bmarcelly 2008-02-20 06:22:18 UTC
Python script bug. Works OK for other scripts (Basic, Beanshell, Javascript...)

On Windows XP my userid is "Propriétaire" (was initialized by my PC vendor).
I cannot run any Python user script because character é is in the file path.
Contents of the log.text :

Wed Feb 20 07:03:38 2008 [ERROR] Error while evaluating file:///C:/
Documents%20and%20Settings/Propri%C3%A9taire/Application%20Data/OpenOffice.org2/user/
Scripts/python/Nothing.py:exceptions.UnicodeEncodeError: 'ascii' codec can't encode 
character u'\xe9' in position 32: ordinal not in range(128)
  C:\Program Files\OpenOffice.org 2.4\program\pythonscript.py:149 in function 
checkForPythonPathBesideScript() [if 1 == os.access( path, os.F_OK) and not path in 
sys.path:]
  C:\Program Files\OpenOffice.org 2.4\program\pythonscript.py:289 in function 
getModuleByUrl() [checkForPythonPathBesideScript( url[0:url.rfind('/')] )]
  C:\Program Files\OpenOffice.org 2.4\program\pythonscript.py:443 in function 
getChildNodes() [self.module = self.provCtx.getModuleByUrl( self.uri )]
Comment 1 kay.ramme 2008-02-20 07:57:51 UTC
Joerg, please take care of this ...
Comment 2 bmarcelly 2008-03-04 15:11:22 UTC
Here is a correction proposal. Tested only on my WindowsXP computer.

pyuno routines systemPathToFileUrl and fileUrlToSystemPath should not be used.

1) Correction of uno.py -> change the existing functions by these versions:

def systemPathToFileUrl( systemPath ):
    "returns a file-url for the given system path"
    sv = 
getComponentContext().ServiceManager.createInstance("com.sun.star.ucb.FileContentProvider")
    ret = sv.getFileURLFromSystemPath("", systemPath)
    return ret

def fileUrlToSystemPath( url ):
    "returns a system path (determined by the system, the python interpreter is 
running on)"
    sv = 
getComponentContext().ServiceManager.createInstance("com.sun.star.ucb.FileContentProvider")
    ret = sv.getSystemPathFromFileURL(url)
    return ret.encode(sys.getfilesystemencoding())


2) Correction of unohelper.py -> change the existing functions by these versions:

def systemPathToFileUrl( systemPath ):
    "returns a file-url for the given system path"
    return uno.systemPathToFileUrl( systemPath )

def fileUrlToSystemPath( url ):
    "returns a system path (determined by the system, the python interpreter is 
running on)"
    return uno.fileUrlToSystemPath( url )


3) Improvement of pythonscript.py :
it would be more efficient to call the uno.fileUrlToSystemPath() and 
uno.systemPathToFileUrl instead of the unohelper.py equivalent routines.
Comment 3 joergbudi 2008-03-06 20:00:45 UTC
Hi,

bug is accepted, but I have to think a little more about it. You can avoid the
Exception by replacing the line 

        if 1 == os.access( path, os.F_OK) and not path in sys.path:

with 

        if 1 == os.access( path.encode(sys.getdefaultencoding()), os.F_OK) and
not path in sys.path:

 (your proposed patch included this already). It helps on my system, can you
check whether this would help on your system as well ? 

I don't want to use a ucb service in uno.py, because atm uno.py does not depend
on these services and pyuno shall be able to run standalone without an office.

There are more bugs, the encode() must be added at all places, where a path is
passed to a python function (e.g. adding it to sys.path).

Additionally there seem to be problems when manipulating a unicode file url with
python string concatination, when I use non ascii characters as a script file
name, it appears garbled in the ui ...

Bye, 

Joerg
Comment 4 joergbudi 2008-03-11 22:35:43 UTC
Hi,

ok, this is a bug/feature in python itself, see

http://bugs.python.org/issue790000

and fixed for python 2.4. The os.access function does not do the correct
conversions for unicode strings. Thus the correct fix is to replace

os.access( path, os.F_OK)

with 

os.access( path.encode(sys.getfilesystemencoding()), os.F_OK))

. Additionally there are problems with the compile function. Attaching a patch
for pythonscript.py (still need to verify this on windows though).

Joerg
Comment 5 joergbudi 2008-03-11 22:36:59 UTC
Created attachment 52039 [details]
patch for pythonscript.py
Comment 6 bmarcelly 2008-03-17 16:42:08 UTC
Hi,
Sorry for the late answer, no time left.

Tested the patch with a user script, OK for me.
Comment 7 joergbudi 2008-04-03 08:04:33 UTC
Created attachment 52484 [details]
Patched version for pythonscript.py for OOo-2.4.0
Comment 8 joergbudi 2008-04-03 08:05:37 UTC
Created attachment 52485 [details]
Patched version pythonloader.py in case you installed OOo to a path with non-ascii charcters for OOo-2.4.0
Comment 9 joergbudi 2008-04-03 08:07:30 UTC
added patched versions of pythonscript.py and pythonloader.py to make it easier
for users to update an OO-2.4.0 installation. Simply replace the files in the
program directory of the office installation.

Joerg
Comment 10 Daniel Darabos 2008-04-06 11:00:35 UTC
Dear Joerg,

Thank you for this quick fix. As developers of several Python extensions, this
bug hit us quite hard :).

However a user has reported that the fix does not work for him, and now that I
took a look at it, it is indeed not complete. Let me attach one more patch
(applying to the already patched version) and a version with this patch.
Comment 11 Daniel Darabos 2008-04-06 11:01:28 UTC
Created attachment 52589 [details]
function "checkForPythonPathBesideComponent" also needs encfile
Comment 12 Daniel Darabos 2008-04-06 11:04:00 UTC
Created attachment 52590 [details]
version with "checkForPythonPathBesideComponent" patched
Comment 13 joergbudi 2008-04-16 18:24:31 UTC
so just to summarize, the following files in source tree need to be patched

* pyuno/source/loader/pythonloader.py  (with
http://www.openoffice.org/nonav/issues/showattachment.cgi/52590/pythonloader.py )

* scripting/source/pyprov/pythonscript.py ( with
http://www.openoffice.org/nonav/issues/showattachment.cgi/52484/pythonscript.py )

@QA: The patches can be tested this way
- Install office to a path with non-ascii characters
- Start writer
- Tools/Macro.../Run Macro...
- Choose OpenOffice.org Macros/TableSample
- If in the "Macro name" box appears a createTable(), the test succeeded.
Comment 14 Daniel Darabos 2008-04-21 09:34:19 UTC
jbu: It is not just the path to OpenOffice.org that matters. QA should also try
installing a Python language extension when their user name have special
characters. I think this exercises a different code-path.
Comment 15 joergbudi 2008-04-21 20:15:54 UTC
@cyhawk: checking the standard sample script will be enough. It both checks the 
changes in pythonloader.py (can a python component be loaded ?) and 
pythonscript.py (can the sample script be executed ?). Both files are loaded 
from absolute pathes, so no reason to force the qa people to have a certain 
localized operating system installed. Let me know by mail when u disagree. If 
this works, it will work on a localized os either.
Comment 16 Frank Schönheit 2008-04-23 21:06:51 UTC
fix committed to CWS pyunosystempaths, on behalf of jbu
Comment 17 Frank Schönheit 2008-04-24 10:03:22 UTC
targeting to 2.4.1, subject to the approval of the Release Status Team.

fs->jbu: please verify in CWS pyunosystempaths. Installation sets can be found
at
http://ooo.services.openoffice.org/pub/OpenOffice.org/cws/upload/pyunosystempaths/.

If anybody else involved in the issue is willing to verify the issue - by
downloading and installing the CWS build from the above location, and checking
the bug is indeed fixed -, then this would be welcome, too. In this case, please
set the issue to VERIFIED in case you have the permissions to do so. Thanks.
Comment 18 joergbudi 2008-04-24 20:44:56 UTC
installation sets work fine for me on both platforms -> verified. 

Will set the cws to approved for qa on 27th April in case no additional negative
comments are raised.
Comment 19 joerg.skottke 2008-04-29 12:12:26 UTC
CC myself, this issue can be automated
Comment 20 uwe.luebbers 2008-04-29 13:16:05 UTC
added "approved" to the title, because it will be easier to work with the 2.4.1 meta issue 
during release status meetings.
Comment 21 thorsten.ziehm 2009-07-20 14:54:51 UTC
This issue is closed automatically and wasn't rechecked in a current version of
OOo. The fixed issue should be integrated in OOo since more than half a year. If
you think this issue isn't fixed in a current version (OOo 3.1), please reopen
it and change the field 'Target Milestone' accordingly.

If you want to download a current version of OOo =>
http://download.openoffice.org/index.html
If you want to know more about the handling of fixed/verified issues =>
http://wiki.services.openoffice.org/wiki/Handle_fixed_verified_issues