Issue 119384 - Python update to version 2.7 or 2.6.8
Python update to version 2.7 or 2.6.8
Status: CLOSED FIXED
Product: Build Tools
Classification: Code
Component: external prerequisites
4.0.0-dev
All All
: P3 normal (vote)
: 4.0.0
Assigned To: Pedro Giffuni
:
Depends on:
Blocks: 122010
  Show dependency treegraph
 
Reported: 2012-05-19 20:18 UTC by Pedro Giffuni
Modified: 2013-07-12 11:34 UTC (History)
2 users (show)

See Also:
Issue Type: TASK
Latest Confirmation on: ---
Developer Difficulty: ---


Attachments
Work-in-Progress (151.32 KB, patch)
2012-06-06 16:00 UTC, Pedro Giffuni
no flags Details | Diff
To get function names from macro file without compiler module (2.59 KB, patch)
2012-06-24 16:34 UTC, hanya
no flags Details | Diff
Patch for linux part only to update 2.7.3 (16.43 KB, patch)
2012-06-28 16:23 UTC, hanya
no flags Details | Diff
Patch contains part for VC++ build (29.48 KB, patch)
2012-06-29 18:17 UTC, hanya
no flags Details | Diff
Python update to 2.7.3 (WIP) (212.55 KB, patch)
2012-07-25 05:32 UTC, Pedro Giffuni
no flags Details | Diff
Update Python to 2.7.3 (476.71 KB, patch)
2012-07-25 21:59 UTC, Pedro Giffuni
no flags Details | Diff
Update internal python (353.60 KB, patch)
2012-07-26 00:25 UTC, Pedro Giffuni
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this issue.
Description Pedro Giffuni 2012-05-19 20:18:32 UTC
AOO currently includes Python version 2.6.1. Most unix packages use the system python which at this time means Python 2.7.x and in a future Python 3.

Python 3 breaks compatibility with Python 2, but in order to make transition easier the python developers have done two things:

- Recent versions of Python 2.6 include some features from Python 3. Python 2.7 includes features from python 3.1.
- There are tools to convert python 2 code into python 3.

There is internal py code that can be converted to work on Python 2 and 3 but there may also have py2 extensions that we may want to keep running so updating internal Python to a 2.7.3 or 2.6.8 release is important.
Comment 1 hanya 2012-05-19 20:47:16 UTC
PyUNO uses uno.ByteSequence class as wrapper of str for []bytes. 
It has been gone on Python 3.X, so it should replaced with bytes or bytearray. 
Or maybe wrapped content should be changed from str to bytes by the ByteSequence class. 
If such kind of change is applied at the time to update Python to 2.7.X, people can prepare their code for 3.X.
Comment 2 Pedro Giffuni 2012-05-20 18:21:18 UTC
I started working on the update. Unfortunately the internal python has some patches and updating to 2.7.x and testing in all platforms is a bit more work than I want to do right now.

I will aim to version 2.6.8 and later we can run a compatibility script on pyuno to see how it goes.
Comment 3 Pedro Giffuni 2012-06-06 16:00:21 UTC
Created attachment 78072 [details]
Work-in-Progress

Just for reference.
This is a very early Work in Progress as I don't have much time to work on it right now.

The ugliest part will be the mingw support patch.
Comment 4 hanya 2012-06-24 16:34:20 UTC
Created attachment 78448 [details]
To get function names from macro file without compiler module

compiler module is deprecated on Python 2.6 and it is removed on 3.X. 
If Python updated to 2.7.X, please use ast module to get list of function names 
from macro file in pythonscript.py.
Comment 5 hanya 2012-06-28 16:23:56 UTC
Created attachment 78517 [details]
Patch for linux part only to update 2.7.3

Only Linux related part to update Python to 2.7.3. 
On 2.7.3, 
- Python-linux3.patch is not required, it looks fixed
- Python-subversion-1-7.patch is also
- Python-disable-dbm.patch also
- Python-parallel-make.patch, it looks own flag is introduced

Python-2.7.3-ssl.patch does not have _ssl.vcproj part because of I meet problem 
to patch it to the file using CRLF as line end. _ssl.vcproj file uses LF on 
Python 2.6.X but CRLF is used on 2.7.X. I do not know how to do during building process. No line number changes in _ssl.vcproj part of Python-ssl.patch, so it can be used.
Python-$(VERSION).patch contains only FreeBSD, Windows and SunOS, this patch does nothing on it.

site-package mechanism requires Makefile, config.h and header files on Python 2.7, they are packed in zipcore to place them in each correct locations.
And importlib, multiprocessing and unittest modules as directories should be added too.
Comment 6 Pedro Giffuni 2012-06-28 16:35:37 UTC
Thank you Hanya!

It is a good advance.

Unfortunately your patches have the same limitation of my original update: the one critical platform for updating python is Windows.

Linux, BSD and Solaris can use the external-python option but if the windows python doesn't work we would break world. 

As a next step, if we just get the windows patch to apply that may be
sufficient.
Comment 7 hanya 2012-06-29 18:17:32 UTC
Created attachment 78523 [details]
Patch contains part for VC++ build

Linux patch plus VC++ 2008 on Windows build.

- Patch files for VC++ build are separated in each file to make maintain easy and to avoid patch problem caused by CRLF, maybe this problem is raised only on my environment. It looks patch 2.6.1 does not have such problem on cygwin.
- md5 and sha modules are now deprecated and just like alias to functions of hashlib. misc/build/Python-2.6.1/Modules/Setup.dist file is not required to modify by Python-ssl.patch.
- If mozilla is enabled, sqlite3 module can be build but not enabled. sqlite module of Python 2.7.3 requires sqlite 3.3.3 and prebuild mozilla has sqlite 3.3.5.

Current (Python 2.6) patch has problem about ssl module to build on Linux. 
When OpenSSL is installed in the system, it is used even own OpenSSL is build. 
In my opinion, openssl should be build with own openssl when the office is not build with 
system openssl.
I tried to use ssl with socket last year on OOo 3.3 and it did not work because of version 
incompatibility or something problem.
Need more task for it.
Comment 8 Pedro Giffuni 2012-06-29 22:00:52 UTC
(In reply to comment #7)
> Created attachment 78523 [details]
> Patch contains part for VC++ build
> 
> Linux patch plus VC++ 2008 on Windows build.
> 

When applying it:

% svn patch patch-python
U         pyuno/zipcore/makefile.mk
U         python/makefile.mk
A         python/Python-2.7.3-ssl.patch
A         python/Python-2.7.3-sysbase.patch
Skipped missing target: 'python/Python-2.7.3-nohardlink.patch'
Summary of conflicts:
  Skipped paths: 1
Comment 9 Pedro Giffuni 2012-06-29 22:12:55 UTC
Comment on attachment 78523 [details]
Patch contains part for VC++ build

There is something wrong with the patch because I get this:
(Note the rejected patches.)

% ls -la
total 152
drwxr-xr-x    3 Pedro  wheel   1024 Jun 29 17:10 .
drwxr-xr-x  194 Pedro  wheel   4608 Jun 29 16:54 ..
-rw-r--r--    1 Pedro  wheel  40848 Jun 28 17:54 Python-2.6.1-mingw.patch
-rw-r--r--    1 Pedro  wheel    889 Jun 28 17:54 Python-2.6.1-nohardlink.patch
-rw-r--r--    1 Pedro  wheel    564 Jun 28 17:54 Python-2.6.1-sysbase.patch
-rw-r--r--    1 Pedro  wheel  25974 Jun 28 17:54 Python-2.6.1.patch
-rw-r--r--    1 Pedro  wheel    224 Jun 29 16:57 Python-2.7.3-ssl.patch
-rw-r--r--    1 Pedro  wheel  10836 Jun 29 16:57 Python-2.7.3-ssl.patch.svnpatch.rej
-rw-r--r--    1 Pedro  wheel     68 Jun 29 16:57 Python-2.7.3-sysbase.patch
-rw-r--r--    1 Pedro  wheel    332 Jun 29 16:57 Python-2.7.3-sysbase.patch.svnpatch.rej
-rw-r--r--    1 Pedro  wheel   1978 Jun 28 17:54 Python-disable-dbm.patch
-rw-r--r--    1 Pedro  wheel    860 Jun 28 17:54 Python-linux3.patch
-rw-r--r--    1 Pedro  wheel   1571 Jun 28 17:54 Python-parallel-make.patch
-rw-r--r--    1 Pedro  wheel   7677 Jun 28 17:54 Python-ssl.patch
-rw-r--r--    1 Pedro  wheel   2970 Jun 28 17:54 Python-subversion-1-7.patch
-rw-r--r--    1 Pedro  wheel   4766 Jun 29 16:57 makefile.mk
drwxr-xr-x    2 Pedro  wheel    512 Jun 28 17:54 prj
-rw-r--r--    1 Pedro  wheel   1675 Jun 28 17:54 pyversion.mk
Comment 10 hanya 2012-06-30 12:03:16 UTC
(In reply to comment #8)
> When applying it:
> 
> % svn patch patch-python
> U         pyuno/zipcore/makefile.mk
> U         python/makefile.mk
> A         python/Python-2.7.3-ssl.patch
> A         python/Python-2.7.3-sysbase.patch
> Skipped missing target: 'python/Python-2.7.3-nohardlink.patch'
> Summary of conflicts:
>   Skipped paths: 1
I'm sorry, it contains parts for Linux patch also.
Comment 11 hanya 2012-07-02 17:45:40 UTC
(In reply to comment #4)
> Created attachment 78448 [details]
> To get function names from macro file without compiler module
> 
> compiler module is deprecated on Python 2.6 and it is removed on 3.X. 
> If Python updated to 2.7.X, please use ast module to get list of function
> names 
> from macro file in pythonscript.py.

I need to mention about this patch. ast module is introduced on Python 2.5. 
If MacOS X version still built against system python 2.3.X, it can not be used. 
We need volunteer to update Python version on MacOS X build.
Comment 12 Pedro Giffuni 2012-07-02 19:29:11 UTC
(In reply to comment #11)
> (In reply to comment #4)
> > Created attachment 78448 [details]
> > To get function names from macro file without compiler module
> > 
> > compiler module is deprecated on Python 2.6 and it is removed on 3.X. 
> > If Python updated to 2.7.X, please use ast module to get list of function
> > names 
> > from macro file in pythonscript.py.
> 
> I need to mention about this patch. ast module is introduced on Python 2.5. 
> If MacOS X version still built against system python 2.3.X, it can not be
> used. 
> We need volunteer to update Python version on MacOS X build.

MacOS X doesnt seem to be special cased in the python module and I
would expect it to behave as in linux/BSD so I
think we could go ahead and commit this patch.

I would prefer, however to update Python first and that would mean a patch that would work for both linux and Windows (I will take care of the BSD part).
Comment 13 Pedro Giffuni 2012-07-25 05:32:07 UTC
Created attachment 78720 [details]
Python update to 2.7.3 (WIP)

This is still experimental but so far it's building (on FreeBSD)
Comment 14 Pedro Giffuni 2012-07-25 21:59:36 UTC
Created attachment 78731 [details]
Update Python to 2.7.3

I made the following changes:
- Rename Python to lowercase-only python in the filenames and updated the version number.
- updated the mingw patch so that it applies and regenerated the configure scripts it should work but is untested.
- I can reproduce hanya's issue with the ssl patch: there is more information on the ssl patch is in issue 106542. Apparently the issue is that GNU patch can't handle mixed LF and CRLF. For now I only kep the md5 part (apparently this can be used for md5 email support)
Comment 15 Pedro Giffuni 2012-07-26 00:25:21 UTC
Created attachment 78732 [details]
Update internal python

OK, This should be much cleaner.

I also figured out there was a conflict between
Python-ssl.patch and
Python-2.6.1-security-flags.patch
that was causing build issues in my previous attempts.
This version supports ssl as the original set of patches.
Comment 16 Pedro Giffuni 2012-07-26 20:24:29 UTC
(In reply to comment #15)
> Created attachment 78732 [details]
> Update internal python
> 

Committed as revision 1366182.
Comment 17 Pedro Giffuni 2012-07-27 19:22:03 UTC
In addition to the compilet --> ast replacement I
ran the reindent tool to the files in pyprov:

Sending        scripting/source/pyprov/mailmerge.py
Sending        scripting/source/pyprov/officehelper.py
Sending        scripting/source/pyprov/pythonscript.py
Transmitting file data ...
Committed revision 1366506.

We could still further improve the portability of that code
with the 2to3 script but this is certainly an improvement.
Comment 18 SVN Robot 2012-08-01 13:40:10 UTC
"af" committed SVN revision 1367986 into trunk:
#i119384# Fixed update sites for python source tarball.
Comment 19 Andre 2012-08-01 13:46:55 UTC
Fixed the download sites for the python source tarball.

Nowadays main/external_deps.lst uses a fixed revision for downloading external tarballs from the SVN server.  I changed the revision for the python tarball to 1366182, the revision in which version 2.7.3 was added.  I also added, with higher priority, the python download server and apache-extras.

The resulting list of URLS searched for the tarball is:
    http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2
    $(OOO_EXTRAS)$(MD5)-$(name)
    http://svn.apache.org/repos/asf/!svn/bc/1366182/incubator/ooo/trun/ext_sources/$(MD5)-$(name)
Comment 20 hdu@apache.org 2013-07-12 11:34:26 UTC
Obsoleted by the update to Python 2.7.5 in issue 122010