Talk:Features/Python 3.3

Packages that have not yet been rebuilt in koji

I'm working on these. See notes below

Dependency chain of the builds is:

• python-sphinx
  • python-execnet
  • python-manuel
• scipy
  • python-networkx
• gcc-python-plugin

python-manuel

https://admin.fedoraproject.org/updates/python-manuel-1.6.0-3.fc18

python-sphinx

https://admin.fedoraproject.org/updates/FEDORA-2012-12707/python-sphinx-1.1.3-5.fc18

python-execnet

python-execnet-1.1-3.fc18 built against 3.3

scipy

Cython sources need regenerating

  File "orthogonal_eval.pyx", line 93, in init scipy.special.orthogonal_eval (scipy/special/orthogonal_eval.c:4126)
ValueError: level must be >= 0

(also some swig issues, I think)

python-networkx

BR: python3-scipy

gcc-python-plugin

https://admin.fedoraproject.org/updates/gcc-python-plugin-0.9-4.fc18

Packages that have been rebuilt in koji

python3
python-setuptools
deltarpm
dreampie
lcgdm
libselinux
libsemanage
loook
metagoofil
mpi4py
mysql-connector-python
nordugrid-arc
pygpgme
pyke
pyparsing
python3-cairo
python3-chardet
python3-postgresql
python-asciitable
python-auth-credential
python-cmd2
python-crypto
python-dirq
python-httplib2
python-inotify
python-messaging
python-mock
python-pbs
python-ply
python-psutil
python-requests
python-restauth-common
python-restauth
python-six
python-sleekxmpp
python-straight-plugin
revisor
ris-linux
sip
stomppy
uwsgi
waf
pygobject3
PyQt4
ATpy
babel
bpython
nfoview
pyshp
python3-cherrypy
python3-dateutil
python3-zope-fixers
python-anyjson
python-coverage
python-cvxopt
python-decorator
python-enchant
python-feedparser
python-flexmock
python-html5lib
python-lxml
python-markupsafe
python-minimock
python-nose-cover3
python-pip
python-pkgwat-api
python-profilehooks
python-pyro
python-simplegeneric
python-tempita
python-tornado
python-urwid
python-zope-event
PyYAML
python-beautifulsoup4
python-BeautifulSoup
python-jinja2
python-easygui
python-smbc
python-smbpasswd
python-py
pycmd
autoarchive
python-psycopg2
python-mglob
python-nose
python-beaker
python-markdown
python-pygments
python-amqplib
python-cement
python-zmq
python-mako
python-markdown2
pytest
numpy
pyfits
pyp2rpm
python-ansi2html
python-setproctitle
python-nose1.1
python-docutils

Scope of python3 in Fedora 18

Getting python3-using packages:

$ for f in $(repoquery --whatrequires python3) ; do echo $f ; repoquery --source $f ; done > sources.txt
$ grep src.rpm sources.txt | sort | uniq | wc -l
130

but it's showing some duplicates; adding filtering shows 105 src.rpms:

$ grep src.rpm sources.txt | python -c "import sys, re; print('\n'.join([re.match('(.*)-(.*)-(.*).src.rpm', line).group(1) for line in sys.stdin.readlines()]))"| sort |uniq
ATpy
babel
bpython
cobbler
deltarpm
dreampie
lcgdm
libselinux
libsemanage
loook
metagoofil
mpi4py
mysql-connector-python
nfoview
nordugrid-arc
numpy
pycmd
pyfits
pygobject3
pyke
pyp2rpm
pyparsing
PyQt4
pyshp
pytest
python3
python3-cairo
python3-chardet
python3-cherrypy
python3-dateutil
python3-postgresql
python3-zope-fixers
python-amqplib
python-ansi2html
python-anyjson
python-asciitable
python-auth-credential
python-beaker
python-BeautifulSoup
python-beautifulsoup4
python-cmd2
python-coverage
python-crypto
python-cvxopt
python-decorator
python-dirq
python-docutils
python-easygui
python-enchant
python-execnet
python-feedparser
python-flexmock
python-html5lib
python-httplib2
python-inotify
python-jinja2
python-lxml
python-mako
python-manuel
python-markdown
python-markdown2
python-markupsafe
python-messaging
python-mglob
python-minimock
python-mock
python-networkx
python-nose
python-nose1.1
python-pbs
python-pip
python-pkgwat-api
python-ply
python-psutil
python-psycopg2
python-py
python-pygments
python-pyro
python-requests
python-restauth
python-restauth-common
python-setproctitle
python-setuptools
python-simplegeneric
python-six
python-sleekxmpp
python-smbc
python-smbpasswd
python-sphinx
python-sqlalchemy
python-straight-plugin
python-tempita
python-tornado
python-urwid
python-zmq
python-zope-event
PyYAML
revisor
ris-linux
scipy
sip
stomppy
uwsgi
waf
znc

What rebuilds cleanly

After building "python3" from the "python3.3" branch, mockchain built the following successfully in order: (using this script to obtain the build order):

[david@surprise python3.3]$ grep Success localrepo/mockchain*.log |python -c"import sys, re; print '\n'.join([re.match('.*Success building (.+)', line).group(1) for line in sys.stdin.readlines()])"

cobbler-2.2.3-3.fc18.src.rpm
deltarpm-3.6-0.9.20110223git.fc18.src.rpm
dreampie-1.1.1-2.fc18.src.rpm
lcgdm-1.8.3.1-4.fc18.src.rpm
libselinux-2.1.11-4.fc18.src.rpm
libsemanage-2.1.8-4.fc18.src.rpm
loook-0.6.7-8.fc18.src.rpm
metagoofil-1.4b-5.fc18.src.rpm
mpi4py-1.3-3.fc18.src.rpm
mysql-connector-python-0.3.2-4.fc18.src.rpm
nordugrid-arc-2.0.0-3.fc18.src.rpm
pygpgme-0.3-2.fc18.src.rpm
pyke-1.1.1-8.fc18.src.rpm
pyparsing-1.5.6-4.fc18.src.rpm
python3-cairo-1.10.0-3.fc18.src.rpm
python3-chardet-2.0.1-7.fc18.src.rpm
python3-postgresql-1.0.2-3.fc18.src.rpm
python-asciitable-0.8.0-3.fc18.src.rpm
python-auth-credential-0.6-2.fc18.src.rpm
python-cmd2-0.6.4-3.fc18.src.rpm
python-crypto-2.6-2.fc18.src.rpm
python-dirq-1.2.2-2.fc18.src.rpm
python-httplib2-0.7.4-5.fc18.src.rpm
python-inotify-0.9.3-2.fc18.src.rpm
python-messaging-0.10-2.fc18.src.rpm
python-mock-0.8.0-3.fc18.src.rpm
python-pbs-4.3.3-2.fc18.src.rpm
python-ply-3.4-2.fc18.src.rpm
python-psutil-0.5.1-2.fc18.src.rpm
python-requests-0.13.1-2.fc18.src.rpm
python-restauth-common-0.5.1-5.fc18.src.rpm
python-restauth-0.5.1-5.fc18.src.rpm
python-six-1.1.0-3.fc18.src.rpm
python-sleekxmpp-1.0-0.10.beta2.fc18.src.rpm
python-straight-plugin-1.1.1-0.9.20111110.git57ef11c.fc18.src.rpm
revisor-2.2-5.fc18.src.rpm
ris-linux-0.4-10.fc18.src.rpm
sip-4.13.3-2.fc18.src.rpm
stomppy-3.0.5-2.fc18.src.rpm
uwsgi-1.2.4-2.fc18.src.rpm
waf-1.7.0-1.fc18.1.src.rpm
pygobject3-3.3.4-1.fc18.src.rpm
PyQt4-4.9.4-2.fc18.src.rpm
python-setuptools-0.6.28-1.fc18.src.rpm
ATpy-0.9.6-2.fc18.src.rpm
babel-0.9.6-3.fc18.src.rpm
bpython-0.11-2.fc18.src.rpm
nfoview-1.10-4.fc18.src.rpm
pyshp-1.1.4-4.fc18.src.rpm
python3-cherrypy-3.2.0-0.rc1.r2567.1.fc18.5.src.rpm
python3-dateutil-2.0-3.fc18.src.rpm
python3-zope-fixers-1.0-3.fc18.src.rpm
python-anyjson-0.3.1-4.fc18.src.rpm
python-coverage-3.5.2-0.2.b1.fc18.src.rpm
python-cvxopt-1.1.5-3.fc18.src.rpm
python-decorator-3.3.3-2.fc18.src.rpm
python-enchant-1.6.5-6.fc18.src.rpm
python-feedparser-5.1.2-3.fc18.src.rpm
python-flexmock-0.9.4-2.fc18.src.rpm
python-html5lib-0.95-2.fc18.src.rpm
python-lxml-2.3.3-2.fc18.src.rpm
python-markupsafe-0.11-5.fc18.src.rpm
python-minimock-1.2.7-3.fc18.src.rpm
python-nose-cover3-0.1.0-3.fc18.src.rpm
python-pip-1.1-2.fc18.src.rpm
python-pkgwat-api-0.3-4.fc18.src.rpm
python-profilehooks-1.6-2.fcf18.src.rpm
python-pyro-4.9-3.fc18.src.rpm
python-simplegeneric-0.8-4.fc18.src.rpm
python-tempita-0.5.1-2.fc18.src.rpm
python-tornado-2.2.1-2.fc18.src.rpm
python-urwid-1.0.0-4.fc18.src.rpm
python-zope-event-3.5.1-3.fc18.src.rpm
PyYAML-3.10-4.fc18.src.rpm
python-beautifulsoup4-4.1.1-4.fc18.src.rpm
python-BeautifulSoup-3.2.1-2.fc18.src.rpm
python-jinja2-2.6-3.fc18.src.rpm
python-easygui-0.96-4.fc18.src.rpm
python-smbc-1.0.13-3.fc18.src.rpm
python-smbpasswd-1.0.1-20.fc18.src.rpm
python-py-1.4.9-2.fc18.src.rpm
pycmd-1.0-6.fc18.src.rpm
pytest-2.2.4-2.fc18.src.rpm
python-execnet-1.1-2.fc18.src.rpm
python-nose-1.1.2-3.fc18.src.rpm
python-ansi2html-0.9.1-6.fc18.src.rpm
python-beaker-1.5.4-4.fc18.src.rpm
python-mako-0.5.0-3.fc18.src.rpm
python-markdown-2.2.0-2.fc18.src.rpm
python-pygments-1.4-5.fc18.src.rpm
python-markdown2-2.0.0-2.fc18.src.rpm
python-amqplib-1.0.2-5.fc18.src.rpm
python-psycopg2-2.4.5-5.fc18.src.rpm
python-cement-1.9.14-4.fc18.src.rpm
python-zmq-2.2.0-4.fc18.src.rpm

Fixes that were needed

Intermittent failures

I had to apply this fix for a localhost-using test in python-setuptools; this change isn't in git yet:

[david@surprise python-setuptools]$ git diff
diff --git a/python-setuptools.spec b/python-setuptools.spec
index 1dc1a9a..db84cb3 100644
--- a/python-setuptools.spec
+++ b/python-setuptools.spec
@@ -21,7 +21,7 @@ Source2:        zpl.txt
 Patch0: distribute-different-exception-message.patch
 # Sometimes this times out in the build system.  Hanging onto the patch in git
 # for a bit in case that behavior returns
-#Patch1: distribute-timeout-exception.patch
+Patch1: distribute-timeout-exception.patch
 
 BuildRoot:      %{_tmppath}/%{name}-%{version}-%{release}-root-%(%{__id_u} -n)
 
@@ -68,7 +68,7 @@ This package contains the distribute fork of setuptools.
 %setup -q -n %{srcname}-%{version}
 
 %patch0 -p1 -b .excmsg
-#patch1 -p1 -b .exctype
+%patch1 -p1 -b .exctype
 
 find -name '*.txt' | xargs chmod -x
 find . -name '*.orig' -exec rm \{\} \;

Packaging tweaks

 DONE: python-easygui/python-easygui-0.96-3.fc18.src.rpm
   RPM build errors:
     File not found by glob: /builddir/build/BUILDROOT/python-easygui-0.96-3.fc18.x86_64/usr/lib/python3.3/site-packages/__pycache__/easygui.cpython-32.py*
   Test built in local mock; fix pushed to git master
 DONE: python-smbc/python-smbc-1.0.13-2.fc18.src.rpm
   RPM build errors:
     File not found by glob: /builddir/build/BUILDROOT/python-smbc-1.0.13-2.fc18.x86_64/usr/lib64/python3.3/site-packages/smbc.cpython-3?mu.so
   Test built in local mock; fix pushed to git master
 DONE: python-smbpasswd/python-smbpasswd-1.0.1-19.fc18.src.rpm
    RPM build errors:
      File not found by glob: /builddir/build/BUILDROOT/python-smbpasswd-1.0.1-19.fc18.x86_64/usr/lib64/python3.3/site-packages/smbpasswd.cpython-3?mu.so
   Test built in local mock; fix pushed to git master
 DONE: python-amqplib/python-amqplib-1.0.2-4.fc18.src.rpm
    RPM build errors:
      nosetests-3.2: command not found
    Test built in local mock; fix pushed to git master
 DONE: python-psycopg2/python-psycopg2-2.4.5-2.fc18.src.rpm
     + for python in python python-debug python3 python3.2dmu
     + python3.2dmu setup.py build
     /var/tmp/rpm-tmp.ZyG8xO: line 34: python3.2dmu: command not found
   (also some fileglobs)

Breaking a circular dep

python-py.spec has:

# we have a circular (build) dependency with the (new) pytest package
# when generating the docs or running the testsuite
%global with_docs 1
%global run_check 1

Disabling both, regening python-py src.rpm and trying again:

[david@surprise python-py]$ git diff
diff --git a/python-py.spec b/python-py.spec
index ef7ad93..8f0eb69 100644
--- a/python-py.spec
+++ b/python-py.spec
@@ -6,8 +6,8 @@
 
 # we have a circular (build) dependency with the (new) pytest package
 # when generating the docs or running the testsuite
-%global with_docs 1
-%global run_check 1
+%global with_docs 0
+%global run_check 0
 
 %global pytest_version 2.2.4

__import__(, , , level=-1)

In python prior to 3.3. this means "The default is -1 which indicates both absolute and relative imports will be attempted." (c.f. http://docs.python.org/library/functions.html#__import__)

In python 3.3, 0 became the default, meaning to only perform absolute imports and "Changed in version 3.3: Negative values for level are no longer supported." (c.f. http://docs.python.org/dev/library/functions.html#__import__ )

DONE: python-cement/python-cement-1.9.14-2.fc18.src.rpm
  Reported as https://github.com/cement/cement/issues/156; have applied downstream patch

DONE: python-zmq/python-zmq-2.2.0-2.fc18.src.rpm
  File "error.pyx", line 34, in init zmq.core.error (zmq/core/error.c:1195)
     ValueError: level must be >= 0
  .c files generated with old Cython; upon forcing regeneration of .c files it works

Unicode issues

DONE: numpy/numpy-1.6.2-2.fc18.src.rpm
  Needed porting: has various things that use internals of unicode
  Upstream has ported; we've cherrypicked patches
    http://projects.scipy.org/numpy/ticket/1471
    (also http://projects.scipy.org/numpy/ticket/2145 )
    Discussed here: http://www.mail-archive.com/numpy-discussion@scipy.org/msg38344.html

Dict ordering issues

Python 3.3 enables hash-randomization by default, making the order of keys in dicts potentially change from run to run.

FIXED: python-ansi2html had an implicit reliance on dict ordering in its selftests.

Temporary Fixes

python-nose

I turned off the %check in python-nose in order to build the deps before having the full selftest suite for nose itself passing.

python-docutils

Tests fail (against 3.3b2): (failures=1, errors=4)

3 failures in test_writers.test_odt.DocutilsOdtTestCase: with "TypeError: expected an Element, not _ElementInterfaceWrapper" odf writer fails when run with Python3.3b1/b2: filed as https://sourceforge.net/tracker/?func=detail&aid=3555164&group_id=38414&atid=422030

Hash-randomization dict ordering issue; sent patch upstream as http://sourceforge.net/tracker/?func=detail&aid=3555160&group_id=38414&atid=422032

Other failures reported as https://sourceforge.net/tracker/?func=detail&aid=3561133&group_id=38414&atid=422030

Remaining work

SWIG issues

TODO: znc/znc-0.206-2.fc18.src.rpm

swig errors

release notes

added to release notes at https://fedoraproject.org/wiki/Documentation_Development_Beat Immanetize (talk) 17:21, 26 October 2012 (UTC)