Python 3 C.UTF-8 locale
Force the C.UTF-8 locale when Python 3 is run under the C locale.
Summary
The standalone Python 3.6 binary will automatically attempt to coerce the C locale to C.UTF-8, unless the new PYTHONCOERCECLOCALE environment variable is set to 0.
Owner
- Name: Charalampos Stratakis
- Name: Nick Coghlan
- Email: cstratak AT redhat.com
- Email: ncoghlan AT redhat.com
- Release notes owner:
Current status
Detailed Description
When run under the C locale, Python 3 doesn't work properly on systems where UTF-8 is the correct encoding for interacting with the rest of the system. This proposed change for Python 3 packaged in Fedora assumes the current locale is misconfigured when it detects that "LC_TYPE" refers to the "C" locale, and in that case, prints a warnings to stderr and forces the use of the C.UTF-8 locale instead.
To avoid unintended side effects, this change takes effect *solely* when Python 3 is invoked by the user—nothing changes in cases where CPython is used as a dynamically linked library. Also, an "escape hatch" environment variable is provided to revert to the old behavior.
The effects of the problem for a popular CLI construction library are explained on Armin Ronacher's blog: http://click.pocoo.org/5/python3/#python-3-surrogate-handling
The problem is described in detail in PEP (Python Enhancement Proposal) 538: https://www.python.org/dev/peps/pep-0538/
The PEP has no strong opposition upstream, but acceptance is held back by considerations for other systems (e.g. those that do not have a C.UTF-8 locale). These considerations do not apply for Fedora. We are commited to follow upstream conversation and recommendations around PEP 538. Also, we hope that testing this in Fedora will help upstream find remaining distro integration issues.
The current version of the PEP specifically encourages backporting to Python 3.6.0: https://www.python.org/dev/peps/pep-0538/#backporting-to-python-3-6-0
Behaviour with the patch:
$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()' Python detected LC_CTYPE=C: LC_ALL & LANG coerced to C.UTF-8 (set another locale or PYTHONCOERCECLOCALE=0 to disable this locale coercion behaviour).
Behaviour without the patch:
$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
return self.main(*args, **kwargs)
File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main
_verify_python3_env()
File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment. Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8
Benefit to Fedora
Linux container technologies like Docker, Kubernetes, and OpenShift are best known for their use in web service development, the related container formats and execution models are also being adopted for Linux command line application development. Technologies like Gnome Flatpak further aim to bring these same techniques to Linux GUI application development.
When using Python 3 for application development in these contexts, it isn't uncommon to see various text encoding related errors.
One good example is the base Docker image for Fedora which does not have any specific locale set, so the POSIX locale is set by default, which is an alias for the ASCII-based default C locale.
The simplest way to get Python 3 (regardless of the exact version) to behave sensibly in Fedora containers is to run it in the C-UTF-8 locale that the distro provides.
Another benefit, directly related to packaging rpm's, is that the locale inside mock chroots will be automatically set to C.UTF-8 when the python interpreter is invoked during rpmbuild (e.g. when running tests), so packagers won't have to rely on various hacks in order to export the appropriate locale required for their package to build.
Scope
- Proposal owners:
A patch will be backported to Fedora's Python 3.6 from the upstream PEP 538 which targets Python 3.7. The patch that will be backported is explicitly targeting Fedora, as the upstream equivalent one will have to be compatible with Windows, macOS, Solaris etc. The Fedora specific patch is provided by Nick Coghlan, a Python core developer.
- Release engineering: No impact with release engineering
Upgrade/compatibility impact
N/A (not a System Wide Change)
How To Test
Rawhide users can already test this change by enabling the force_c.utf8_python3 copr repository where a working build exists.
dnf copr enable cstratak/force_c.utf8_python3
Then install python3-click:
dnf install python3-click
And then run this command to check if python3 coerces the locale correctly to c.utf8:
LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
The previous behaviour can also be restored by setting the PYTHONCOERCECLOCALE variable to 0, which disables the locale coersion:
PYTHONCOERCECLOCALE=0 LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'
User Experience
N/A (not a System Wide Change)
Dependencies
No new dependencies will be added.
Contingency Plan
- Contingency mechanism: Remove the patch from Python 3.6
- Contingency deadline: If users experience any issues this change can be reverted at any time
- Blocks release? No
- Blocks product? No
Documentation
N/A (not a System Wide Change)