From Fedora Project Wiki


Change Proposal Name

Force C.UTF-8 when Python 3 is run under the C locale

Summary

The standalone Python 3.6 binary will automatically attempt to coerce the C locale to C.utf8, unless the new PYTHONALLOWLOCALE environment variable is set to 0.

Owner

Current status

  • Targeted release: Fedora 26
  • Last updated: 2017-02-14
  • Tracker bug: <will be assigned by the Wrangler>

Detailed Description

When run under the C locale, Python 3 doesn't work properly on systems where UTF-8 is the correct encoding for interacting with the rest of the system.


This is described in detail by Armin Ronacher in the click documentation: http://click.pocoo.org/5/python3/#python-3-surrogate-handling


This proposed change for the system Python 3, assumes the current process is misconfigured when it detects that "LC_CTYPE" refers to the "C" locale, and in that case, prints a warnings to stderr and forces the use of the C.UTF-8 locale instead.


To avoid unintended side effects, it *solely* changes the actual python3.6 command line utility - nothing changes for cases where CPython is used as a dynamically linked library.


Behaviour with the patch:


$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()' 

Python detected LC_CTYPE=C. Setting LC_ALL & LANG to C.UTF-8 (set PYTHONALLOWLOCALE to disable this locale coercion behaviour).

Behaviour without the patch:

$ LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()' 

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/core.py", line 675, in main
    _verify_python3_env()
  File "/home/ncoghlan/.local/lib/python3.5/site-packages/click/_unicodefun.py", line 119, in _verify_python3_env
    'mitigation steps.' + extra)
RuntimeError: Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.  Either run this under Python 2 or consult http://click.pocoo.org/python3/ for mitigation steps.
This system supports the C.UTF-8 locale which is recommended.
You might be able to resolve your issue by exporting the
following environment variables:
export LC_ALL=C.UTF-8
export LANG=C.UTF-8

Benefit to Fedora

Scope

  • Proposal owners:

A patch will be backported to Fedora's Python 3.6 from the upstream PEP 538 which targets Python 3.7. The patch that will be backported is explicitly targeting Fedora, as the upstream equivalent one will have to be compatible with Windows, macOS, Solaris etc. The Fedora specific patch is provided by Python's core developer Nick Coghlan.

  • Release engineering: No impact with release engineering

Upgrade/compatibility impact

N/A (not a System Wide Change)

How To Test

Rawhide users can already test this change by enabling the force_c.utf8_python3 copr repository where a working build exists. 

dnf copr enable cstratak/force_c.utf8_python3

Then install python3-click: 

dnf install python3-click

And then run this command to check if python3 coerces the locale correctly to c.utf8: 

LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'

The previous behaviour can also be restored by setting the PYTHONALLOWLOCALE variable to 1, which disables the locale coersion: 

PYTHONALLOWLOCALE=1 LANG=C /usr/bin/python3 -c 'import click; cli = click.command()(lambda:None); cli()'

User Experience

N/A (not a System Wide Change)

Dependencies

No new dependencies will be added.

Contingency Plan

  • Contingency mechanism: Remove the patch from Python 3.6
  • Contingency deadline: If users experience any issues this change can be reverted at any time
  • Blocks release? No
  • Blocks product? No

Documentation

N/A (not a System Wide Change)

Release Notes

Retrieved from "https://fedoraproject.org/w/index.php?title=User:Cstratak/python3.6_c.utf-8_locale&oldid=486148"