From Fedora Project Wiki
mNo edit summary
 
(23 intermediate revisions by 4 users not shown)
Line 5: Line 5:
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->
<!-- A sentence or two summarizing what this change is and what it will do. This information is used for the overall changeset summary page for each release. -->


At the present time, running <code>sudo pip3</code> in Fedora is not safe. It can overwrite files managed by dnf and generally break the Python 3 interpreter. We propose a series of measures that will make it safe to use.
At the present time, running <code>sudo pip3</code> in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter. We propose a series of measures that will make it safe to use.
 
{{admon/important|<code>sudo pip install</code> or <code>sudo pip3 install</code> is still not safe!|While this Fedora change took steps to make sudo pip installing less dangerous, there are still corner cases where it might destroy parts of your system. '''Never ever ever ever use <code>pip</code> or <code>pip3</code> with <code>sudo</code>.''' Use <code>pip --user</code> or [https://developer.fedoraproject.org/tech/languages/python/python-installation.html Python virtual environments] instead.}}


== Owner ==
== Owner ==
Line 28: Line 30:


== Current status ==
== Current status ==
* Targeted release: [[Releases/26| Fedora 26]]
* Targeted release: [[Releases/27| Fedora 27]]
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
* Last updated: <!-- this is an automatic macro — you don't need to change this line -->  {{REVISIONYEAR}}-{{REVISIONMONTH}}-{{REVISIONDAY2}}
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
<!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
Line 38: Line 40:
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
-->
-->
* Tracker bug: <will be assigned by the Wrangler>
* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1421694 #1421694]


== Detailed Description ==
== Detailed Description ==
Line 46: Line 48:
The danger of using <code>sudo pip3</code> stems from the fact that both Python dnf packages and <code>sudo pip3</code> install modules to the same location, namely <code>/usr/lib/pythonX.Y/site-packages</code>.
The danger of using <code>sudo pip3</code> stems from the fact that both Python dnf packages and <code>sudo pip3</code> install modules to the same location, namely <code>/usr/lib/pythonX.Y/site-packages</code>.


We aim to move the working directory for <code>sudo pip3</code> to a more appropriate location: <code>'''/usr/local/'''lib/pythonX.Y/site-packages</code>, and modify the Python 3 interpreter in Fedora to scan both above mentioned locations when importing modules. In addition, [[Changes/System_Python|system-python]]—a stripped down version of Python 3 for use by system tools—will not read the <code>sudo pip3</code> install location, making it more sucure by being less susceptible to interference by user-downloaded modules.
We aim to move the working directory for <code>sudo pip3</code> to a more appropriate location: <code>'''/usr/local/'''lib/pythonX.Y/site-packages</code>, and modify the Python 3 interpreter in Fedora to scan both above mentioned locations when importing modules. In addition, [[Changes/System_Python|system-python]]—a stripped down version of Python 3 for use by system tools—will not read the <code>sudo pip3</code> install location, making it more secure by being less susceptible to interference by user-downloaded modules.


From the technical standpoint, this will be accomplished by changing the <code>sys.prefix</code> setting in the <code>/usr/bin/python3</code> executable from <code>/usr/</code> to <code>/usr/local</code>. <code>pip3</code> will thereafter use this prefix when determining where to install modules. In addition, the original path <code>/usr/lib/pythonX.Y/site-packages</code> will be added to the <code>sys.path</code> variable (so that modules at that location are still processed when importing), because this path will not be automatically scanned anymore as it no longer lies inside the <code>sys.prefix</code> path. These settings, however, will not be modified for the [[Changes/System_Python|system-python]] binary, and the <code>%{__python3}</code> macro will be changed from <code>/usr/bin/python3</code> to <code>/usr/libexec/system-python</code>. Therefore, Python dnf packages will continue to be built with the correct installation path for system modules.
From the technical standpoint, this will be accomplished by changing the install prefix setting of the distutils install command in the <code>/usr/bin/python3</code> executable from <code>/usr/</code> to <code>/usr/local</code>. <code>pip3</code> and <code>distutils</code> will thereafter use this prefix when determining where to install modules. In addition, the paths <code>/usr/local/lib/pythonX.Y/site-packages</code> and  <code>/usr/local/lib64/pythonX.Y/site-packages</code> will be added to the front of the <code>sys.path</code> variable so that modules are imported preferentially from there. These settings, however, will not be modified for the [[Changes/System_Python|system-python]] binary, the <code>/usr/bin/python3</code> executable when running with -I option specified, nor when an RPM build is detected. Therefore, Python RPM packages will continue to be built with the correct installation path for system modules.


Note that using <code>sudo pip3</code> is not strictly necessary, as using <code>pip3 install --user</code> would satisfy the vast majority of use cases. Nevertheless, <code>sudo pip</code> is far too prevalent an instruction in various guides and installation notes throughout the Internet that there is little hope of changing users' behaviour in this regard.
The purpose of this change is not to make <code>sudo pip</code> a standard way to install Python packages. Virtual environments and <code>pip3 install --user</code> should still be the prefered options. Nevertheless, <code>sudo pip</code> is far too prevalent an instruction in various guides and installation notes throughout the Internet that there is little hope of changing users' behaviour in this regard.
 
'''Important:''' The changes made during the [[Changes/System Python|System Python]] change were reverted as part of the [[Changes/Platform Python Stack|Platform Python Stack]] change and will be replaced by a different concept. The specific behaviour of system-python binary described above, will be exactly the same for the platform-python binary. Also, that one was later removed as well. The other (now only) option to make sure system tools won't be affected by any package installed by sudo pip3 command is to use some of the flags provided by Python interpreter (-I, -s) in its shebang.


== Benefit to Fedora ==
== Benefit to Fedora ==
Line 56: Line 60:
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->
<!-- What is the benefit to the platform?  If this is a major capability update, what has changed?  If this is a new functionality, what capabilities does it bring? Why will Fedora become a better distribution or project because of this proposal?-->


Fedora users will benefit from the increased stability of the ecosystem. No longer will they wonder why their Python interpreter or system modules don't work after using <code>sudo pip3</code>.
Fedora users will benefit from the increased stability of the ecosystem. No longer will they wonder why their Python interpreter or system modules don't work after using <code>sudo pip</code> (see [https://bugzilla.redhat.com/show_bug.cgi?id=1400581], [https://bugzilla.redhat.com/show_bug.cgi?id=1400377], [https://bugzilla.redhat.com/show_bug.cgi?id=1397575], [https://bugzilla.redhat.com/show_bug.cgi?id=1396158]).
 
In particular, <code>sudo pip3 uninstall contextlib2</code> should completely undo the effects of <code>sudo pip3 install contextlib2</code>.
The following operations should be completely indistinguishable at the system level from not having done anything at all (except in the sudo logs):
 
<code>
$ sudo pip3 install contextlib2
 
$ sudo pip3 uninstall contextlib2
</code>
 
Additionally, Fedora will increase it's compliance with the [https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard Filesystem Hierarchy Standard],
as user-installed host-specific Python modules will now be correctly located under <code>/usr/local</code>.


== Scope ==
== Scope ==
* Proposal owners:
* Proposal owners:
** Modify the Python 3 executable as described above.
** Modify the distutils install command as described above.
** Modify the <code>%{__python3}</code> macro so that it points to <code>/usr/libexec/system-python</code>
** Modify the site.py script to add additional paths to sys.path when it is needed.
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Other developers: Spec files that use <code>pip3 install</code> without the use of a macro will need to be modified accordingly. Only 3 like packages were identified (python-flit, python-entrypoints, python-setuptools).
 
* Other developers: N/A (not a System Wide Change)  
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
<!-- What work do other developers have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->


* Release engineering: A rebuild of all Python packages will be necessary.
* Release engineering: [https://pagure.io/releng/issue/6820 ticket]
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  If a rel-eng ticket exists, add a link here.
<!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuid required?  If a rel-eng ticket exists, add a link here.
Please work with releng prior to feature submission, and ensure that someone is on board to do any process development work and testing; don't just assume that a bullet point in a change puts someone else on the hook.-->
Please work with releng prior to feature submission, and ensure that someone is on board to do any process development work and testing; don't just assume that a bullet point in a change puts someone else on the hook.-->
** [[Fedora_Program_Management/ReleaseBlocking/Fedora26|List of deliverables]]: All Fedora deliverables will be affected in a minor way that does not jeopardize their delivery.
** [[Fedora_Program_Management/ReleaseBlocking/Fedora26|List of deliverables]]: N/A (not a System Wide Change)
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->
<!-- Please check the list of Fedora release deliverables and list all the differences the feature brings -->


* Policies and guidelines: The definition of the <code>%{__python3}</code> macro will be updated as mentioned above.
* Policies and guidelines: N/A (not a System Wide Change)
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->
<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. -->


Line 106: Line 123:
3. Run <code>python3</code> and try to import said module.
3. Run <code>python3</code> and try to import said module.


4. Run <code>/usr/libexec/system-python</code> and try to import said module.
4. Run <code>python3 -I</code> and try to import said module.


=== Test <code>sudo pip3</code>–installed modules ===
=== Test <code>sudo pip3</code>–installed modules ===
Line 115: Line 132:
3. Run <code>python3</code> and try to import said module.
3. Run <code>python3</code> and try to import said module.


4. Run <code>/usr/libexec/system-python</code> and try to import said module—the import should fail.
4. Run <code>python3 -I</code> and try to import said module—the import should fail.


== User Experience ==
== User Experience ==
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
<!-- If this change proposal is noticeable by its target audience, how will their experiences change as a result?  Describe what they will see or notice. -->
Running <code>sudo pip3</code> won't randomly result in a broken Python 3 on the user's system.
Running <code>sudo pip3</code> won't randomly result in a broken Python 3 stack on the user's system.


== Dependencies ==
== Dependencies ==
Line 139: Line 156:


=== Discussions ===
=== Discussions ===
0. TODO: Email thread about sudo pip install
0. [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/3ZB2AZ77WN53E3XOB4AU7XKCJOJVHHHE/ python-devel Fedora mailing list (introduction)]
 
1. [https://lists.fedoraproject.org/archives/list/python-devel@lists.fedoraproject.org/thread/KXPVVL4J2UGHDANP76DGBYVBTGWILOJN python-devel Fedora mailing list (implementation)]
 
2. [https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0 pypa-dev mailing list]


1. https://groups.google.com/forum/#!topic/pypa-dev/r6qsAmJl9t0


== Release Notes ==
== Release Notes ==
Line 150: Line 170:
-->
-->


The location where <code>sudo pip3</code> installs modules has been changed to <code>/usr/local/lib/pythonX.Y/site-packages</code>, and is henceforth safe to use. No other changes in user experience are expected.
The location where <code>sudo pip3</code> installs modules has been changed to <code>/usr/local/lib/pythonX.Y/site-packages</code>, and <code>sudo pip3</code> is henceforth safe to use. No other changes in user experience are expected.


[[Category:ChangePageIncomplete]]
[[Category:ChangeAcceptedF27]]
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- When your change proposal page is completed and ready for review and announcement -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
<!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
Line 159: Line 179:


<!-- Select proper category, default is Self Contained Change -->
<!-- Select proper category, default is Self Contained Change -->
[[Category:SystemWideChange]]
[[Category:SelfContainedChange]]

Latest revision as of 13:00, 11 April 2018

Making sudo pip Safe (Again)

Summary

At the present time, running sudo pip3 in Fedora is not safe. Pip shares its installation directory with dnf, can remove dnf-managed files and generally break the Python 3 interpreter. We propose a series of measures that will make it safe to use.

sudo pip install or sudo pip3 install is still not safe!
While this Fedora change took steps to make sudo pip installing less dangerous, there are still corner cases where it might destroy parts of your system. Never ever ever ever use pip or pip3 with sudo. Use pip --user or Python virtual environments instead.

Owner

Current status

Detailed Description

The danger of using sudo pip3 stems from the fact that both Python dnf packages and sudo pip3 install modules to the same location, namely /usr/lib/pythonX.Y/site-packages.

We aim to move the working directory for sudo pip3 to a more appropriate location: /usr/local/lib/pythonX.Y/site-packages, and modify the Python 3 interpreter in Fedora to scan both above mentioned locations when importing modules. In addition, system-python—a stripped down version of Python 3 for use by system tools—will not read the sudo pip3 install location, making it more secure by being less susceptible to interference by user-downloaded modules.

From the technical standpoint, this will be accomplished by changing the install prefix setting of the distutils install command in the /usr/bin/python3 executable from /usr/ to /usr/local. pip3 and distutils will thereafter use this prefix when determining where to install modules. In addition, the paths /usr/local/lib/pythonX.Y/site-packages and /usr/local/lib64/pythonX.Y/site-packages will be added to the front of the sys.path variable so that modules are imported preferentially from there. These settings, however, will not be modified for the system-python binary, the /usr/bin/python3 executable when running with -I option specified, nor when an RPM build is detected. Therefore, Python RPM packages will continue to be built with the correct installation path for system modules.

The purpose of this change is not to make sudo pip a standard way to install Python packages. Virtual environments and pip3 install --user should still be the prefered options. Nevertheless, sudo pip is far too prevalent an instruction in various guides and installation notes throughout the Internet that there is little hope of changing users' behaviour in this regard.

Important: The changes made during the System Python change were reverted as part of the Platform Python Stack change and will be replaced by a different concept. The specific behaviour of system-python binary described above, will be exactly the same for the platform-python binary. Also, that one was later removed as well. The other (now only) option to make sure system tools won't be affected by any package installed by sudo pip3 command is to use some of the flags provided by Python interpreter (-I, -s) in its shebang.

Benefit to Fedora

Fedora users will benefit from the increased stability of the ecosystem. No longer will they wonder why their Python interpreter or system modules don't work after using sudo pip (see [1], [2], [3], [4]).

In particular, sudo pip3 uninstall contextlib2 should completely undo the effects of sudo pip3 install contextlib2. The following operations should be completely indistinguishable at the system level from not having done anything at all (except in the sudo logs):

$ sudo pip3 install contextlib2

$ sudo pip3 uninstall contextlib2

Additionally, Fedora will increase it's compliance with the Filesystem Hierarchy Standard, as user-installed host-specific Python modules will now be correctly located under /usr/local.

Scope

  • Proposal owners:
    • Modify the distutils install command as described above.
    • Modify the site.py script to add additional paths to sys.path when it is needed.


  • Other developers: N/A (not a System Wide Change)
  • Policies and guidelines: N/A (not a System Wide Change)
  • Trademark approval: Not needed for this Change

Upgrade/compatibility impact

Upgraded systems will continue to work as expected. Modules previously installed using sudo pip3 will remain in the location now reseved for system modules, and will need to be manually deleted if removal is required. However, they won't pose a threat to the stability of the system.

How To Test

Test dnf-installed modules

1. Install any Python 3 module using dnf.

2. Check if a corresponding file or directory was created at /usr/lib/pythonX.Y/site-packages/.

3. Run python3 and try to import said module.

4. Run python3 -I and try to import said module.

Test sudo pip3–installed modules

1. Install any Python 3 module using sudo pip3

2. Check if a corresponding file or directory was created at /usr/local/lib/pythonX.Y/site-packages/.

3. Run python3 and try to import said module.

4. Run python3 -I and try to import said module—the import should fail.

User Experience

Running sudo pip3 won't randomly result in a broken Python 3 stack on the user's system.

Dependencies

N/A

Contingency Plan

  • Contingency mechanism: Do not push changes to python3 and Python macros into rawhide.
  • Contingency deadline: Alpha freeze
  • Blocks release? No

Documentation

Discussions

0. python-devel Fedora mailing list (introduction)

1. python-devel Fedora mailing list (implementation)

2. pypa-dev mailing list


Release Notes

The location where sudo pip3 installs modules has been changed to /usr/local/lib/pythonX.Y/site-packages, and sudo pip3 is henceforth safe to use. No other changes in user experience are expected.

Retrieved from "https://fedoraproject.org/w/index.php?title=Changes/Making_sudo_pip_safe&oldid=515700"