Latest revision as of 05:38, 12 April 2016

Glibc locale subpackaging

Summary

This change should make it possible to install or uninstall locales individually.

Owner

Name: Mike Fabian, Siddhesh Poyarekar, Carlos O’Donell
Email: <mfabian At redhat DOT com>, <spoyarek AT redhat DOT com>, <codonell AT redhat DOT com>
Release notes owner:

Current status

Targeted release: Fedora 24
Last updated: (2015-12-08)
Tracker bug: Bug 1238406

Detailed Description

Currently the file /usr/lib/locale/locale-archive contains all locales and is thus huge (103MB).

For small systems (and containers) it would be useful to be able to install only a small number of locales.

Recently we made it possible to install a small number of locales by supplying the rpm-macro “_install_langs”, for example

   rpm -i -D _install_langs="en:de_DE" glibc-common.rpm

will install all English locales and all German locales which start with “de_DE”,

   rpm -i -D _install_langs="en_US.utf8" glibc-common.rpm

will install only the en_US.utf8 locale,

   rpm -i -D _install_langs="POSIX" glibc-common.rpm

will install nothing (but the POSIX/C is still available because it is builtin into glibc).

But this approach works only during an Anaconda based install when Anaconda supplies the _install_langs rpm-macro.

When glibc is updated later, the _install_langs macro will not be supplied on the command line during the update and the default value “all” of “_install_langs” from /usr/lib/rpm/macros will be used and all locales come back during an update.

Therefore, this solution is far from perfect.

It should be made possible to install and uninstall locales individually, for example by having a separate package for the locales for each language. Installing such a package would add these locales to locale-archive, uninstalling it would remove them.

Anaconda then needs to be changed to handle such language packages.

Other implementation options

Benefit to Fedora

This change should make it possible to reduce the minimum size of an installation considerably.

Scope

Proposal owners:
1. Figure out the best approach to to install/uninstall locales
2. Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)

Other developers:

Anaconda needs to be made aware of the new approach to handle installation/uninstallation of locales

Release engineering:

I am not sure whether this has affects release engineering, probably the packages in the install image change when parts are split out of glibc-common.

Policies and guidelines:

No, this change does not require updates to policies and guidelines.

Trademark approval: not needed for this Change

Upgrade/compatibility impact

The user should not notice any change after the update. All locales which were installed before should still be installed after the update.

After the update it should be possible to uninstall locales for unneeded languages but this is optional.

How To Test

Please visit How to Test page and participate in Fedora 24 i18n testday

Try to install/uninstall packages containing locales for some languages.
- For example,
```
dnf install glibc-langpack-hi
```
  should install the Hindi locales
```
locale -a | grep ^hi
```
  should then show that the Hindi loclales are installed.
Installing/uninstalling the meta packages langpacks-<language> should automatically install/uninstall the respective glibc packages.
- For example,
```
dnf install langpacks-hi
```
  should install some stuff useful for Hindi including glibc-langpack-hi.
```
dnf remove langpacks-hi
```
  should remove the Hindi stuff again and should also remove glibc-langpack-hi.
There should be a package glibc-all-langpacks, installing that should install all available locales into /usr/lib/locales/locale-archive
- For example, after
```
dnf install glibc-all-langpacks
```
  the file
```
/usr/lib/locales/locale-archive
```
  should exist and all locales available should be installed. I.e.
```
locale -a
```
  should output a very long list:
```
$locale -a | wc
```
  should count more then 800 lines.
- Packages like
```
glibc-langpack-hi
```
  maybe be installed in addition to
```
glibc-all-langpacks
```
  but as long as
```
glibc-all-langpacks
```
  is installed, they are redundant in that case and waste a bit of disk space, only the locales from /usr/lib/locales/locale-archive are used in that case.
Make sure that other locales, especially custom locales added by the user, are not affected.
- For example, if you install the locale sources with
```
dnf install glibc-locale-source
```
  and then create a locale with a custom name
```
sudo localedef --no-archive -i de_DE -f UTF-8 de_DE.utf8@myspecialgermanlocale
```
  , then a folder
```
/usr/lib/locale/de_DE.utf8@myspecialgermanlocale/
```
  should exist.
```
locale -a | grep ^de
```
  should show that such a locale is available and
```
LC_ALL=de_DE.UTF-8@myspecialgermanlocale locale -k language
```
  should show that the language of this custom locale is "German". As long as the folder name of such a user locale does not conflict with a folder name in any of the glibc-langpack-<language> packages, there should be no problem of installing and removing any of the glibc-langpack-<languages> and/or glibc-all-langpacks packages, that user defined locale should be available no matter how many of the glibc-langpack-<language> or glibc-all-langpacks packages are installed.

User Experience

It should be possible to save disk space by having fewer locales installed.

Dependencies

Anaconda
dnf-langpacks

Contingency Plan

Contingency mechanism: Revert changes to glibc packaging
Contingency deadline: Before F23 Beta release eg. Beta freeze.
Blocks release? No
Blocks product? No

Documentation, notes

QA and validation
- Validated binaries and files should be downloaded and installed.
- No slow compilation should happen on the host.
- For RHEL we want to be able to QA the binary artifacts and ship those. Shipping and compiling from source files is not an option we should consider.

Compatibility with locale-gen:
- Strong consideration should be given to writing a dummy wrapper that provides as much functionality from locale-gen as possible but uses dnf install foo in the background to meet the requirements.
- This helps users transition packages from debian/ubuntu to Fedora.
- For example /etc/locale.gen could be parsed and appropriate packages installed to cover the set of locales requested, or with a 1:1 mapping (lots of locale sub-packages) it would be trivial.
- Update of glibc could run locale-gen wrapper.
- Avoid using locales sub-package except to install common locale support files and binaries. Ubuntu uses locales package for this purpose, but provides language-pack-* for languages (like Windows)

Simple for the user to understand.
- Users should install "English" or "German" not "en_US.UTF-8".
- Users know what languages they need but shouldn't care about codesets or any other factors.
- Example: dnf install en-locales;
  - Either en-locales is a meta-package that installs all of the English locales, or is a package that contains all the English locales.
  - Probably 217 packages if we split it by lang name before underscore, but that would be way too much. We run into political issues with package counts.
- Example: vi /etc/locale.gen; locale-gen;
  - locale.gen must support simple "en" or "de" lines to install all formats of english or german.
- Example: dnf install locale-source;
  - Installs locale source files.
- Example: dnf install all-locales;
  - Install all binary locales. No locale may be called "all" (same restriction as build-locale-archive).
  - Includes installing a other-locales package which contains anything not split out. Adds limit that no locale may be called "other".

In summary:

The simplest implementation is to split important languages into sub-packages e.g. en-locales, de-locales. Then provide an all-locales metapackage which installs them all. Default to having glibc-common depend on all-locales. Each locale package like en-locales includes the prebuilt en locales. Each locale package is responsible for installing or uninstalling their particular locale from the locale-archive, with _install_langs overriding. This support is put in and tested.

Next work with Anaconda team to support selecting language, and translating that into dnf install en-locales or vi /etc/locale.gen; locale-gen;

Remove glibc-common dependency on all-locales. At this point user must have a /etc/locale.gen which contains automatically added entries if they were using dnf install en-locales or their manual entry.

Notes:

May need a /etc/sysconfig/locales to track state, but would like to make this as stateless as possible.
How does this tie into langpacks dnf support if at all?

Test packages:

Glibc packages build with copr for F23 and rawhide with locales split into sub-packages for testing are here: Test packages

Release Notes

@@ Line 32: / Line 32: @@
 This should link to your home wiki page so we know who you are.
 -->
-* Name: [[User:Mfabian| Mike Fabian]][[User:Siddhesh| Siddhesh Poyarekar]]
+* Name: [[User:Mfabian| Mike Fabian]],[[User:Siddhesh| Siddhesh Poyarekar]], [[User:Codonell| Carlos O’Donell]]
 <!-- Include you email address that you can be reached should people want to contact you about helping with your change, status is requested, or technical issues need to be resolved. If the change proposal is owned by a SIG, please also add a primary contact person. -->
-* Email: <mfabian At redhat DOT com, spoyarek AT redhat DOT com
+* Email: <mfabian At redhat DOT com>, <spoyarek AT redhat DOT com>, <codonell AT redhat DOT com>
 * Release notes owner: <!--- To be assigned by docs team [[User:FASAccountName| Release notes owner name]] <email address> -->
 <!--- UNCOMMENT only for Changes with assigned Shepherd (by FESCo)
@@ Line 45: / Line 45: @@
 == Current status ==
-* Targeted release: [[Releases/23 | Fedora 23 ]]
+* Targeted release: [[Releases/24 | Fedora 24 ]]
-* Last updated: (2015-06-17)
+* Last updated: (2015-12-08)
 <!-- After the change proposal is accepted by FESCo, tracking bug is created in Bugzilla and linked to this page
 Bugzilla states meaning as usual:
@@ Line 55: / Line 55: @@
 CLOSED as NEXTRELEASE -> change is completed and verified and will be delivered in next release under development
 -->
-* Tracker bug: <will be assigned by the Wrangler>
+* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1238406 Bug 1238406]
 == Detailed Description ==
@@ Line 61: / Line 61: @@
 <!-- Expand on the summary, if appropriate.  A couple sentences suffices to explain the goal, but the more details you can provide the better. -->
 Currently the file /usr/lib/locale/locale-archive contains
-*all* locales and is thus huge (103 megabytes).
+''all'' locales and is thus huge (103MB).
 For small systems (and containers) it would be useful to be able
@@ Line 99: / Line 99: @@
 Anaconda then needs to be changed to handle such language packages.
+Other [https://fedoraproject.org/wiki/I18N/Locale_Subpackaging implementation options]
 == Benefit to Fedora ==
@@ Line 109: / Line 111: @@
 == Scope ==
 * '''Proposal owners:'''
-. Figure out the best approach to to install/uninstall locales
+*# Figure out the best approach to to install/uninstall locales
-. Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)
+*# Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)
 <!-- What work do the feature owners have to accomplish to complete the feature in time for release?  Is it a large change affecting many parts of the distribution or is it a very isolated change? What are those changes?-->
@@ Line 156: / Line 158: @@
 <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
-* Try to install/uninstall packages containing locales for some languages.
+Please visit [https://fedoraproject.org/wiki/QA:Glibc_locale_subpackaging '''How to Test'''] page and participate in Fedora 24 i18n testday
+# Try to install/uninstall packages containing locales for some languages.
-* Check with “locale -a” whether the locales for these language have been correctly installed or uninstalled.
+#* For example, <pre>dnf install glibc-langpack-hi</pre> should install the Hindi locales <pre>locale -a | grep ^hi</pre> should then show that the Hindi loclales are installed.
+# Installing/uninstalling the meta packages langpacks-<language> should automatically install/uninstall the respective glibc packages.
-* Make sure that other locales, especially custom locales added by the user, are not affected.
+#* For example, <pre>dnf install langpacks-hi</pre> should install some stuff useful for Hindi including glibc-langpack-hi. <pre>dnf remove langpacks-hi</pre> should remove the Hindi stuff again and should also remove glibc-langpack-hi.
+# There should be a package glibc-all-langpacks, installing that should install all available locales into /usr/lib/locales/locale-archive
+#* For example, after <pre>dnf install glibc-all-langpacks</pre> the file <pre>/usr/lib/locales/locale-archive</pre> should exist and all locales available should be installed. I.e. <pre>locale -a</pre> should output a very long list: <pre>$locale -a | wc</pre> should count more then 800 lines.
+#* Packages like <pre>glibc-langpack-hi</pre> maybe be installed in addition to <pre>glibc-all-langpacks</pre> but as long as <pre>glibc-all-langpacks</pre> is installed, they are redundant in that case and waste a bit of disk space, only the locales from /usr/lib/locales/locale-archive are used in that case.
+# Make sure that other locales, especially custom locales added by the user, are not affected.
+#* For example, if you install the locale sources with <pre>dnf install glibc-locale-source</pre> and then create a locale with a custom name <pre>sudo localedef --no-archive -i de_DE -f UTF-8 de_DE.utf8@myspecialgermanlocale</pre>, then a folder <pre>/usr/lib/locale/de_DE.utf8@myspecialgermanlocale/</pre> should exist. <pre>locale -a | grep ^de</pre> should show that such a locale is available and <pre>LC_ALL=de_DE.UTF-8@myspecialgermanlocale locale -k language</pre> should show that the language of this custom locale is "German". As long as the folder name of such a user locale does not conflict with a folder name in any of the glibc-langpack-<language> packages, there should be no problem of installing and removing any of the glibc-langpack-<languages> and/or glibc-all-langpacks packages, that user defined locale should be available no matter how many of the glibc-langpack-<language> or glibc-all-langpacks packages are installed.
 == User Experience ==
@@ Line 184: / Line 191: @@
 * Blocks product? No <!-- Applicable for Changes that blocks specific product release/Fedora.next -->
-== Documentation ==
+== Documentation, notes ==
 <!-- Is there upstream documentation on this change, or notes you have written yourself?  Link to that material here so other interested developers can get involved. -->
+<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* QA and validation
+** Validated binaries and files should be downloaded and installed.
+** No slow compilation should happen on the host.
+** For RHEL we want to be able to QA the binary artifacts and ship those. Shipping and compiling from source files is not an option we should consider.
-<!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Compatibility with locale-gen:
-N/A
+** Strong consideration should be given to writing a dummy wrapper that provides as much functionality from locale-gen as possible but uses `dnf install foo` in the background to meet the requirements.
+** This helps users transition packages from debian/ubuntu to Fedora.
+** For example /etc/locale.gen could be parsed and appropriate packages installed to cover the set of locales requested, or with a 1:1 mapping (lots of locale sub-packages) it would be trivial.
+** Update of glibc could run locale-gen wrapper.
+** Avoid using `locales` sub-package except to install common locale support files and binaries. Ubuntu uses `locales` package for this purpose, but provides `language-pack-*` for languages (like Windows)
+* Simple for the user to understand.
+** Users should install "English" or "German" not "en_US.UTF-8".
+** Users know what languages they need but shouldn't care about codesets or any other factors.
+** Example: dnf install en-locales;
+*** Either `en-locales` is a meta-package that installs all of the English locales, or is a package that contains all the English locales.
+*** Probably 217 packages if we split it by lang name before underscore, but that would be way too much. We run into political issues with package counts.
+** Example: vi /etc/locale.gen; locale-gen;
+*** locale.gen must support simple "en" or "de" lines to install all formats of english or german.
+** Example: dnf install locale-source;
+*** Installs locale source files.
+** Example: dnf install all-locales;
+*** Install all binary locales. No locale may be called "all" (same restriction as build-locale-archive).
+*** Includes installing a `other-locales` package which contains anything not split out. Adds limit that no locale may be called "other".
+===In summary:===
+* The simplest implementation is to split important languages into sub-packages e.g. en-locales, de-locales. Then provide an all-locales metapackage which installs them all. Default to having glibc-common depend on all-locales. Each locale package like en-locales includes the prebuilt en locales. Each locale package is responsible for installing or uninstalling their particular locale from the locale-archive, with _install_langs overriding. This support is put in and tested.
+* Next work with Anaconda team to support selecting language, and translating that into `dnf install en-locales` or `vi /etc/locale.gen; locale-gen;`
+* Remove glibc-common dependency on all-locales. At this point user must have a /etc/locale.gen which contains automatically added entries if they were using `dnf install en-locales` or their manual entry.
+===Notes:===
+* May need a /etc/sysconfig/locales to track state, but would like to make this  as stateless as possible.
+* How does this tie into langpacks dnf support if at all?
+===Test packages:===
+Glibc packages build with copr for F23 and rawhide with locales split into sub-packages for testing are here: [https://copr.fedoraproject.org/coprs/mfabian/glibc/ Test packages]
 == Release Notes ==
@@ Line 197: / Line 243: @@
 -->
-[[Category:ChangePageIncomplete]]
 <!-- When your change proposal page is completed and ready for review and announcement -->
 <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
 <!-- The Wrangler announces the Change to the devel-announce list and changes the category to Category:ChangeAnnounced (no action required) -->
-<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangePageIncomplete-->
+<!-- After review, the Wrangler will move your page to Category:ChangeReadyForFesco... if it still needs more work it will move back to Category:ChangeReadyForWrangler-->
 <!-- Select proper category, default is Self Contained Change -->
-[[Category:SelfContainedChange]]
+<!-- [[Category:SelfContainedChange]] -->
-<!-- [[Category:SystemWideChange]] -->
+[[Category:ChangeAcceptedF24]]
+[[Category:SystemWideChange]]

Search

Changes/Glibc locale subpackaging: Difference between revisions