Changes/Glibc locale subpackaging

Glibc locale subpackaging

Summary

This change should make it possible to install or uninstall locales individually.

Owner

• Name: Mike Fabian, Siddhesh Poyarekar, Carlos O’Donell
• Email: <mfabian At redhat DOT com>, <spoyarek AT redhat DOT com>, <codonell AT redhat DOT com>
• Release notes owner:

Current status

• Targeted release: Fedora 24
• Last updated: (2015-12-08)
• Tracker bug: Bug 1238406

Detailed Description

Currently the file /usr/lib/locale/locale-archive contains all locales and is thus huge (103MB).

For small systems (and containers) it would be useful to be able to install only a small number of locales.

Recently we made it possible to install a small number of locales by supplying the rpm-macro “_install_langs”, for example

   rpm -i -D _install_langs="en:de_DE" glibc-common.rpm

will install all English locales and all German locales which start with “de_DE”,

   rpm -i -D _install_langs="en_US.utf8" glibc-common.rpm

will install only the en_US.utf8 locale,

   rpm -i -D _install_langs="POSIX" glibc-common.rpm

will install nothing (but the POSIX/C is still available because it is builtin into glibc).

But this approach works only during an Anaconda based install when Anaconda supplies the _install_langs rpm-macro.

When glibc is updated later, the _install_langs macro will not be supplied on the command line during the update and the default value “all” of “_install_langs” from /usr/lib/rpm/macros will be used and all locales come back during an update.

Therefore, this solution is far from perfect.

It should be made possible to install and uninstall locales individually, for example by having a separate package for the locales for each language. Installing such a package would add these locales to locale-archive, uninstalling it would remove them.

Anaconda then needs to be changed to handle such language packages.

Other implementation options

Benefit to Fedora

This change should make it possible to reduce the minimum size of an installation considerably.

Scope

• Proposal owners:
  1. Figure out the best approach to to install/uninstall locales
  2. Make sure that locales added manually by the user are not destroyed (currently they are lost when glibc is updated)

• Other developers:

Anaconda needs to be made aware of the new approach to handle installation/uninstallation of locales

• Release engineering:

I am not sure whether this has affects release engineering, probably the packages in the install image change when parts are split out of glibc-common.

• Policies and guidelines:

No, this change does not require updates to policies and guidelines.

• Trademark approval: not needed for this Change

Upgrade/compatibility impact

The user should not notice any change after the update. All locales which were installed before should still be installed after the update.

After the update it should be possible to uninstall locales for unneeded languages but this is optional.

How To Test

Please visit How to Test page and participate in Fedora 24 i18n testday

1. Try to install/uninstall packages containing locales for some languages.
   • For example,

     dnf install glibc-langpack-hi

     should install the Hindi locales

     locale -a | grep ^hi

     should then show that the Hindi loclales are installed.
2. Installing/uninstalling the meta packages langpacks-<language> should automatically install/uninstall the respective glibc packages.
   • For example,

     dnf install langpacks-hi

     should install some stuff useful for Hindi including glibc-langpack-hi.

     dnf remove langpacks-hi

     should remove the Hindi stuff again and should also remove glibc-langpack-hi.
3. There should be a package glibc-all-langpacks, installing that should install all available locales into /usr/lib/locales/locale-archive
   • For example, after

     dnf install glibc-all-langpacks

     the file

     /usr/lib/locales/locale-archive

     should exist and all locales available should be installed. I.e.

     locale -a

     should output a very long list:

     $locale -a | wc

     should count more then 800 lines.
   • Packages like

     glibc-langpack-hi

     maybe be installed in addition to

     glibc-all-langpacks

     but as long as

     glibc-all-langpacks

     is installed, they are redundant in that case and waste a bit of disk space, only the locales from /usr/lib/locales/locale-archive are used in that case.
4. Make sure that other locales, especially custom locales added by the user, are not affected.
   • For example, if you install the locale sources with

     dnf install glibc-locale-source

     and then create a locale with a custom name

     sudo localedef --no-archive -i de_DE -f UTF-8 de_DE.utf8@myspecialgermanlocale

     , then a folder

     /usr/lib/locale/de_DE.utf8@myspecialgermanlocale/

     should exist.

     locale -a | grep ^de

     should show that such a locale is available and

     LC_ALL=de_DE.UTF-8@myspecialgermanlocale locale -k language

     should show that the language of this custom locale is "German". As long as the folder name of such a user locale does not conflict with a folder name in any of the glibc-langpack-<language> packages, there should be no problem of installing and removing any of the glibc-langpack-<languages> and/or glibc-all-langpacks packages, that user defined locale should be available no matter how many of the glibc-langpack-<language> or glibc-all-langpacks packages are installed.

User Experience

It should be possible to save disk space by having fewer locales installed.

Dependencies

• Anaconda
• dnf-langpacks

Contingency Plan

• Contingency mechanism: Revert changes to glibc packaging
• Contingency deadline: Before F23 Beta release eg. Beta freeze.
• Blocks release? No
• Blocks product? No

Documentation, notes

• QA and validation
  • Validated binaries and files should be downloaded and installed.
  • No slow compilation should happen on the host.
  • For RHEL we want to be able to QA the binary artifacts and ship those. Shipping and compiling from source files is not an option we should consider.

• Compatibility with locale-gen:
  • Strong consideration should be given to writing a dummy wrapper that provides as much functionality from locale-gen as possible but uses dnf install foo in the background to meet the requirements.
  • This helps users transition packages from debian/ubuntu to Fedora.
  • For example /etc/locale.gen could be parsed and appropriate packages installed to cover the set of locales requested, or with a 1:1 mapping (lots of locale sub-packages) it would be trivial.
  • Update of glibc could run locale-gen wrapper.
  • Avoid using locales sub-package except to install common locale support files and binaries. Ubuntu uses locales package for this purpose, but provides language-pack-* for languages (like Windows)

• Simple for the user to understand.
  • Users should install "English" or "German" not "en_US.UTF-8".
  • Users know what languages they need but shouldn't care about codesets or any other factors.
  • Example: dnf install en-locales;
    • Either en-locales is a meta-package that installs all of the English locales, or is a package that contains all the English locales.
    • Probably 217 packages if we split it by lang name before underscore, but that would be way too much. We run into political issues with package counts.
  • Example: vi /etc/locale.gen; locale-gen;
    • locale.gen must support simple "en" or "de" lines to install all formats of english or german.
  • Example: dnf install locale-source;
    • Installs locale source files.
  • Example: dnf install all-locales;
    • Install all binary locales. No locale may be called "all" (same restriction as build-locale-archive).
    • Includes installing a other-locales package which contains anything not split out. Adds limit that no locale may be called "other".

In summary:

• The simplest implementation is to split important languages into sub-packages e.g. en-locales, de-locales. Then provide an all-locales metapackage which installs them all. Default to having glibc-common depend on all-locales. Each locale package like en-locales includes the prebuilt en locales. Each locale package is responsible for installing or uninstalling their particular locale from the locale-archive, with _install_langs overriding. This support is put in and tested.

• Next work with Anaconda team to support selecting language, and translating that into dnf install en-locales or vi /etc/locale.gen; locale-gen;

• Remove glibc-common dependency on all-locales. At this point user must have a /etc/locale.gen which contains automatically added entries if they were using dnf install en-locales or their manual entry.

Notes:

• May need a /etc/sysconfig/locales to track state, but would like to make this as stateless as possible.
• How does this tie into langpacks dnf support if at all?

Test packages:

Glibc packages build with copr for F23 and rawhide with locales split into sub-packages for testing are here: Test packages