Latest revision as of 15:33, 15 July 2021

Memory Constraints macros for RPM

Summary

Introduce macros, similar to openSUSE's memory-constraints), for optionally limiting build parallelism for build-time memory-bound packages

Owner

Name: Michel Alexandre Salim
Email: michel AT michel-slm DOT name

Current status

Targeted release: Fedora Linux 35
Last updated: 2021-07-15
FESCo issue: #2642
Tracker bug: #1982748
Release notes tracker: <will not be assigned by the Wrangler - no external impact>

Detailed Description

Some source packages have a memory usage per build thread higher than the RAM:CPU ratio available in some of our builders. Further, this ratio can be different for different build server on different architectures.

At the moment, such packages (ceph, chromium, mcrouter) have to implement their own logic for determining the safe amount of parallelism, and redefine _smp_build_ncpus.

When this proposal is implemented, they can instead declaratively specify the amount of RAM needed per build thread, e.g.

 %limit_build -m 8192

for declaring a build thread should be allocated 8GB of RAM.

Since Koji supports setting default values for macros, there will be a macro for the default memory limit (name TBD) that, if set, will be used to cap _smp_build_ncpus unless overridden by %limit_build -m. 0 I'm proposing to tentatively call the macro package build-constraints-rpm-macros to allow the possibility of adding macros for related needs e.g. build timeouts to the same package.

Feedback

Benefit to Fedora

This change simplifies maintaining specs for software that are memory-bounded rather than CPU-bounded on our build servers

It could potentially improve build reliability for these packages, by reducing the number of jobs failing because of OOM errors, and reduce the need for package maintainers to debug these failures.

By keeping the user-facing API aligned with what openSUSE is using, we open up the possibility to collaborate with them and with the upstream RPM project to get such macros upstreamed into RPM itself (see previous attempt). **note** that is not in scope for this Change.

Scope

Proposal owners:
- Introduce new macros
- Update known packages to use the new macros, replacing their custom _smp_build_ncpus overrides

Other developers:
- The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros

Release engineering: #10188

No mass rebuild needed. Affected packages should be rebuilt using the new macro

Policies and guidelines: Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages

Trademark approval: N/A (not needed for this Change)

Alignment with Objectives: N/A

Upgrade/compatibility impact

No impact, affects package building only. Also, the use of the new macros are optional.

How To Test

1. Install build-constraints-rpm-macros 2. Modify spec to set %limit_build -n AMOUNT_IN_MB in %build 3. Rebuild in koji and make sure it passes on all supported architectures

User Experience

Dependencies

This can optionally be added as dependencies of redhat-rpm-config and epel-rpm-macros, depending on how many packages need this

Contingency Plan

Contingency mechanism: (What to do? Who will do it?)

Revert changed packages to their previous way of capping the number of build jobs

Contingency deadline: beta
Blocks release? No

Documentation

Previous discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU/#GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU

openSUSE implementation: https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints

Release Notes

@@ Line 2: / Line 2: @@
 == Summary ==
-Introduce macros, similar to openSUSE's [https://build.opensuse.org/package/show/network:chromium/memory-constraints https://build.opensuse.org/package/show/network:chromium/memory-constraints]), for optionally limiting build parallelism for build-time memory-bound packages
+Introduce macros, similar to openSUSE's [https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints memory-constraints]), for optionally limiting build parallelism for build-time memory-bound packages
 == Owner ==
@@ Line 13: / Line 13: @@
 == Current status ==
-[[Category:ChangePageIncomplete]]
+[[Category:ChangeAcceptedF35]]
 <!-- When your change proposal page is completed and ready for review and announcement -->
 <!-- remove Category:ChangePageIncomplete and change it to Category:ChangeReadyForWrangler -->
@@ Line 30: / Line 30: @@
 ON_QA -> change is fully code complete
 -->
-* FESCo issue: <will be assigned by the Wrangler>
+* FESCo issue: [https://pagure.io/fesco/issue/2642 #2642]
-* Tracker bug: <will be assigned by the Wrangler>
+* Tracker bug: [https://bugzilla.redhat.com/show_bug.cgi?id=1982748 #1982748]
-* Release notes tracker: <will be assigned by the Wrangler>
+* Release notes tracker: <will not be assigned by the Wrangler - no external impact>
 == Detailed Description ==
@@ Line 46: / Line 46: @@
 for declaring a build thread should be allocated 8GB of RAM.
+Since Koji supports [https://docs.pagure.org/koji/release_notes/release_notes_1.18/#system-changes setting default values for macros], there will be a macro for the default memory limit (name TBD) that, if set, will be used to cap `_smp_build_ncpus` unless overridden by `%limit_build -m`.
 I'm proposing to tentatively call the macro package `build-constraints-rpm-macros` to allow the possibility of adding macros for related needs e.g. [https://pagure.io/copr/copr/issue/1678 build timeouts] to the same package.
@@ Line 68: / Line 70: @@
 ** The proposal owners might not catch all references of such logic. Individual package maintainers can try refactoring their packages using these new macros
-* Release engineering: [https://pagure.io/releng/issues #Releng issue number] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Release engineering: [https://pagure.io/releng/issue/10188 #10188] <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
 <!-- Does this feature require coordination with release engineering (e.g. changes to installer image generation or update package delivery)?  Is a mass rebuild required?  include a link to the releng issue.
 The issue is required to be filed prior to feature submission, to ensure that someone is on board to do any process development work and testing and that all changes make it into the pipeline; a bullet point in a change is not sufficient communication -->
 No mass rebuild needed. Affected packages should be rebuilt using the new macro
-* Policies and guidelines: N/A (not needed for this Change) <!-- REQUIRED FOR SYSTEM WIDE CHANGES -->
+* Policies and guidelines: Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages
-<!-- Do the packaging guidelines or other documents need to be updated for this feature?  If so, does it need to happen before or after the implementation is done?  If a FPC ticket exists, add a link here. Please submit a pull request with the proposed changes before submitting your Change proposal. -->
-Packaging guideline can be updated to recommend using these macros for build-time memory-bound packages
 * Trademark approval: N/A (not needed for this Change)
@@ Line 143: / Line 143: @@
 Previous discussion: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU/#GWSXTGOCMNXHPGMTW32LI5EKPGHDKCFU
-openSUSE implementation: https://build.opensuse.org/package/show/network:chromium/memory-constraints
+openSUSE implementation: https://build.opensuse.org/package/show/openSUSE:Factory/memory-constraints
 == Release Notes ==

Search

Changes/MemoryConstraintsMacros: Difference between revisions