Changes/Java11

java-11-openjdk as system JDK in F33

Summary

Update the system JDK in Fedora from java-1.8.0-openjdk to java-11-openjdk.

Owner

• Name: Jiri Vanek
• Email: <jvanek@redhat.com>
• Product: java and java stack
• Responsible WG: java-sig (java and java-maint)
• side tag rcm ticket: https://pagure.io/releng/issue/9574

Current status

• Targeted release: Fedora 33
• Last updated: 2020-11-05
• FESCo issue: #2370
• Tracker bug: #1825969
• Release notes tracker: #495

Expected schedule

• 1st May 2020 mass rebuild in copr
  • all maintainers spammed with results
• 1st June 2020 second mass rebuild in copr
  • all maintainers will be spammed
• aprox 1st July 2020 mass rebuild in rawhide
  • FTBFS bugs will be filled
• 1st August change finalisation
• 11th August f33 branched - https://fedorapeople.org/groups/schedule/f-33/f-33-key-tasks.html
  • hard deadline for feature completed

Detailed Description

Fedora currently ships:

• java-1.8.0-openjdk (LTS)
• java-11-openjdk (LTS)
• an java-latest-openjdk (on jdk14, STS).

where the version-less java and javac (and friends) are provided by java-1.8.0-openjdk.

So every package honoring the packaging rules and requiring java , java-headless or java-devel is built in koji by java-1.8.0-openjdk-devel and pulls java-1.8.0-openjdk(-headless) in runtime (See java ). Also javapackaging-tools are using java-1.8.0-openjdk as hardcoded runtime (see changes)

Debian already moved to JDK11 as system JDK, and in Fedora community we can hear for last two years increasing voices for jdk11 to become the system one. And apparently the java stack is really quite ready for JDK11. Fedora is considered to be bleeding edge distribution, so we should move forward, however JDK11 is not 100% compatible with JDK8, so there may rise nasty issues and very unhappy users.

Major incompatibility is in encapsulation. What was private, is now really private and if one was using it (and it was common) then the code will stop working. Unluckily, most of those issues are not only build time issues, but runtime issues, so hard to spot without proper test coverage.

A bit of history to recall:

version (upstream support) - fedora state

• Java SE 6 (LTS 2006-cca2014) - added as techpreview in F8 replaced GCJ around F13. See https://fedoraproject.org/wiki/Releases
• Java SE 7 (LTS 2011-2020(?)) - entered Fedora as tech preview around F16/17 replaced JDK 6 as System JDK in F17
• Java SE 8 (LTS 2014-2026(???)) - entered Fedora as tech preview in F19 replaced JDK 7 as System JDK in F21. JDK6 and JDK7 later returned as community driven java-1-{6,7}-0-openjdk-legacy packages for a while
• Java SE 9 (2017-2018) - entered Fedora around F26 as secondary, java-1.9.0-openjdk alongside java-1.8.0-openjdk
• Java SE 10 (2018-2019) - entered F28 as secondary, java-latest-openjdk alongside java-1.8.0-openjdk
• Java SE 11 (LTS 2018-??) - entered Fedora as tech preview in F29 together with java-latest-openjdk and java-1.8.0-openjdk
• Java SE 12 (2019) - updated java-latest-openjdk transparently and keept going alongside java-1.8.0-openjdk and alongside java-11-openjdk
• Java SE 13 (2019-2020) - updated java-latest-openjdk transparently and keept going alongside java-1.8.0-openjdk and alongside java-11-openjdk
• Java SE 14 (March 2020-2021(?)) - is updating java-latest-openjdk transparently and will continue to live next to any system JDK(s)
• Java SE 15 (2020(?)-2021(?)) - will update java-latest-openjdk transparently and will continue to live next to any system JDK(s)
• Java SE 16 () - will update java-latest-openjdk transparently and will continue to live next to any system JDK(s)
• Java SE 17 (LTS) - will very likely update java-latest-openjdk transparently and will become future system JDK java-17-openjdk some year later
• Java SE 18 () - will update java-latest-openjdk transparently and will continue to live next to any system JDK(s)
• ....

"Political disclaimer"

In previous bumps, we, Red Hat's openjdk team, were a driving force to bump the system JDK. In this case, we are a bit reluctant, but the desire from users to bump the JDK seems strong. We are quite happy to skip JDK11 as system JDK at all, and jump directly from 8 to 17 in some three years.

Benefit to Fedora

JDK11 is out for some time, and most of the bleeding edge distributions already make it default. Most of the projects already adapted new features and make necessary forward-compatible chnages. Although we can can expect some family of packages to remain on jdk8 for ever, the spice should flow

Scope

keep java-1.8-0-openjdk but remove its java/javac versionless provides, make java-11-openjdk providing java, javac and other versionless provides

• will guarantee fedora to be pure JDK11 distro.
• will allow maintainers of JDK9 or up incompatible packages to keep using JDK8, however this is just false hope.
  • if such an package depends on package build by JDK11, JDK8 will not be able to pick up that dependency.
  • this may lead to quite a lot of bundling or compat packages, but may be acceptable
  • people developing JDK8 applications will very likely stay with fedora:)
• xmvn javadoc is now used as a result of to much failing javadoc builds: https://src.fedoraproject.org/rpms/javapackages-tools/pull-request/3#comment-46283

While quite a lot of users will rejoice, there may be cases where application is very hard to migrate to JDK11, so the contingency plan should be taken very serious.

Bytecode version

• It appeared, that several applications have to build by jdk8, while works fine with jdk11
• it lead to manual work on align libraries on 1.8 byte code version. see https://pagure.io/java-maint-sig/issue/7
• Other approaches how to avoid this in next update (jdk17, aprox f36, minimal bytecode 11) were mentioned here: https://src.fedoraproject.org/rpms/javapackages-tools/pull-request/3#comment-50266

Workflow

Change owners

• Feature will be implemented in side tag
  • --target f33-java11 is tag of choice
  • In its mass rebuild, aprox 650 packages were built, and 150 failed. FTBFS bugs filled, most of them later fixed by decathorpe (Fabio Valentini); everybody owes him.
• the jdk 8 and 11 will be changed for this side tag
• the mass rebuild of javastack will be then launched
  • if necessary, several rounds of them
• Failures will be gathered by me and few other volunteers
  • Most common issues and theirs fixes will be published
  • Package maintainers will be notified in case of failure via bugzilla
• Depending on the fail rate, importance of failed packages and effort to fix them
  • the side tag will be merged to Fedora
  • or the contingency plan will be activated

Other developers

• should fix their packages
  • this usually means to update to newer version, which supports jdk11
• or to retire them if they appear non-fixable
• or to base them on JDK8 without much warranty (as they will need to compat whole dependency chain)

Other possible approaches already discarded by various discussions

• java-sig no longer exists
• as only one option remained in scope, Fesco probably do not need to decide, but still should evaluate, and eventually chnage the way

remove java-1.8-0-openjdk, make java-11-openjdk providing java, javac and other versionless provides

• will guarantee fedora to be pure JDK11 distro.
• will force everybody to fix theirs packages for jdk11
  • this may lead to huge orphaning of packages.
• will guarantee that developers using JDK8 for theirs projects will move to other providers or distros.
• Jdk8 will very likely appear as community driven legacy package, but its support may be poor.
• We should keep JDK8 in Fedora for couple of few more years at least as secondary JDK

• deprecated by discussions

keep java-1.8-0-openjdk, remove its java/javac versionless provides, make java-11-openjdk providing java, javac and other versionless provides, but build in koji (by JDK11) JDK8 compatible bytecode

• Idea is to provide some kind a koji specific rpm macro, which will add -target 1.8 to all javac calls
  • We are not going to do this via the jdk. We already did once for javadoc in 8, and it was an mistake.
    • Developers developing on Fedora's java-1.8.0-openjdk were then failing to deploy on other plaftorms
  • the change should be koji only
  • maybe enough would be to incorporate this to %mvn call only, and let other build types on default
• this hybrid approach would make a lot, but needs a lot of cooperation
  • all packages will be able to try to build and run by jdk11 directly
  • packages doomed for jdk8 for ever, will be able to use by jdk11 built dependencies
  • Fedora will not be pure jdk11
  • this is temporary workaround
  • this is untested and without precedent
  • the removal of the macro/target will be another system wide change
• the macro must be allowed to be disabled, to allow JDK11 specific packages. The usage of -target 8 is killing jdk11 language features
• this is practically killing any jdk9 and up language features, but for initial mass rebuilding of currently jdk8 comaptible packages, it may be good first step

• deprecated by discussions

keep java-1.8-0-openjdk, remove its java versionless provides but keep javac versionless provides , make java-11-openjdk providing java, nut not javac versionless provides

• this will keep javastack build in koji being still built by jdk8, and thus reusable by both JDK8 and 11
• this will allow both jdk somehow coexists and will allow moreover smooth transition
• this may cause chaos, once some packages starts to require JDK11 and some JDK8. But that is indirectly threatening all except first case.
• another disadvantage is possible occurrence of runtime failure, which would otherwise be discovered by building by javac-11

• not recommended by discusiions

Other

• Proposal owners:
  • based on above, adapt jdk8 and jdk11 package provides
  • If necessary tune the build environment

• Other developers:
  • based on selected approach to tune the main build tools
    • at least jpackage-tools and maven will be very likely affected
  • based on selected approach to tune the rpmbuild/macros
  • many java package maintainers will maybe need to adapt theirs packages
    • After the approach is agreed, mass rebuild must be performed
    • FTBFS bugs connected with this proposal, maybe with pagure ticket to allow discussion.
    • Solutions to most common errors should be gathered and published

• Release engineering: 9347 (a check of an impact with Release Engineering is needed)
  • mass rebuild will be required for this change

• Policies and guidelines: how to deal with build failures, eventually how to use some jdk11 specific build features will be provided

• Trademark approval: N/A (not needed for this Change)

Upgrade/compatibility impact

Once the change is implemented properly, the update should be flawless and seamless

How To Test

• Depends on selected approach, only JRE11 should remain the only system JDK after installing base javastack on clean system
• Both jdk8 and jdk11 can live togehter on system
• jdk11 will be selected by default and will run most of the base java stack
• todo, add few java-package to isntall examples, hopefully for jdk11 only

User Experience

• Standard user should be still be able to use java stack without even noticing the chnage.
• Standard developer should be still be able developer any java application comfortably
• Standard packager will not suffer to much, and should be able to pack any java application for fedora

Dependencies

Around 2600 packages will need attendance

$ repoquery -q --whatrequires java-headless |wc -l
2481
$ repoquery -q --whatrequires java | wc -l
78
$ repoquery -q --whatrequires java-devel | wc -l
21

Packages needing major work will be

• java-1.8.0-openjdk
• java-11-openjdk
• javapackages-tools
• icedtea-web
• https://pagure.io/fedora-comps/blob/master/f/comps-f33.xml.in
• java-atk-wrapper
• openjfx

Contingency Plan

• If the mass rebuild, after the change application, breaks to much packages, or some important will be unfixable, jdk8 must be restored back to the position of system jdk.
• Contingency mechanism: Return jdk8 as system jdk and mass rebuild again. Note, that this may be very hard, because during build of packages by jdk8, by jdk11 built dependencies will be picekd up, so build will fail. Maybe several iterations of mass rebuild will be needed.
• Contingency deadline: beta freeze
• Blocks release? Yes
• Blocks product? N/A
• Generally going back will be imho impossible. Once the decision is taken, javastack should be fixed, and where it can not be fixed, it had to migrate to compat packages, or bundled-dependencies pacages or be orphaned.

side tag

https://fedoraproject.org/wiki/Package_update_HOWTO#Creating_a_side-tag

• for this fedora have technology known as side tag
• I do not have experience with it, but is making the contingency plan much smoother
• it is a way, which will be done first

copr preliminary rebuild

We run an preliminary mass rebuild of javastack in copr repo https://copr.fedorainfracloud.org/coprs/jvanek/java11/builds/ on packages requiring java,javac, java-devel, maven-local, ant, ivy & comp for build. The first result was quite dramatic:

1225  total; attempted to rebuild
483   failed; from those 191 are trivial failures (but once that is fixed, real troubles may start)
186   succeeded
556   orphans or dead or otherwise tragic so the build did not even started

• If there is "failed" but contains "- -" then it is probably orphan. If you wish to resurrect it, please ensure it runs against JDK11 (see lower)
• If there is "failed" but failed in "seconds", then those packages failed so quickly, that the build was in initial phases. That usually mean that you build with source/target lower then 1.6 JDK11 supports 1.6 and up. We recommend to bump the source/target to 1.8, to allow existence of compat 1.8 packages alongside main javastack
• If there is "failed", and its none of above, then your package simply failed. Very often the scary error may be fixed by bump to latest upstream version. JDK 11 is out for several years. Please, try to fix the package. Don't hesitate to ask,. If you fix the fail, feel free to share your fix, it may help others.

Debugging failures wiht help of this copr

The copr repo we maintain, contains builds of java-11-openjdk as system JDK, javapackages-tools honoring that, and java-1.8.0-openjdk as non system JDK. Also it contains successfully rebuilt packages. You can directly use this copr repo in several ways.

• first glance on error. On https://copr.fedorainfracloud.org/coprs/jvanek/java11/builds/ find yours build (select "all" instead of "25" at the bottom),
  • Click its number, select chroot (currently fedora-rawhide-x86_64 ) and check the logs. Main log is build.log.gz.
• anything you push to rawhide, will automatically rebuild here in rawhide chroot (we have jdk in rawhide broken a bit currently)
  • It is the best approach. If you can fix your package in rawhide directly, without breaking the rawhide to much, go for it
  • If yo need to experiment, I have a mock config for you (generated from copr-cli mock-config jvanek/java11 fedora-rawhide-x86_64) which you can copy to your /etc/mock and use - https://jvanek.fedorapeople.org/java11/jvanek-java11-fedora-rawhide-x86_64.cfg . Eg:

sudo cp jvanek-java11-fedora-rawhide-x86_64.cfg /etc/mock/jvanek-java11-fedora-rawhide-x86_64.cfg
#or
cp jvanek-java11-fedora-rawhide-x86_64.cfg ~/.config/mock/jvanek-java11-fedora-rawhide-x86_64.cfg
# change spec, bump sources, apply patches
fedpkg srpm
mock -r jvanek-java11-fedora-rawhide-x86_64  *.src.rpm

Or any other packaging workflow you use, and you can use against the copr repo.

Documentation

• oracle11 release notes: https://www.oracle.com/technetwork/java/javase/11-relnote-issues-5012449.html
• openjdk11 jeps: https://openjdk.java.net/projects/jdk/11/
• As we are moving from java8 to 11, and jdk9 is what brought in breaking changes, providing also 9 and 10:
  • openjd10 https://openjdk.java.net/projects/jdk/10/
  • openjd9 https://openjdk.java.net/projects/jdk9/
  • oracle10 https://www.oracle.com/technetwork/java/javase/10-relnotes-4108314.html
  • oracle9 https://www.oracle.com/technetwork/java/javase/9-relnotes-3622618.html
    • where jdk9 migration https://docs.oracle.com/javase/9/migrate/toc.htm#JSMIG-GUID-7744EF96-5899-4FB2-B34E-86D49B2E89B6 is quite useful for this proposal

common issues packagers can face and gathered solutions

Contacts: ask on devel@lists.fedoraproject.org or java-devel@lists.fedoraproject.org or directly to me jvanek@redhat.com Threads of "F33 system wide change, java-11-openjdk as system jdk"

Major database can be browsing of closed bugs of blockers of https://bugzilla.redhat.com/show_bug.cgi?id=1825969 ; unluckily, when it was analysed, it was not summarised up (that would actually double the work)

My package can not work with jdk11

Generally it is not true. Generally, no program can say, that it do not support jdk11, because any javac/java application can be *hacked* to work with java11 - see https://jvanek.fedorapeople.org/devconf/2017/portingjavaInternalToJdk9/portingOfItwToJdk9-II.pdf (really all except package split over modules, which is impossible)

Now above mentioned approaches are indeed *hacked*, and I discourage everybody to do so. The upstream should be moved to jdk11, and not much excuses are around to support to not to do so. If you package is really bound to jdk8, you can move to the version-full requires: BuildRequires: java-1.8.0-openjdk(-devel) and * Requires: java-1.8.0-openjdk(-headless). In addition, if you work with maven/ant or simialr, you must set export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk before calling it. japckage tools and comp are made to accept it.

However there is an trap - packages you depends on. Once some of your dependencies will be compiled with --target > 8, you are doomed, and you have to bundle it or create its compat version. By doing so you can easily end in dependency hell.

Please, try to avoid this as much as possible!

Intermediate step build with java-1.8.0-openjdk-devel and run with java (that means any sytem java, eg java-11-openjdk)

Some projects support JDK11 for runtime, but not for compile time. I'm aware at least about 5, now I do rember icedtea-web and java-mysql-connector.

Buildrequires: java-1.8.0-openjdk-devel
...
Requires: java(-headless)
...
%build
...
 export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
...

Should work for a while. See the "My package can not work with jdk11" section

My package is not in your copr!

If your package is not listed in the copr rebuild repo, I can see two causes

• You have very indirect BR on java.
  • Solution
  • Email me (jvanek@redhat.com) or ping me (jvanek), I will gladly add you package(s)
• You have exact requires on java-1.8.0-openjdk(-devel)
  • Solution
  • Unless you have good reason, you are actually breaking packaging guidelines. Switch to java-devel. Once done, again let me know and I will gladly add your package
  • If you can't, then you most likely can not bump to jdk11, and you will live with jdk8 until it dies,

Wrong source/target version

maven

[ERROR] Source option 1.3 is no longer supported. Use 6 or later.
[ERROR] Target option 1.3 is no longer supported. Use 1.6 or later.

ant

-do-compile:
    [mkdir] Created dir: /builddir/build/BUILD/jpanoramamaker-5/build/empty
    [javac] Compiling 45 source files to /builddir/build/BUILD/jpanoramamaker-5/build/classes
    [javac] warning: [options] bootstrap class path not set in conjunction with -source 5
    [javac] error: Source option 5 is no longer supported. Use 6 or later.
    [javac] error: Target option 1.5 is no longer supported. Use 1.6 or later.
BUILD FAILED

• Solution
  • Your javac is run with wrong source/tag parameters. net search will give you quick answer
  • Fixes are:
    • example patch for netbeans generated ant

usage of sun.misc.Unsafe.defineClass

• use public replacement, java.lang.invoke.MethodHandles.Lookup.defineClass
• eg https://src.fedoraproject.org/rpms/procyon/blob/master/f/lookupPatch.patch
• Note taht it is not 100% replacement, and you should consult upstream first
• Note that it is JDK8 incompatible change

package XYZ does not exists ... in javadoc generation!

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:3.1.1:aggregate (default-cli) on project  classloader-leak-test-framework: An error has occurred in Javadoc report generation: 
[ERROR] Exit code: 1 - /builddir/build/BUILD/classloader-leak-prevention-classloader-leak-test-framework-1.1.1/src/main/java/se/jiderhamn/classloader/leak/JUnitClassloaderRunner.java:9: error: package org.junit does not exist
[ERROR] import org.junit.Assert;

• nasty workaround is to skip javadoc
  • eg: https://src.fedoraproject.org/rpms/classloader-leak-test-framework/c/c49efd4a19d8d918b700bda9c2f962f1eb1e865a?branch=master
    • reverted by below approach: https://src.fedoraproject.org/rpms/classloader-leak-test-framework/c/0926fda19da9433984dd2012b1eeaf925e4036b7?branch=master
• solution is be to replace javadoc plugin by xmvn --javadoc:
  • https://src.fedoraproject.org/rpms/byteman/c/d145b9e4af8952b1a63432ec67d7109161f83617?branch=master
  • it is bug in fedora mavenjavadocplugin
• see this thread: https://lists.fedoraproject.org/archives/list/java-devel@lists.fedoraproject.org/thread/UD7Q5DYAWI7YO4VW7UZPDWR644V7S462/

javah: No such file or directory

The javah command was retired and is no longer available in jdk11; you should switch the package to use javac -h instead. The new javac -h syntax works just fine with jdk8, so you can switch to this now without worrying about backwards compatibility.

For example: https://src.fedoraproject.org/rpms/snappy-java/blob/2981aaa5b8a597129ad2f12cdf1f4617303a8425/f/javah-adaptation.patch