From Fedora Project Wiki
 
(40 intermediate revisions by the same user not shown)
Line 8: Line 8:
This is a vast improvement over our previous process, but Publican includes features that make publishing even easier and more robust, since Publican is designed to package documentation in RPM packages to install on a webserver. [http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html/Users_Guide/sect-Users_Guide-Website.html Chapter 6 of the ''Publican Users' Guide''] provides an overview of the mechanism.
This is a vast improvement over our previous process, but Publican includes features that make publishing even easier and more robust, since Publican is designed to package documentation in RPM packages to install on a webserver. [http://docs.fedoraproject.org/en-US/Fedora_Contributor_Documentation/1/html/Users_Guide/sect-Users_Guide-Website.html Chapter 6 of the ''Publican Users' Guide''] provides an overview of the mechanism.


'''Important''' -- note that since the webserver runs Red Hat Enterprise Linux 5, the documentation packages need to be built for Red Hat Enterprise Linux 5. Publican is also designed to build documentation to install for desktop use, but this is a separate set of considerations and not the aim here.
'''Important''' -- note that since the webserver runs Red Hat Enterprise Linux 6, the documentation packages need to be built for Red Hat Enterprise Linux 6. Publican is also designed to build documentation to install for desktop use, but this is a separate set of considerations and not the aim here.


==Steps to automate publishing==
==Steps to automate publishing==
===Install Publican 2 on the webserver===
===Install Publican 4.0 on the webserver===
Publican 2 is the first version that includes the web publishing components. However, Publican 2 has a number of dependencies that are either not in Red Hat Enterprise Linux 5 at all, or which are in  Red Hat Enterprise Linux 5 in versions too old for Publican 2. While we could build dependencies in the former group in EPEL, we cannot build dependencies in the latter group in EPEL since [[EPEL/GuidelinesAndPolicies#Policy|EPEL packages must not update packages in Red Hat Enterprise Linux]], only provide packages that are not shipped at all.
The version of Publican shipped in Red Hat Enterprise Linux 6 is very old (2.1) and lacks many bug fixes and enhancements that have been added since then (current upstream is 4.0, due in Fedora in the next few weeks).


To install Publican 2 on the webserver, we need to  
Some of the Publican 4.0 deps in Red Hat Enterprise Linux 6 are too old or not available on all arches; we will need to build these for el6-docs
* build the dependencies not in el5 for EPEL
* build the dependencies too old in el5 for dist-5E-epel-infra.
* find a solution to using fonts-chinese-zysong
* build Publican 2 and the Fedora brand package for dist-5E-epel-infra.


====Dependencies not in el5====
It's also possible that Publican 4.0 has dependencies that are not in Red Hat Enterprise Linux 6; at all. If so, we will need to add these to EPEL.
* <s>perl(Config::Simple) -- sixy (stable)</s>
* <s>perl(DateTime::Format::DateParse) -- rlandmann</s>
* <s>perl(File::pushd) -- mmaslano (stable)</s>
* <s>perl(Locale::PO) -- iarnell -- (stable)</s>
* <s>perl(Syntax::Highlight::Engine::Kate) -- mmaslano (stable)</s>
** <s>perl(XML::TokeParser) -- iarnell (stable)</s>
* [fop -- nb/lkundrak (testing -- installs, but depends on batik and newer xml-commons-apis)]
** <s>xmlgraphics-commons -- nb/rlandmann (stable)</s>
*** <s>jakarta-commons-io -- pcheung/rlandmann (stable)</s>
* [dep for batik]
** <s>jython -- overholt/rlandmann (stable)</s>
*** <s>ht2html --  mjakubicek (stable)</s>
*** <s>libreadline-java -- akurtakov/rlandmann (stable)</s>
* [deps for perl(Makefile::Parser)]
** <s>perl(IPC::Run3) --  corsepiu/rlandmann (stable)</s>
** <s>perl(Makefile::DOM) -- ryanlerch/rlandmann (stable)</s>


====Dependencies to update in el5====
To install Publican 4.0 on the webserver, we need to  
* perl(XML::LibXML) ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2563163 built])
* <s>get a new el6-docs build root -- [https://fedorahosted.org/rel-eng/ticket/5214 rel-eng request here]</s> Done! Thanks ausil!
** libxml2 (http://koji.fedoraproject.org/koji/taskinfo?taskID=2563167 built)
* <s>identify and build any dependencies not in el6 for EPEL</s>
* perl(XML::LibXSLT)
* <s>identify and build any dependencies too old or not on all arches in el6 for el6-docs.</s>
** libxslt
* build <s>Publican 4.0 and</s> the Fedora brand package for el6-docs.
* perl(XML::TreeBuilder) ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2560919 built])
* docbook-style-xsl ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2560917 built])


Also these packages that cannot be built for el5 because of a dependency that is too old in el5:
====Dependencies not in el6====
* perl(Makefile::Parser)  
* <s>wkhtmltopdf</s>
** perl(Class::Trigger) ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2560915 built])
** <s>wkhtmltopdf-qt</s>
* batik
* <s>perl-DateTime-Format-DateParse</s>
** xml-commons-apis ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2560913 built])
* <s>perl-Locale-Msgfmt (branch requested for EPEL)</s>
* <s>perl-Lingua-EN-Fathom (coming in EPEL)</s>
* <s>perl-XML-Catalog (for XML::TreeBuilder, coming in EPEL)</s>
* <s>perl-IO-Compress</s>
** <s>perl-Compress-Raw-Bzip2</s>
** <s>perl-Compress-Raw-Zlib</s>


This package builds for el5 but cannot be installed because of dependency problems:
====Dependencies to update in el6====
* fop (depends on batik and xml-common-apis) ([http://koji.fedoraproject.org/koji/taskinfo?taskID=2563189 built])


* <s>perl-HTML-Tree -- too old</s>
** <s>perl-XML-TreeBuilder -- too old</s>
* <s>docbook5-style-xsl -- too old</s>
* <s>perl-Archive-Tar -- too old</s>
* <s>perl-Syntax-Highlight-Engine-Kate -- too old</s>
** <s>perl-Test-Differences (for Syntax-Highlighting-Kate) -- too old</s>
<!--
====fonts-chinese-zysong====
====fonts-chinese-zysong====
The EL5 version of Publican 2 depends on fonts-chinese-zysong, which is shipped only on the Red Hat Enterprise Linux&nbsp;5 supplementary CD due to licensing reasons.[https://rhn.redhat.com/errata/RHEA-2007-0833.html] Since we won't be able to build fonts-chinese-zysong in Koji, we will need to find a way to work around this dependency. It would mean modification to (at least) the Publican spec file, and the pdf.xsl file that ships with Publican.
The EL5 version of Publican 2 depends on fonts-chinese-zysong, which is shipped only on the Red Hat Enterprise Linux&nbsp;5 supplementary CD due to licensing reasons.[https://rhn.redhat.com/errata/RHEA-2007-0833.html] Since we won't be able to build fonts-chinese-zysong in Koji, we will need to find a way to work around this dependency. It would mean modification to (at least) the Publican spec file, and the pdf.xsl file that ships with Publican.
-->
===Update the Fedora brand===
The Fedora brand needs some minor updates to make it Publican 4 ready. Probably just book titles in HTML, and renaming a CSS file for the -web subpackage.
* needs fonts-overpass to go stable in EPEL (requested)


===Configure an Publican-generated website on the webserver===
===Configure an Publican-generated website on the webserver===
Line 65: Line 61:
* the '''/var/www/html/docs''' directory in which Publican publishes documentation
* the '''/var/www/html/docs''' directory in which Publican publishes documentation


docs.fedoraproject.org would need to point to '''/var/www/html/docs''' on the server.


===Set up webserver for automatic updates===
The webserver should check the el6-docs repo on a cronjob to look for new, updated, or deleted packages. Rudi can supply a script for this.
===Build the docs packages===
As a team, we would need to build all the docs packages in Koji. I think as long as we do all packages for F19 and F20, we can proceed to the next step (cutover) and gradually rebuild our legacy docs over a longer period of time.
==Cutover==
When the documentation for currently maintained versions (F19 and F20) has all been rebuilt, we request that Infra points
docs.fedoraproject.org to '''/var/www/html/docs''' on the server.
<!--
====Staging site====
====Staging site====
Documentation also has a stage at http://docs.stg.fedoraproject.org. This host should also have Publican 2 running on it so that we can stage documents there that are not yet ready for even the "drafts" section of the main docs site. The most important use case would be the nightly builds of translated documentation that the Documentation project produces for the Localization project in the last few weeks before a release. Besides this, however, writers sometimes want to share extremely embryonic documentation. Being able to do this through the automated packaging system keeps even these early stages of development inside the main docs workflow.
Documentation also has a stage at http://docs.stg.fedoraproject.org. This host should also have Publican 2 running on it so that we can stage documents there that are not yet ready for even the "drafts" section of the main docs site. The most important use case would be the nightly builds of translated documentation that the Documentation project produces for the Localization project in the last few weeks before a release. Besides this, however, writers sometimes want to share extremely embryonic documentation. Being able to do this through the automated packaging system keeps even these early stages of development inside the main docs workflow.


===Set up a build system for documents===
===Set up Koji for documents===
To build documentation packages for docs.fedoraproject.org (and a staging site), members of the docs-writers group need to be able to build packages and place them in a repo accessible by the web servers.
We would need a target/tag for dist-6E-docs and perms for docs project members to build in there
 
-->
We need to explore:
 
* could these packages be built for dist-5E-epel-infra? Ideally we would need separate subtags for documents destined for public consumption and those that will only appear on the stage. Members of the docs-writers group would be able to build any docs package and would be able to tag it for the stage; members of the docs-publishers group would additionally be able to tag any docs package for the public site.


* could these packages be built on koji.fedoraproject.org or will we need a separate Koji instance? The biggest issue here is not technical, but procedural, since the current package approval mechanism or packager approval mechanism are not a good fit for the docs project workflow. Since each document package is specific to a particular Fedora release, and since each language is packaged separately, we would have generated 101 new packages for Fedora 13, and needed to grant packager permissions to every member of the Docs team and Localization team working on these packages. It is unlikely that the current review process has the bandwidth to handle this number of new packages every release in a timely fashion; furthermore, since the packages are produced by Publican in a completely automated fashion, it is unlikely that most writers or translators will possess (or will ever possess) the knowledge required to gain packager permissions (but nor do they need this knowledge to build docs packages).
===Contents of el6-docs===
As a footnote, these are the packages we are carrying in el6-docs. Ones marked in bold can (and should) be carried in EPEL itself -- we're working on this!


==The future==
* docbook-style-xsl-1.78.1-2.el6
It appears that Publican will ship with Red Hat Enterprise Linux&nbsp;6, although the version of Publican in the Beta is a late version of Publican 1, which lacks the web publishing components. If Publican 2 ships with Red Hat Enterprise Linux&nbsp;6, and if docs.fedoraproject.org can be hosted on a server that runs Red Hat Enterprise Linux&nbsp;6, we will no longer need to maintain Publican and its dependencies in dist-5E-epel-infra.
* '''perl-HTML-FormatText-WithLinks-0.14-9.el6'''
* '''perl-HTML-FormatText-WithLinks-AndTables-0.02-5.el6'''
* wkhtmltopdf-qt-4.8.0-6.git20111107.el6
* wkhtmltopdf-0.10.0_rc2-8.el6
* perl-HTML-Tree-4.2-1.el6
* '''perl-Locale-Msgfmt-0.15-9.el6'''
* docbook5-style-xsl-1.78.1-2.el6
* perl-Archive-Tar-1.92-4.el6
* perl-XML-TreeBuilder-5.1-0.el6
* perl-Test-Differences-0.5000-11.el6
* perl-Syntax-Highlight-Engine-Kate-0.08-4.el6
* perl-Compress-Raw-Bzip2-2.062-2.el6
* perl-Compress-Raw-Zlib-2.062-2.el6
* perl-IO-Compress-2.062-2.15.el6
* publican-4.0.0-0.el6


Also, if Publican 2 ships with Red Hat Enterprise Linux&nbsp;6, we could build the documentation packages for EPEL as well, assuming that we could get them into Fedora first.


[[Category:Docs Project]]
[[Category:Docs Project]]

Latest revision as of 00:29, 4 February 2014

This page tracks progress towards implementing a fully automated publishing process for the Fedora Documentation Project.

Background

Formal Fedora documentation is authored in DocBook XML and transformed into multi-page HTML, single-page HTML, PDF, and EPUB by our publishing tool, Publican. Publican also maintains the Fedora documentation site (docs.fedoraproject.org) and automatically generates and updates the navigation menus in each language.

At present, the entire documentation suite hosted at http://docs.fedoraproject.org resides in a Git repository to which certain members of the Documentation Project (members of the docs-publishers group) commit changes. The webserver publishes whatever is in that repo.

This is a vast improvement over our previous process, but Publican includes features that make publishing even easier and more robust, since Publican is designed to package documentation in RPM packages to install on a webserver. Chapter 6 of the Publican Users' Guide provides an overview of the mechanism.

Important -- note that since the webserver runs Red Hat Enterprise Linux 6, the documentation packages need to be built for Red Hat Enterprise Linux 6. Publican is also designed to build documentation to install for desktop use, but this is a separate set of considerations and not the aim here.

Steps to automate publishing

Install Publican 4.0 on the webserver

The version of Publican shipped in Red Hat Enterprise Linux 6 is very old (2.1) and lacks many bug fixes and enhancements that have been added since then (current upstream is 4.0, due in Fedora in the next few weeks).

Some of the Publican 4.0 deps in Red Hat Enterprise Linux 6 are too old or not available on all arches; we will need to build these for el6-docs

It's also possible that Publican 4.0 has dependencies that are not in Red Hat Enterprise Linux 6; at all. If so, we will need to add these to EPEL.

To install Publican 4.0 on the webserver, we need to

  • get a new el6-docs build root -- rel-eng request here Done! Thanks ausil!
  • identify and build any dependencies not in el6 for EPEL
  • identify and build any dependencies too old or not on all arches in el6 for el6-docs.
  • build Publican 4.0 and the Fedora brand package for el6-docs.

Dependencies not in el6

  • wkhtmltopdf
    • wkhtmltopdf-qt
  • perl-DateTime-Format-DateParse
  • perl-Locale-Msgfmt (branch requested for EPEL)
  • perl-Lingua-EN-Fathom (coming in EPEL)
  • perl-XML-Catalog (for XML::TreeBuilder, coming in EPEL)
  • perl-IO-Compress
    • perl-Compress-Raw-Bzip2
    • perl-Compress-Raw-Zlib

Dependencies to update in el6

  • perl-HTML-Tree -- too old
    • perl-XML-TreeBuilder -- too old
  • docbook5-style-xsl -- too old
  • perl-Archive-Tar -- too old
  • perl-Syntax-Highlight-Engine-Kate -- too old
    • perl-Test-Differences (for Syntax-Highlighting-Kate) -- too old


Update the Fedora brand

The Fedora brand needs some minor updates to make it Publican 4 ready. Probably just book titles in HTML, and renaming a CSS file for the -web subpackage.

  • needs fonts-overpass to go stable in EPEL (requested)

Configure an Publican-generated website on the webserver

Publican can automatically generate its own website structure on the server. This consists of:

  • a configuration file at /etc/publican-website.cfg
  • an SQLite database at /var/lib/publican/publican-website.db
  • the /var/www/html/docs directory in which Publican publishes documentation


Set up webserver for automatic updates

The webserver should check the el6-docs repo on a cronjob to look for new, updated, or deleted packages. Rudi can supply a script for this.

Build the docs packages

As a team, we would need to build all the docs packages in Koji. I think as long as we do all packages for F19 and F20, we can proceed to the next step (cutover) and gradually rebuild our legacy docs over a longer period of time.

Cutover

When the documentation for currently maintained versions (F19 and F20) has all been rebuilt, we request that Infra points

docs.fedoraproject.org to /var/www/html/docs on the server.


Contents of el6-docs

As a footnote, these are the packages we are carrying in el6-docs. Ones marked in bold can (and should) be carried in EPEL itself -- we're working on this!

  • docbook-style-xsl-1.78.1-2.el6
  • perl-HTML-FormatText-WithLinks-0.14-9.el6
  • perl-HTML-FormatText-WithLinks-AndTables-0.02-5.el6
  • wkhtmltopdf-qt-4.8.0-6.git20111107.el6
  • wkhtmltopdf-0.10.0_rc2-8.el6
  • perl-HTML-Tree-4.2-1.el6
  • perl-Locale-Msgfmt-0.15-9.el6
  • docbook5-style-xsl-1.78.1-2.el6
  • perl-Archive-Tar-1.92-4.el6
  • perl-XML-TreeBuilder-5.1-0.el6
  • perl-Test-Differences-0.5000-11.el6
  • perl-Syntax-Highlight-Engine-Kate-0.08-4.el6
  • perl-Compress-Raw-Bzip2-2.062-2.el6
  • perl-Compress-Raw-Zlib-2.062-2.el6
  • perl-IO-Compress-2.062-2.15.el6
  • publican-4.0.0-0.el6
Retrieved from "https://fedoraproject.org/w/index.php?title=Automating_publishing&oldid=369016"