Latest revision as of 20:30, 1 April 2017

Unified software database for DNF and PackageKit

Contact information

Name: Eduard Čuba
Email Address: edynox AT fedoraproject DOT org
Blog URL: https://edynox.github.io
IRC: edynox - #yum

Mentor: Jaroslav Mráček - jmracek AT redhat DOT com

Abstract

Nowadays DNF is using obsoleted YUM databases. PackageKit is accessing flat YUM database yumdb in very hacky way.

Therefore there is no connection between DNF and PackageKit transactions causing unintended user installed package removal hazard - F24 blocker.

I'm working on solution for replacing these old YUM databases with new, universal one.

New design called "Unified Software database for DNF and PackageKit" or shortly "SWDB" replaces flat "yumdb", "history.sqlite" and "groups.json" databases.

Project

Problem

SWDB is only integrated into DNF, PackageKit integration is missing. Integration is done using dnf-swdb library (going to be part of libdnf), which is providing all required functionality.
When dnf-swdb will be merged with DNF upstream, there will be need to provide support for PackageKit to avoid package conflicts and unintended removals.
There is no shared database API between DNF and PackageKit. PackageKit is using libdnf modules to operate yumdb, DNF has its own ones.

Solution

Connect libdnf modules used by PackageKit with dnf-swdb library.
Use dnf-swdb shared API to access SWDB in both package managers
Correctly log every PackageKit and DNF transaction into SWDB
Provide transaction history support for PackageKit shared with DNF

Goal and benefits

Better DNF and PackageKit(PK) integration.
With SWDB integrated both in DNF and PK, it should be possible to install packages via PK and list, undo, redo or rollback these transactions with DNF.
No unintended user installed package removal hazard
Better database integrity - shared API
Better performance and reduced disk usage
No data duplicity

Related work

libdnf dnf-swdb modules
- Active pull request: https://github.com/rpm-software-management/libdnf/pull/199
I presented SWDB project at developer conference DevConf 2017 in Brno.
- Slides
Nowadays I'm finalizing integration of dnf-swdb into DNF.
- Active pull request: https://github.com/rpm-software-management/dnf/pull/630

About me

Student of Brno University of Technology, Faculty of Information Technology - Czech Republic.
7 years of robotics at high school (8 year). I loved robot programming so I decided for career in IT.
This is my first Google Summer of Code proposal
Contributing to project SWDB for DNF since May 2016
Some of my contributions:
I like when my work is my hobby and when I can see real impact of my work
I prefer to work in calm environment - using notes and emails for work organization

Why fedora and this project?

Fedora became my favorite operating system at university. I was excited that even my work can be part of it.

We are facing new requirements for software every day and sometimes it is really hard to keep things organized. Package manager YUM is great example of this issue.

While significant part if YUM is already rewritten, YUM databases remained. Outdated design, data duplicity and no shared API is dangerous combination for cooperation of multiple package managers.

SWDB is providing new design suitable for current needs of DNF. Problem is, that DNF is not only package manager in the system.

Gnome Software Package Manager is probably more convenient way how to install package for regular user. It is using PackageKit which is logging into outdated yumdb, so DNF will not remove user installed packages.

Unifying DNF software databases is opportunity to make connection between PackageKit and DNF stronger and safer.

I want to continue providing support for SWDB after GSoC, maybe start a new fedora project for my bachelor's thesis.

Schedule

Weeks	Due date	Task	Description	Deliverable
1 - 2	June 9.	Design and research	Find out what needs to be changed	List of required changes
2 - 4	June 23.	Module integration	dnf-swdb integration with libdnf - evaluation	SWDB integrated in libdnf using compilation switch
4 - 6	July 7.	Writing transactions	Writing transaction and package data from PackageKit into SWDB	PK trans data being logged into SWDB
6 - 8	July 21.	Reading data	Providing data from swdb to PackageKit(installed packages, updates etc.) - evaluation	PK trans data being loaded from SWDB
8 - 10	August 4.	DNF-PK interoperation	DNF-PK cooperation debugging (conflicts, RO mode, packages uninstalled with RPM)	Reading PK trans data from DNF, preview
10 - 12	August 18.	Preparation for release	Testing and bugfixing (CI DNF stack tests, data migration from various versions - Fedora, RHEL)	Continuous integration tests
13	August 21 - 29.	Submission	Final Evaluation

Why me?

strong experience with this project
knowledge of DNF and libdnf source code

My plans

I don't have any specific plans for summer. I will be able to work as required. From time to time I will need a free day for travelling between my university in Czech Republic and home in Slovakia.

@@ Line 6: / Line 6: @@
 * Name: Eduard Čuba
 * Email Address: ''edynox'' AT ''fedoraproject'' DOT ''org''
-* Blog URL: [https://edynox.github.io edynox.github.io]
+* Blog URL: https://edynox.github.io
 * IRC: ''edynox'' - #yum
+* Mentor: Jaroslav Mráček - ''jmracek'' AT ''redhat'' DOT ''com''
 == Abstract ==
-Nowadays DNF is using old YUM databases.
+Nowadays DNF is using obsoleted YUM databases.
-PackageKit is accessing yumdb in very hacky way.
+PackageKit is accessing flat YUM database yumdb in very hacky way.
-Therefore there is no connection between DNF and PackageKit
-transactions and unintended user installed package removal hazard ([https://bugzilla.redhat.com/show_bug.cgi?id=1259865 F24 blocker]).
+Therefore there is no connection between DNF and PackageKit transactions causing unintended
+user installed package removal hazard - [https://bugzilla.redhat.com/show_bug.cgi?id=1259865 F24 blocker].
 I'm working on solution for replacing these old YUM databases with new, universal one.
-New database called ''SWDB'' replaces flat ''yumdb'', ''history sqlite'' and ''groups'' json databases.
-== Goal and benefits ==
+New design called "Unified Software database for DNF and PackageKit" or shortly "SWDB" replaces flat "yumdb", "history.sqlite" and "groups.json" databases.
-'''Better DNF and PackageKit integration.'''
+== Project ==
-With this database integrated both in DNF and PackageKit, it should be possible to install packages
+=== Problem ===
-via PackageKit and list, undo, redo or rollback these transactions with DNF.
+* SWDB is only integrated into DNF, PackageKit integration is missing. Integration is done using dnf-swdb library (going to be part of libdnf), which is providing all required functionality.
+* When dnf-swdb will be merged with DNF upstream, there will be need to provide support for PackageKit to avoid package conflicts and unintended removals.
+* There is no shared database API between DNF and PackageKit. PackageKit is using libdnf modules to operate yumdb, DNF has its own ones.
-== Problem ==
+=== Solution ===
-Swdb is only integrated into DNF using dnf-swdb library (part of libdnf), which is providing all
+* Connect libdnf modules used by PackageKit with dnf-swdb library.
-required functionality to operate with swdb. PackageKit integration is missing.
+* Use dnf-swdb shared API to access SWDB in both package managers
+* Correctly log every PackageKit and DNF transaction into SWDB
+* Provide transaction history support for PackageKit shared with DNF
-When dnf-swdb will be merged with DNF upstream, there will be need to provide
+== Goal and benefits ==
-support for PackageKit to avoid package conflicts and unintended removals.
-PackageKit is using libdnf modules to operate yum and history database.
-I would like to:
+* Better DNF and PackageKit(PK) integration.
-* rewrite libdnf modules to cooperate with dnf-swdb
+* With SWDB integrated both in DNF and PK, it should be possible to install packages via PK and list, undo, redo or rollback these transactions with DNF.
-* provide transaction history support for PackageKit shared with DNF
+* No unintended user installed package removal hazard
+* Better database integrity - shared API
+* Better performance and reduced disk usage
+* No data duplicity
 == Related work ==
-* [https://github.com/edynox/libdnf libdnf dnf-swdb modules]
+* libdnf [https://github.com/edynox/libdnf dnf-swdb modules]
-* Project was presented at developer conference DevConf. [http://www.stud.fit.vutbr.cz/~xcubae00/devconf.pdf Slides]
+** Active pull request: https://github.com/rpm-software-management/libdnf/pull/199
+* I presented ''SWDB'' project at developer conference DevConf 2017 in Brno.
-Nowadays I'm finalizing integration of dnf-swdb into [https://github.com/edynox/dnf DNF].
+** [http://www.stud.fit.vutbr.cz/~xcubae00/devconf.pdf Slides]
+* Nowadays I'm finalizing integration of dnf-swdb into  [https://github.com/edynox/dnf DNF].
+** Active pull request: https://github.com/rpm-software-management/dnf/pull/630
 == About me ==
 * Student of Brno University of Technology, Faculty of Information Technology - Czech Republic.
-* I was doing robotics for 7 years at high school (8 year). Programming robots was fun so I decided for career in IT.
+* 7 years of robotics at high school (8 year). I loved robot programming so I decided for career in IT.
-* Contributing to project '''SWDB''' for DNF team since May 2016
+* This is my first Google Summer of Code proposal
+* Contributing to project SWDB for DNF since May 2016
 * Some of my contributions:
 ** [https://github.com/rpm-software-management/dnf/pull/630 My work on DNF-SWDB]
@@ Line 63: / Line 73: @@
 Fedora became my favorite operating system at university. I was excited that even my work can be part of it.
 We are facing new requirements for software every day and sometimes it is really hard to keep things organized.
-Package manager YUM is great example of this issue. While significant part if YUM is already rewritten, YUM databases
+Package manager YUM is great example of this issue.
+While significant part if YUM is already rewritten, YUM databases
 remained. Outdated design, data duplicity and no shared API is dangerous combination for cooperation of multiple
-package managers. SWDB is providing new design suitable for current needs of DNF. Problem is, that DNF is not only
+package managers.
-package manager in the system. Gnome Software Package Manager is probably more convenient way how to install package for regular user. It is using PackageKit and PackageKit is logging into outdated yumdb, so DNF will not remove user installed
-packages. Unifying DNF software databases is opportunity to make connection between PackageKit and DNF stronger and safer.
+SWDB is providing new design suitable for current needs of DNF. Problem is, that DNF is not only
+package manager in the system.
+Gnome Software Package Manager is probably more convenient way how to install package for regular user.
+It is using PackageKit which is logging into outdated yumdb, so DNF will not remove user installed
+packages.
+Unifying DNF software databases is opportunity to make connection between PackageKit and DNF stronger and safer.
 I want to continue providing support for SWDB after GSoC, maybe start a new fedora project for my bachelor's thesis.
@@ Line 79: / Line 99: @@
 ! style="text-align: center;" | Task
 ! style="text-align: center;" | Description
+! style="text-align: center;" | Deliverable
 |-
 | style="text-align: center;" | 1 - 2
@@ Line 84: / Line 105: @@
 | style="text-align: center;" | Design and research
 | style="text-align: center;" | Find out what needs to be changed
+| style="text-align: center;" | List of required changes
 |-
 | style="text-align: center;" | 2 - 4
 | style="text-align: center;" | June 23.
 | style="text-align: center;" | Module integration
-| style="text-align: center;" | dnf-swdb integration with libdnf
+| style="text-align: center;" | dnf-swdb integration with libdnf - '''evaluation'''
+| style="text-align: center;" | ''SWDB'' integrated in libdnf using compilation switch
 |-
 | style="text-align: center;" | 4 - 6
 | style="text-align: center;" | July 7.
 | style="text-align: center;" | Writing transactions
-| style="text-align: center;" | Writing transaction and package data from PackageKit into swdb
+| style="text-align: center;" | Writing transaction and package data from PackageKit into ''SWDB''
+| style="text-align: center;" | PK trans data being logged into ''SWDB''
 |-
 | style="text-align: center;" | 6 - 8
 | style="text-align: center;" | July 21.
 | style="text-align: center;" | Reading data
-| style="text-align: center;" | Providing data from swdb to PackageKit(installed packages, updates etc.)
+| style="text-align: center;" | Providing data from swdb to PackageKit(installed packages, updates etc.) - '''evaluation'''
+| style="text-align: center;" | PK trans data being loaded from ''SWDB''
 |-
 | style="text-align: center;" | 8 - 10
@@ Line 104: / Line 129: @@
 | style="text-align: center;" | DNF-PK interoperation
 | style="text-align: center;" | DNF-PK cooperation debugging (conflicts, RO mode, packages uninstalled with RPM)
+| style="text-align: center;" | Reading PK trans data from DNF, preview
 |-
 | style="text-align: center;" | 10 - 12
-| style="text-align: center;" | August 21.
+| style="text-align: center;" | August 18.
 | style="text-align: center;" | Preparation for release
-| style="text-align: center;" | Testing and bugfixing (data migration from various versions - Fedora, RHEL, CentOs)
+| style="text-align: center;" | Testing and bugfixing (CI DNF stack tests, data migration from various versions - Fedora, RHEL)
+| style="text-align: center;" | Continuous integration tests
+|-
+| style="text-align: center;" | 13
+| style="text-align: center;" | August 21 - 29.
+| style="text-align: center;" | Submission
+| style="text-align: center;" | Final Evaluation
+| style="text-align: center;" |
 |}
 == Why me? ==
-I have strong experience with this project.
+* strong experience with this project
+* knowledge of DNF and libdnf source code
 == My plans ==

Search

GSOC 2017/Student Application edynox: Difference between revisions