| Chapter 9. Advanced packaging | ||
|---|---|---|
 |
 |
|
| Chapter 9. Advanced packaging | ||
|---|---|---|
| |
|
|
Table of Contents
Let’s describe advanced topics on Debian packaging.
Let me oversimplify historical perspective of Debian packaging practices focused on the non-native packaging.
Debian was started in 1990s when upstream packages were available from public FTP sites such as Sunsite. In those early days, Debian packaging used Debian source format currently known as the Debian source format “1.0”:
The Debian source package ships a set of files for the Debian source package.
The modern Debian source format “3.0 (quilt)” was invented around 2008 (see “ProjectsDebSrc3.0”):
The Debian source package ships a set of files for the Debian source package.
package_version-revision.debian.tar.?z : tarball of debian/ for Debian modifications.
Most Debian packages adopted the Debian source formats “3.0 (quilt)” and “3.0 (native)”.
Now, the git(1) is popular with upstream and Debian developers. The git and its associated tools are important part of the modern Debian packaging workflow. This modern workflow involving git will be mentioned later in “Chapter 10, Packaging with git”.
Current Debian packaging practices and their trends are moving target. See:
“Debian Trends” — Hints for “De facto standard” of Debian practices
You can also search entire Debian source code data by yourself, too.
“Debian Sources” — code search tool
Auto-generated files of the build system may be found in the released upstream tarball. These should be regenerated when Debian package is build. E.g.:
Some modern build system may be able to download required source codes and binary files from arbitrary remote hosts to satisfy build requirements. Don’t use this download feature. The official Debian package is required to be build only with packages listed in Build-Depends: of the debian/control file.
The dh_auto_test(1) command is a debhelper command that tries to automatically run the test suite provided by the upstream developer during the Debian package building process.
The autopkgtest(1) command can be used after the Debian package building process. It tests generated Debian binary packages in the virtual environment using the debian/tests/control RFC822-style metadata file as continuous integration (CI). See:
There are several other CI tools on Debian for you to explore.
Debian cares about supporting new ports or flavours. The new ports or flavours require bootstrapping operation for the cross-build of the initial minimal native-building system. In order to avoid build-dependency loops during bootstrapping, the build-dependency needs to be reduced using the DEB_BUILD_PROFILES environment variable.
See Debian wiki: “BuildProfileSpec”.
![[Tip]](images/tip.png) | Tip |
|---|---|
|
If a core package foo build depends on a package bar with deep build dependency chains but bar is only used in the test target in foo, you can safely mark the bar with <!nocheck> in the Build-depends of foo to avoid build loops. |
The compiler hardening support spreading for Debian jessie (8.0) demands that we pay extra attention to the packaging.
You should read the following references in detail.
The debmake command adds template comments to the debian/rules file as needed for DEB_BUILD_MAINT_OPTIONS, DEB_CFLAGS_MAINT_APPEND, and DEB_LDFLAGS_MAINT_APPEND (see “Chapter 5, Simple packaging” and dpkg-buildflags(1)).
Here are some recommendations to attain a reproducible build result.
Read more at “ReproducibleBuilds”.
The control file source-name_source-version_arch.buildinfo generated by dpkg-genbuildinfo(1) records the build environment. See deb-buildinfo(5)
The debian/control file also defines the package dependency in which the “variable substitutions mechanism” (substvar) may be used to free package maintainers from chores of tracking most of the simple package dependency cases. See deb-substvars(5).
The debmake command supports the following substvars:
For the shared library, required libraries found simply by “objdump -p /path/to/program | grep NEEDED” are covered by the shlib substvar.
For Python and other interpreters, required modules found simply looking for lines with “import”, “use”, “require”, etc., are covered by the corresponding substvars.
For other programs which do not deploy their own substvars, the misc substvar covers their dependency.
For POSIX shell programs, there is no easy way to identify the dependency and no substvar covers their dependency.
For libraries and modules required via the dynamic loading mechanism including the “GObject introspection” mechanism, there is no easy way to identify the dependency and no substvar covers their dependency.
Packaging library software requires you to perform much more work than usual. Here are some reminders for packaging library software:
Before packaging shared library software, see:
For the historic background study, see:
“Escaping the Dependency Hell” [17]
“Debian Library Packaging guide” [18]
Multiarch support for cross-architecture installation of binary packages (particularly i386 and amd64, but also other combinations) in the dpkg and apt packages introduced in Debian wheezy (7.0, May 2013), demands that we pay extra attention to packaging.
You should read the following references in detail.
Ubuntu wiki (upstream)
Debian wiki (Debian situation)
The multiarch is enabled by using the <triplet> value such as i386-linux-gnu and x86_64-linux-gnu in the install path of shared libraries as /usr/lib/<triplet>/, etc..
The <triplet> value used in override_dh_* target scripts must be explicitly set in the debian/rules file by the maintainer. The <triplet> value is stored in the $(DEB_HOST_MULTIARCH) variable in the following debian/rules snippet example:
DEB_HOST_MULTIARCH = $(shell dpkg-architecture -qDEB_HOST_MULTIARCH) ... override_dh_install: mkdir -p package1/lib/$(DEB_HOST_MULTIARCH) cp -dR tmp/lib/. package1/lib/$(DEB_HOST_MULTIARCH)
See:
For well behaving build systems, the split of a Debian binary package into small ones can be realized as follows.
Please check examples in this guide:
An intuitive and flexible method to create the initial template debian/control file defining the split of the Debian binary packages is accommodated with the -b option. See “Section 15.2, “debmake -b””.
Here are some typical multiarch package split scenarios for the following upstream source examples using the debmake command:
| binarypackage | type | Architecture: | Multi-Arch: | Package content |
|---|---|---|---|---|
libfoo1 | lib* | any | same | the shared library, co-installable |
libfoo-dev | dev* | any | same | the shared library header files etc., co-installable |
libfoo-tools | bin* | any | foreign | the run-time support programs, not co-installable |
libfoo-doc | doc* | all | foreign | the shared library documentation files |
bar | bin* | any | foreign | the compiled program files, not co-installable |
bar-doc | doc* | all | foreign | the documentation files for the program |
baz | script | all | foreign | the interpreted program files |
Debian policy requires to comply with the “Filesystem Hierarchy Standard (FHS), version 3.0”, with the exceptions noted in “File System Structure”.
The most notable exception is the use of /usr/lib/<triplet>/ instead of /usr/lib<qual>/ (e.g., /lib32/ and /lib64/) to support a multiarch library.
Table 9.1. The multiarch library path options
| Classic path | i386 multiarch path | amd64 multiarch path |
|---|---|---|
/lib/ | /lib/i386-linux-gnu/ | /lib/x86_64-linux-gnu/ |
/usr/lib/ | /usr/lib/i386-linux-gnu/ | /usr/lib/x86_64-linux-gnu/ |
For Autotools based packages under the debhelper package (compat>=9), this path setting is automatically taken care by the dh_auto_configure command.
For other packages with non-supported build systems, you need to manually adjust the install path as follows.
All files installed simultaneously as the multiarch package to the same file path should have exactly the same file content. You must be careful with differences generated by the data byte order and by the compression algorithm.
The shared library files in the default path /usr/lib/ and /usr/lib/<triplet>/ are loaded automatically.
For shared library files in another path, the GCC option -l must be set by the pkg-config command to make them load properly.
GCC includes both /usr/include/ and /usr/include/<triplet>/ by default on the multiarch Debian system.
If the header file is not in those paths, the GCC option -I must be set by the pkg-config command to make "#include <foo.h>" work properly.
Table 9.2. The multiarch header file path options
| Classic path | i386 multiarch path | amd64 multiarch path |
|---|---|---|
/usr/include/ | /usr/include/i386-linux-gnu/ | /usr/include/x86_64-linux-gnu/ |
/usr/include/packagename/ | /usr/include/i386-linux-gnu/packagename/ | /usr/include/x86_64-linux-gnu/packagename/ |
/usr/lib/i386-linux-gnu/packagename/ | /usr/lib/x86_64-linux-gnu/packagename/ |
The use of the /usr/lib/<triplet>/packagename/ path for the library files allows the upstream maintainer to use the same install script for the multiatch system with /usr/lib/<triplet> and the biarch system with /usr/lib<qual>/. [19]
The use of the file path containing packagename enables having more than 2 development libraries simultaneously installed on a system.
The pkg-config program is used to retrieve information about installed libraries in the system. It stores its configuration parameters in the *.pc file and is used for setting the -I and -l options for GCC.
Table 9.3. The *.pc file path options
| Classic path | i386 multiarch path | amd64 multiarch path |
|---|---|---|
/usr/lib/pkgconfig/ | /usr/lib/i386-linux-gnu/pkgconfig/ | /usr/lib/x86_64-linux-gnu/pkgconfig/ |
The symbols support in dpkg introduced in Debian lenny (5.0, May 2009) helps us to manage the backward ABI compatibility of the library package with the same package name. The DEBIAN/symbols file in the binary package provides the minimal version associated with each symbol.
An oversimplified method for the library packaging is as follows.
Extract the old DEBIAN/symbols file of the immediate previous binary package with the “dpkg-deb -e” command.
Copy it to the debian/binarypackage.symbols file.
Build the binary package.
If the dpkg-gensymbols command warns about some new symbols:
If the dpkg-gensymbols command does not warn about new symbols:
For the details, you should read the following primary references.
You should also check:
| Tip |
|---|---|
|
For C++ libraries and other cases where the tracking of symbols is problematic, follow “8.6.4 The shlibs system” of the “Debian Policy Manual”, instead. Please make sure to erase the empty debian/binarypackage.symbols file generated by the debmake command. For this case, the DEBIAN/shlibs file is used. |
Let’s consider that the upstream source tarball of the libfoo library is updated from libfoo-7.0.tar.gz to libfoo-8.0.tar.gz with a new SONAME major version which affects other packages.
The binary library package must be renamed from libfoo7 to libfoo8 to keep the unstable suite system working for all dependent packages after the upload of the package based on the libfoo-8.0.tar.gz.
![[Warning]](images/warning.png) | Warning |
|---|---|
|
If the binary library package isn’t renamed, many dependent packages in the unstable suite become broken just after the library upload even if a binNMU upload is requested. The binNMU may not happen immediately after the upload due to several reasons. |
The -dev package must follow one of the following naming rules:
Use the unversioned -dev package name: libfoo-dev
Only one version of the library source package is allowed in the archive.
Use the versioned -dev package names: libfoo7-dev and libfoo8-dev
Two versions of the library source packages are allowed simultaneously in the archive.
| Tip |
|---|---|
|
If the data encoding scheme changes (e.g., latin1 to utf-8), the same care as the API change needs to be taken. |
When you package a new library package version which affects other packages, you must file a transition bug report against the release.debian.org pseudo package using the reportbug command with the ben file and wait for the approval for its upload from the Release Team.
Release team has the “transition tracker”. See “Transitions”.
![[Caution]](images/caution.png) | Caution |
|---|---|
|
Please make sure to rename binary packages as in “Section 9.17, “Library package name””. |
A “binNMU” is a binary-only non-maintainer upload performed for library transitions etc. In a binNMU upload, only the “Architecture: any” packages are rebuilt with a suffixed version number (e.g. version 2.3.4-3 will become 2.3.4-3+b1). The “Architecture: all” packages are not built.
The dependency defined in the debian/control file among binary packages from the same source package should be safe for the binNMU. This needs attention if there are both “Architecture: any” and “Architecture: all” packages involved in it.
“Architecture: any” package: depends on “Architecture: any” foo package
“Architecture: any” package: depends on “Architecture: all” bar package
“Architecture: all” package: depends on “Architecture: any” baz package
The Debian package is built with the debugging information but packaged into the binary package after stripping the debugging information as required by “Chapter 10 - Files” of the “Debian Policy Manual”.
See
The debugging information is automatically packaged separately as the debug package using the dh_strip command with its default behavior. The name of such a debug package normally has the -dbgsym suffix.
The debconf package enables us to configure packages during their installation in 2 main ways:
interactively from the menu interface (dialog, gnome, kde, …)
All user interactions for the package installation must be handled by this debconf system using the following files.
debian/binarypackage.config
debian/binarypackage.template
These debconf files are called by package configuration scripts in the binary Debian package
See dh_installdebconf(1), debconf(7), debconf-devel(7) and “3.9.1 Prompting in maintainer scripts” in the “Debian Policy Manual”.
[17] This document was written before the introduction of the symbols file.
[18] The strong preference is to use the SONAME versioned -dev package names over the single -dev package name in “Chapter 6. Development (-DEV) packages”, which does not seem to be shared by the former ftp-master (Steve Langasek). This document was written before the introduction of the multiarch system and the symbols file.
[19] This path is compliant with the FHS. “Filesystem Hierarchy Standard: /usr/lib : Libraries for programming and packages” states “Applications may use a single subdirectory under /usr/lib. If an application uses a subdirectory, all architecture-dependent data exclusively used by the application must be placed within that subdirectory.”
| |
|
|
| Chapter 8. More on packaging |
|
Chapter 10. Packaging with git |