Packaging for PCLinuxOS

From PCLinuxOSHelp Knowledge Base
Jump to: navigation, search
  • Relevant to all editions of PCLinuxOS.

A packager is the life blood of PCLinuxOS. Packager is the one who keeps PCLinuxOS updated. Learning to package is an important step towards making useful contributions to the betterment of PCLinuxOS.

The whole process is easy to learn with a little attention to detail and perseverance. In case you run into difficulties just go to the forum to get help.

Introduction

Packaging is the process of preparing installable packages from source code of individual software. It is an umbrella word used to cover all the things including building right from the source to adjusting spec files and making a PCLinuxOS package out of RPMs available elsewhere to updating existing PCLinuxOS packages.

In order to become a successful packager, it is necessary to have patience. Do not give up easily. If you do not succeed, try again and again. It is also very important to read a lot initially until your mind gets the picture.

Even if initially you do not understand whatever you are reading or supposed to do, continue reading or doing it. The puzzle will be solved in the end.

Rough overview

Please understand that this section is only aimed at providing an approximate picture of what and how building a package works. This is unlikely to be perfectly correct and hence you have to verify the correctness of this information on your own.

During packaging, we convert original software/items from coders into a form which is ready for download and installation by end users.

Now the original software/items could be media (wallpapers, icons, sounds etc.) or software (binary executables, source code) The most common use of packaging however is to convert source into installable .rpms for end users.

  • Software from source - Most of the software written by humans is in the form of a human readable language such as Assembly, C / C++, Java / Python. However a computer will only understand instructions in machine language which is in binary form. So we need to convert the software from human readable languages to machine language. This is done by a method known generally as compiling / building.
- In case of C / C++ etc., this conversion is done in the process of packaging by compiling / building.
- Java looks like a gray area what with there being an intermediate conversion to bytecode. Please read this guide for more information on packaging Java programs - Creating an RPM for a Java Application' - http://javaworkshop.wordpress.com/2008/10/22/rolling-up-an-rpm-for-a-java-application/
- In case of interpreted languages, this conversion is done on the fly during their execution in the user's computer. Hence here the source is packaged as is.
Now this building can happen at 3 different levels :
- At the original author of software : Most commercial software is distributed pre-built.
- At the distributor : That is what our Bacon Brigade does.
- At the end user : Luckily users of PCLinuxOS do not have to do this. This is the standard method in source based distros.
So in our case the following is done :
- Software is procured from source by the packager.
- Source is extracted to BUILD area and build commands run there to get the target software.
- A directory structure mimicking the user's computer is created inside BUILDROOT. The target software is now installed into BUILDROOT as if it is getting installed into the user's computer.
- The resulting files are picked out of this BUILDROOT environment and put together inside an archive and this is called a package. This will be found in /src/rpm/RPMS/i586 or x64 depending on the target.
- A copy of the source code along with the .spec file used to create this package is placed in /src/rpm/SPRM. This is what is used by the maintainer of PCLinuxOS to recreate the same package and enter it into the official repository after quality control checks.
  • Software binaries and media - Things are slightly different when it comes to binary packages and media as they already come in compiled binary form. They mostly only need to be placed into their proper location.

Approach to learning

Please read through the complete guide. Parts of the guide are verbatim or barely modified posts from the forum by the authors referred. Considering the high level of knowledge of the respective authors, it might seem intimidating at first. (I had a similar experience too). Just read the rest of the guide.

  • To begin with you should learn rebuilding something which is already present in our repository. Look for software of size less than 1 MB at first.
  • Next you should try to package something similar by your own. Try very simple items containing just 1 or 2 files.
  • Next you should try updating something in the present repository. (You will need to study the version numbers of existing packages and compare them to those from the official websites in order to know which can be updated.)
  • Now you could try and package a software not present in the repository or without looking at the already packaged version.
  • Once you are comfortable here, try packaging libraries after understanding about them.
  • Now you could try your hand at very complex and interlinked packaging tasks such as whole desktop environments, wine etc.
  • At this level, you could try to package binary software such as graphic drivers etc.

New packagers - How to get started

Get an idea of our repository set-up by reading this page Repository and Sections and Mirrors.

Posted by Neal on the forum

  • Go to http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/ and scroll down to the SRPMS.(*) sections. Clicking any of those links will open that section, where you will find SRPMs for PCLinuxOS.
  • These are the best src.rpms to start out with learning to package upto PCLinuxOS standards. Study them. Look closely at how the spec files are written.
  • You should not start with an SRPM from another distro, as these will require changes you may be unaware of. They often use different naming conventions, so that the BuildRequires and Requires in their spec files will call for something that has a different name in PCLinuxOS. They may install to different directories, which will cause difficulties, also.
  • The point is that you should start with learning to rebuild PCLinuxOS packages, proceed to learning how to upgrade our packages and only after learning how to create a PCLinuxOS package should you attempt to convert an SRPM from elsewhere.
  • Same folder from other mirrors of PCLinuxOS can also be used instead of NLUGG.

Setting up the build environment

PCLinuxOS provides a set of packages to help with the process of setting up a build environment and building packages. The package pkgutils provides a set of helper scripts which are useful for carrying out functions such as setting up the build environment, installing build dependencies and converting archive types. Some of the helper scripts provided are:

  • addpkg - Install required development packages (-devel) for building SPECFILE or RPMFILE. This can also create a file of the type .remove which can be later used to remove those packages after the build using the rmvpkg script.
  • gbd -Uses genbasedir to update a local repository.
  • mkrepo - Set up the build environment including creating a local repository folder structure and .rpmmacros files and optionally update sources.list file. Note: in pkgutils 2.0 this has been replaced with pkgutils-setup which does the same job without the user having to answer unnecessary prompts.
  • rmvpkg -Remove development packages installed by addpkg.

In addition there are desktop specific packages (for example pkgutils-kde) which provide useful "right click" menu options for the packaging file-types (e.g. SPEC files, SRPMs etc.).

Please follow these steps to set up the build environment with pkgutils :

  • Download and install the KDE Darkstar on a separate partition.
  • Install pkgutils-kde or pkgutils-thunar which will also install pkgutils.
  • The package install automatically runs mkrepo. Select the user name by clicking on it when prompted and then answer the prompts to setup that user as packager. All the packaging related directories will be created under src directory in this user's /home.
  • NOTE : Even if there is only one user in this installation, it is necessary to click and highlight the user name. Else the process of setup will be cancelled and you will need to re-do it from commandline using mkrepo command.
  • Avoid using special characters in the user name of packaging environment.
  • If everything went correctly, the /home/xxxx where xxxx is the packaging user name will have a new directory called src. If you go deeper into it, there should be apt rpm tmp.
apt should be your local repository for self created packages.
rpm is where your packaging will happen.
tmp is a temporary directory.
  • /home/xxxx/src/rpm/RPMS will store built .rpms either in i586 or x86_64 depending on which architecture you are packaging for.
  • /home/xxxx/src/rpm/SOURCES will store your sources in tar.xz format for building.
  • /home/xxxx/src/rpm/SPECS will store the .spec files needed for the build.
  • /home/xxxx/src/rpm/SRPMS will store the final .srpms. These will be needed when you have succeeded in building, testing the package and ensuring that it is bug free, to upload for the consideration of PCLinuxOS maintainer.
  • There will be 2 hidden files under /home/xxxx namely .rpmmacros and .rpmrc

Building - step by step

Now to the actual process of building a PCLinuxOS package:

If you have downloaded an existing PCLinuxOS SRPM then with pkgutils the process should be as simple as:

  • Right click the SRPM and select Install SRPM for build. This will install the package source under the src/rpm directory. The SPEC file will be in the SPECS folder and the sources in the SOURCES folder. If you are not using the GUI then the equivalent command is: rpm -i xxxxxx.src.rpm
  • Right click the SPEC file and then select Install Dependencies. This will interrogate the SPEC file and install the necessary -devel files for the build. If you are not using the GUI the equivalent command is: addpkg xxxxx.spec
  • Right click the SPEC file and then select Build All (or one of the other build options if required). This will start the build process. The output from the build can be very verbose but you will get used to it as you gain experience in packaging. If you are not using the GUI the equivalent command is: rpmbuild -ba xxxxx.spec

A successful build finish with some "Wrote: " lines showing you new SRPM and RPM files being output, followed by some cleanup lines and an exit 0 line. For example: 

Wrote: /home/terry/src/rpm/SRPMS/pkgutils-0.2.0-1pclos2020.src.rpm
Wrote: /home/terry/src/rpm/RPMS/noarch/pkgutils-0.2.0-1pclos2020.noarch.rpm
Wrote: /home/terry/src/rpm/RPMS/noarch/pkgutils-thunar-0.2.0-1pclos2020.noarch.rpm
Wrote: /home/terry/src/rpm/RPMS/noarch/pkgutils-kde5-0.2.0-1pclos2020.noarch.rpm
Executing(%clean): /bin/sh -e /home/terry/src/tmp/rpm-tmp.Y8GdNr
+ umask 022
+ cd /home/terry/src/rpm/BUILD
+ cd pkgutils-0.2.0
+ /bin/rm -rf /home/terry/src/rpm/BUILDROOT/pkgutils-0.2.0-1pclos2020.x86_64
+ exit 0


If there are errors during the build you will not get these lines and instead the output will finish with RPM build errors or error: bad exit status from: lines. For example: 

error: Bad exit status from /home/testinstall/src/tmp/rpm-tmp.bOkEi4 (%build)
RPM build errors:
   Bad exit status from /home/testinstall/src/tmp/rpm-tmp.bOkEi4 (%build)

If the build is successful the new packages will be placed in the local repository created by pkgutils. Run the gbd command to generate a new base directory index including the new packages and then they should be available to Synaptic or apt.

Gathering information

Before or during building / troubleshooting a build, you would need to have some information regarding settings, options etc. There are many places to look for such information and the location would also depend on the language, type of package etc.

  • Homepage - If the package has a well maintained homepage, it should be your first port of call.
  • Source - The source tarball or the directory will have files such as README, INSTALL, CHANGELOG which are also very good sources of information.
  • configure.ac - This file could provide very useful information about BuildRequires.
  • ./configure --help - The options available for build could be ascertained by extracting the package to a directory and then running this command from a terminal.

Dos and Don'ts

Slightly modified version of a post by Texstar on the forum :

  • Name your specfile - pclos-name-of-package.spec (example: pclos-doxygen.spec)
This makes it handy if you are updating a package and encounter a build error. You can install another distribution's source package and compare their spec file with yours to see if they added a patch or a different build option to make the new version build.
  • Convert your source tarball to tar.xz from tar.gz and tar.bz2
The public servers do not have infinite space and bandwidth. By re-compressing your source tarball with tar.xz you will generate a much smaller SRPM. This saves space on the mirrors and makes uploading the SRPMs from the build server to go much faster as there is less to upload.
  • Use proper versions of your package release
If this is a new package going into the PCLinuxOS repository even if it has been based on another distribution's specfile then your release macro for PCLinuxOS is %mkrel 1
  • Make a complete changelog
This gives information as to when the package was built, who built it, an email contact, version number and a description of what you did to the package. If you are importing a package from another distribution then all you need to do is add in the description area based on original specfile from fedora, Opensuse, mdv etc.

For example: 

%changelog
* Wed Nov 03 2010 Texstar <texstar at gmail.com> 1.3.6-1pclos2010
- new version
- patch1 merged upstream
- comment out patch2 as it does not build with it.
  • gcc/glibc updates need additional requires
When you update your gcc/glibc you will need to add the following to your specfile to force the user to update the following 2 packages to ensure the new programs compiled with gcc 4.5.2 will install and run properly on their existing system.
Requires: glibc >= 2.12.1
Requires: libstdc++6 >= 4.5.2

Neal on the forum :

  • Test your packages before uploading them.
  • If you package beta software, be sure you put a note with it saying so. We will want to add it to testing first.
  • Do not package alpha software. Period!
  • Periodically check to see if a package you have submitted has an update available. If it does, create a new package, test and submit it.
  • Don't forget your changelog entry. Be sure you use the day, date (Month and numbered day of the month) and year of the day you build the package. Add your info to the entry. Do not use the info of the previous packager.

Dependency management

Slightly modified version of a post by TerryN on the forum :

Many people who are new to packaging think that dependency management is only about the Provides: and Requires: tags in the spec file. These are only part of the story because rpm does automatic dependency generation as part of the rpmbuild process. This is handled by 2 scripts find-provides and find-requires which you will see run towards the end of the build.

  • find-provides :
    This script examines all the files in the %files list and if any shared libraries are found the soname of the lib is added to the list of capabilities provided by the rpm. This means that if another package depends on that shared lib your package can be selected by apt/synaptic to satisfy that dependency.
- Be very careful when packaging apps where the SOURCES contain pre-built libraries as they will be added and you could find your package being selected as a dependency of a seemingly totally unrelated application (Calibre springs to mind). Worse still, this can lead to instability if those libraries are used with another application (because they were not built on PCLinuxOS).
- To avoid this you can add Autoprov: No to the summary section of the spec file (that is where the Provides/BuildRequires/Requires/Obsoletes tags go) for these types of package (and only these types of package!).
Autoprov: No
- Documentation for these tags is here.
  • find-requires :
    This script examines all the files in the %files list and effectively runs an ldd ( more about ldd ) on any executable to see what shared-libraries are needed by the executable. The sonames for these libraries are added to the list of capabilities required by the rpm and will need to be resolved when the rpm is installed.
- Once again pre-built binaries in the SOURCE can cause issues as they may reference sonames which cannot be resolved by existing packages in the repo.
- An example would be because the SOURCE contains binaries for multiple architectures (see example here)
  • Devel dependencies :
There is a further twist with shared objects which are to be used as system shared libraries (there are other uses for .so files). The build will produce 3 "copies" of the lib (actually only 1 is a real file the others are links but that doesn't matter for the purposes here) :
/usr/lib/libxyz.so.1.2.3
/usr/lib/libxyz.so.1
/usr/lib/libxyz.so
The first 2 (the versioned files) are required at runtime and should be in the main package. The 3rd one (libxyz.so) is only required by the linker when building apps against the shared lib and should therefore be in the -devel package (along with header files).
RPM is clever enough to know this and generates "devel" provides/requires for that file. Therefore if you put the unversioned .so in the main package the automatic dependency management will generate a "devel" provides:
[terry@localhost ~]$ rpm -q --provides kde-workspace-core | grep devel
devel(libkwinglesutils)  
devel(libkwinglutils)  
devel(liboxygenstyleconfig)  
devel(libpowerdevilconfigcommonprivate)  
devel(libpowerdevilui)
and therefore devel dependencies:
[terry@localhost ~]$ rpm -qR kde-workspace-core | grep devel
devel(libdl)  
devel(libEGL)  
devel(libgcc_s)  
devel(libGL)  
devel(libGLESv2)  
devel(libICE)  
devel(libkdecore)  
devel(libkdeui)  
devel(libkephal)  
devel(libkwineffects)  
devel(libm)  
devel(libpowerdevilcore)  
devel(libpowerdevilui)  
devel(libQtCore)  
devel(libQtDBus)  
devel(libQtGui)  
devel(libQtSvg)  
devel(libSM)  
devel(libstdc++)  
devel(libX11)  
devel(libXau)  
devel(libXdmcp)  
devel(libXext)  
devel(libXft)  
devel(libXpm
and these devel dependencies will have their own dependencies etc. etc. This could result in a large amount of unnecessary -devel packages and other "build" packages being installed on systems which are not used for software building. This can consume a lot of space and make things like remastering more difficult.
If you are sure that nothing will ever want to link/build against such a library you could avoid creating a -devel subpackage by either deleting the .so at the end of the %install section
rm -f %buildroot/%{_libdir}/libxyz.so
or using a %exclude' for the file in the %files list.
%files
...
%exclude %{_libdir}/libxyz.so
but if there is any doubt it is safer to provide a -devel package.

To summarize, great care must be taken with dependency management when dealing with system shared libraries. It is complicated, tedious and certainly not glamorous but it IS necessary to avoid the repo descending into bloat and chaos.

Packagers Tips and Hints

Notes

  • SRPMs : http://ftp.nluug.nl/pub/os/Linux/distr/pclinuxos/pclinuxos/srpms/SRPMS.pclos/
  • The importance of being organised can never be over emphasized. Note down everything. It may help with the work you are currently engaged in or in another work later on.
  • Do not package as root.
  • Always keep a clean and fully updated packaging environment. This means run a full update before beginning a task and remove any dependencies / libraries installed for a particular work before going on to another.
  • When you run rpm -ivh nameofthepackage.srpm you may get a complaint about user Neal does not exist or something like that. It is because the original packager is not you. Do not worry.
  • Enabling the Components section of Synaptic will enable you to more easily locate the relevant section under which the required .srpm is present in the repository.
  • If you are running from the commandline and want to capture all the messages of the build in a text file for easy reference, run this command :

daniel - http://www.pclinuxos.com/forum/index.php/topic,109252.msg933761.html#msg933761 

rpmbuild -ba pclos-nameofpackage.spec > ~/Desktop/nameofpackage.spec.log 2>&1

Errors and Warnings

It would have to be the rarest of rare packages that builds without a single Error or Warning. Here is a list of some of them and wherever known, how to deal with them.

  • file not found by glob
  • make: *** No rule to make target `install'. Stop.

This is seen in case of packages where a separate subdirectory is created for config/build files. Commonly it is called build but the exact name can be made out by reading the build log. This can be fixed by adding -C nameofsubdirectory to the make install macro in %install section of .spec file. TerryN - forum.

Reading references

Retrieved from "http://pclinuxoshelp.com/index.php?title=Packaging_for_PCLinuxOS&oldid=1368"