The original idea behind src2pkg was as a sort of generic build script written in the BASH shell language. The generic script could be used as a template for writing scripts to make Slackware-type packages of software from compiled sources. A script is simply a list of instructions to be carried out in a certain order. Having a script of the actions needed to unpack, configure, compile and package each program or library makes it easy to repeat the build process anytime, without having to remember all the exact steps or having to wotk them all out again.

If you ask how to compile and install software on your system, you'll usually get a pretty standard answer like: “unpack the sources somewhere convenient where you have write priviledges, then cd into the sources and run, configure, make, make install. These are the standard trio of commands used to configure sources for your system, compile them and install them -assuming they are 'autoconf' sources -that is, that they use the autoconf and automake programs and their configuration files. The majority of software for Linux and other POSIX systems is made available using the 'autoconf' system.

Of course, the detailed commands actually run into more than three or four lines of code -especially when we consider creating a package of the software. An extremely basic script for creating a simple package with just a few files can easily stretch to 30 or 40 lines. A script to package some large, complex programs or libraries may be much longer. A pretty universal script of 75-125 lines can be written which will cover many thousands of programs with only a few lines needing to be changed for each individual script.

Two things are quickly apparent when using such a script template. First, most of the code is exactly the same in every script. By using shell 'variables' to represent the filenames, versions and paths in the code, most scripts only need to have a few variables chaneged at the top of each script. But, secondly, when minor changes are needed later in the script it is very hard to see the changes among so many line of code.

Designing and using script templates quickly showed the need to analyze the whole process of compiling and packaging software and divide the process into a number of separate steps. Each step is performed by a block of code which works together to accomplish one step of the process. But, the order in which steps are taken is also important. So, each block of code was written as a shell 'function'. Functions are like tiny programs since they can contain many lines of code which can be given a name. By calling just the function name all the lines of code in the function are executed.

Using functions provides an ideal way to modularize the code for src2pkg so that it can be more easily re-arranged, added to or left out when unneeded. It also provides a way to create very short scripts which make it easier to see any changes that are made to the script for packaging a particular program. This is achieved by dividing the code into two parts. The first part contains what is specific to a particular package. The second part contains all the code which is the same for (nearly) every package. This redundnat code is kept in a separate file. src2pkg uses the BASH 'source' command to read in this code and run it as if it were written in the same file. Each src2pkg build script uses a single 'source' command to read in all the functions which contain most of the code for src2pkg.

Since src2pkg was originally written to always use a small build script, it is helpful to illustrate the flow of events for a typical build with a small example script. The src2pkg 'program' which lets you build packages using a command-line interface operates in a similar way.

A typical src2pkg build script can be divided into three sections. First, variables are declared, the the functions are read in, and then the functions are performed in a certain order. src2pkg divides the whole process of configuring, compiling and packaging software into a list if 16 steps. The function names, and their purpose, are based on a line-by-line description of the typical build process. The list includes 'pre_process' and 'post_process' functions which provide a 'catch-all' place for code which might not fit logically elsewhere in the list.

The three sections look like this (using an example source archive named foo-0.1.tar.bz2:

I. The first section sets the package variables. This just means giving the name of the source archive like this:

SOURCE_NAME=foo-0.1.tar.bz2

II. The second 'section simply 'sources' the src2pkg functions.

source /usr/libexec/src2pkg/FUNCTIONS

Actually, the '.' (period) command is used. It does the same as the 'source' command. This reads in all the code for the src2pkg functions after reading the src2pkg configuration file located in /etc/src2pkg/src2pkg.conf.

II. The third section actually performs the code in the functions by calling each one by its' name:

pre_process

find_source

make_dirs

unpack_source

fix_source_perms

configure_source

compile_source

fake_install

fix_pkg_perms

strip_bins

create_docs

compress_man_pages

make_description

make_doinst

make_package

post_process

These are the main 16 functions of src2pkg, although each of them calls other smaller sub-functions. It is not usually necessary to know about what the sub-functions are called or what they do. There is also a “super-function” called 'do_all_processes' which simply runs each of the functions in the above list, in order. So, a complete *.src2pkg build script can actually consist of just three lines, like this:

SOURCE_NAME=foo-0.1.tar.bz2 . /usr/libexec/src2pkg/FUNCTIONS do_all_processes

Of course that assumes that no extra commands or options are needed in order to configure, compile and package our example 'foo' program. But this makes it easy to imagine the three steps of setting variable values, reading in the code for the functions, and then carrying out each of the steps.

But, a detailed look at how the code is executed reveals that, in reality, most of the variable values aren't finalized until the first function, 'pre_process', is run and some still later. src2pkg actually uses many variable values in various ways. The SOURCE_NAME variable is obviously related to the name of the archive for the program we are trying compile and package and is the only one which must be supplied. Only variables which apply to the particular package must be in the src2pkg build script for the package.

src2pkg also uses variables as 'switches' to enable or disable certain features or options which control what sr2pkg does or does not do. Some of these are used to fine-tune what src2pkg does with a particular package. In these cases, the variable should be included in the first part of a src2pkg build script. Other variables control your general preferences when running src2pkg -like whether to use color prompts, or to adjust the level of verbosity in the prompts or program feedback from src2pkg. Still, others are used to configure src2pkg for your specific OS version or let you change the deafult settings for standard commands or the working directories. These are all automatically set to sane defaults, but can easily be changed when needed.

Src2pkg used to only use what is now called REAL mode. This means that the files get installed right to your system -the same way that checkinstall works. This has the advantage of being the most reliable way of tracking the files and directories which get created. It also is the most reliable way of creating package content because it does exactly what 'make install' -no more and no less. It has the disadvantage that you must be running src2pkg as root, and that the process may overwrite files which are already on your system. This last disadvantage can be overcome by using the SAFE method. The SAFE method is just like REAL, except that it invokes the backup facility of libsentry which creates backups of any files, links or directories before they are overwritten, and then restores the originals before packaging is finished. Both of these methods were possible with early versions of src2pkg.

Since both the above methods are 100% invasive, I wrote in support for DESTDIR. Most people are familiar with using DESTDIR, it simply means that, for instance, when a file is supposed to be install to /usr/bin, by setting DESTDIR, you can temporarily install the file into an alternate directory $DESTDIR/usr/bin. This is the way *most* SlackBuilds work. The JAIL method does something similar, but instead of relying on the Makefiles containing support for DESTDIR, it uses an internal function of libsentry to create the temporary path. This sounds good because it allows you build packages while running as non-root user and keeps you from overwriting files on your system. Most people trust DESTDIR. The very first versions of src2pkg used only DESTDIR. But, when I started creating support for installwatch/libsentry, I ran a lot of tests and saw that DESTDIR doesn't always work completely. And of course, for many sources it will not work at all -the python module in this thread is a good example. But the worst is that many sources only *partial* support DESTDIR. That means that when you run 'make DESTDIR=$PKG_DIR install' some of the files do get installed under $DESTDIR, but that others actually get installed to the real root filesystem *and you don't know it*. This is the worst case of all, because you think you are doing a safe and complete installation and creation of content in an isolated directory, when in fact files have gone to your root fs and are not included in the package.

The JAIL method would appear to provide the perfect solution to all the above problems, but unfortunately it doesn't always work as expected. And for a long time it suffered from some really weird problems -the same problems which checkinstall had. That is that building some packages would do really weird unexpected things, like including the gcc compiler or other system files in the package! I finally forked installwatch to libsentry and fixed these and other problems. But the JAIL method is still not entirely dependable. Sometimes it still fails. This usually happens when the installation uses 'if' statements to ask questions about your real filesystem. For instance, say you are packaging a service or daemon which is run during startup by the init scripts. The Makefile might have an if statement which checks to see what kind of system you have by looking to see if the file /etc/redhat-release or /etc/slackware-version exists, in order to decide which init scripts to install. Since the JAIL method prepends the JAIL prefix to *every* path, it will be looking for these files in the $PKG_DIR instead of on the real filesystem -this causes the installation to fail because none of those files are found in the $PKG_DIR. The upshot is that *no* single method will always work except the REAL or SAFE method. But many, many people like being able to build packages as a normal user. Most people *wrongly* trust DESTIR. For the last couple of years, src2pkg has quietly used DESTDIR as the default method since people trust it and it is usually a bit more dependable than JAIL. But, src2pkg now includes code which *checks* all DESTDIR installations and warns you (rather loudly with yellow text), when the DESTDIR installation is broken and suggest that you re-build the package using the JAIL, REAL or SAFE methods.

Since long ago, Tim advocated making the JAIL method the default, and for awhile it was. For awhile I switched the default back and forth from JAIL to DESTDIR to see which was more dependable. The fact is that they both partially fail about the same percentage of the times. I finally settled on using DESTDIR simply because most people trust it, are used to using it and understand it. The code that I have now included, which checks for broken DESTDIR support, is meant to dispel this implicit trust of DESTIR. And it lists all files which were installed outside of the DESTDIR, so it provides concrete information which helps you to restore your system to its' original state. src2pkg automatically switches the default from DESTDIR to JAIL method for installations where it knows that DESTDIR will not work.

So, reviewing all the above, the conclusion is that the very best way to get complete packages, without any orphaned files or having your system files overwritten, is to use the SAFE method. If you are packaging programs which are not already installed on your system, the SAFE method is just as good. But, if you like the secure feeling which creating packages as a normal user gives, then the JAIL method is probably the best default to use. When the occasional package fails using the JAIL method, you can always try it using the DESTDIR method.

But, it should be pointed out that not all packages can be created as a normal user. Packages like those mentioned above, which install files or dirs which do not belong to root:root can not be correctly built as a normal user, because a normal user can not use chown to change files or directories to ownership/group which the user does not belong to. The method which src2pkg uses to allow packaging by normal users involves using the 'tar' command to change all ownership/groups to root:root as the package content is being compressed. While this is correct for *nearly* all cases, it is not *always* correct. Still, I found this method to be much less intrusive and dangerous than having it use 'fakeroot' or other such tools to allow users to create packages. Most users will have no trouble creating all their packages as a normal user. src2pkg does now alert you when it sees that 'chown' or 'install' will be used to change ownership or group to root.