Abstract
This paper describes best practices in Conary packaging: writing recipes that take advantage of Conary features; avoiding redundancy with recipe inheritance and design; implementing release management using branches, shadows, labels, redirects, and flavors; and designing and writing dynamic tag handlers. It describes how Conary policy prevents common packaging errors. It provides examples from our rPath Linux distribution, illustrating the design principles of the Conary build process. It then describes the steps needed to create a new distribution based on the rPath Linux distribution, using the distributed branch and shadow features of Conary.
Conary is a distributed software management system for Linux distributions. Based on extensive experience developing Linux distributions and package management tools, it replaces traditional package management solutions (such as RPM and dpkg) with one designed to enable loose collaboration across the Internet. It enables sets of distributed and loosely connected repositories to define the components which are installed on a Linux system. Rather than having a full distribution come from a single vendor, it allows administrators and developers to branch a distribution, keeping the pieces which fit their environment while grabbing components from other repositories across the Internet.
If you do not have a basic working knowledge of Conary terminology and design, read “ Repository-Based System Management Using Conary”, which introduces Conary's design and vocabulary in greater detail. Terms called out in boldface in this paper without an explicit definition are defined in that overview.
The original version of this document was presented at the 2005 Linux Symposium in Ottawa, Canada.
Unlike legacy package management tools, Conary has integral management for source code and binaries, and the binaries are directly associated with the source code from which they have been built.
Conary stores source files in source components, and then uses a recipe (described later) to build binary components that it can install on a system. While most of the code that handles the two kinds of components is actually the same, the interface is different. The source components are managed using a Software Configuration Management (SCM) model, and the binary components are managed using a system management model.
The SCM model for managing source components is sufficiently familiar to experienced Concurrent Versioning System (cvs) or Subversion (svn) users; you can create a new source component, check out an existing source component, add files to a source component, remove files from a source component (this is a single action in Conary, unlike cvs's two-step operation), rename files in a source component (like svn, but unlike cvs), and commit the current set of changes since the last commit (like svn, and like cvs except that the commit is atomic).
Conary has not been optimized as a complete SCM system. For example, we do not use it to manage subdirectories within a source component, and instead of importing source code into vendor branches, we import archives and apply patches to them. Conary may someday support a richer SCM model, since there are no significant structural or design barriers within Conary. It was a choice made for the sake of simplicity and convenience, and to focus attention on Conary as a system management tool rather than as an SCM tool—to encourage using Conary to track upstream development, rather than encourage using it to create forks.
In addition, Conary stores in the repository (without cluttering the directory in which the other files are stored) all files that are referenced by URL. When they are needed, they are downloaded (from the repository if they have been added to the repository; otherwise, via the URL) and stored in a separate directory. Then, when committing, Conary stores those automatically added source files (auto-source files) in the repository so that the exact same source is always available, enabling repeatable builds.
Like cvs, Conary can create branches to support divergent development (forks); unlike cvs, those branches can span repositories, and the repository being branched from is not modified, so the user only needs write privileges in the repository in which the branch is created. Unlike cvs (and any other SCM we are aware of), Conary also has two features that support converging source code bases: shadows that act like branches but support convergent instead of divergent development by providing intentional tracking semantics, and redirects that allow redirecting from the current head of the branch (or shadow) to any other branch (or shadow), including but not limited to that branch's parent. The redirect is not necessarily permanent; the branch can be revived. A redirect can even point to a different name package entirely, and so is useful when upstream names change, or when obsoleting one package in favor of another.
Using a recipe to turn source into binary is called cooking.
The exact output produced by building source code into a binary is defined by several factors, among them the instruction set (or sets) that the compiler emits, and the set of features selected to be built. Conary encodes each combination of configuration and instruction set as a flavor. The configuration items can be system-wide or package-local. When cooking, Conary builds a changeset file that represents the entire contents of the cooked package.
There are three ways to cook:
A local cook builds a changeset on the special local@local:COOK branch. It loads the recipe from the local filesystem, and can cook with recipes and sources that are not checked into the repository. It will download any automatic sources required to build.
A repository cook builds a transient changeset on the same branch as the source component, and then commits it to the repository. It loads the source component (including the recipe and all sources) from the repository, not the local filesystem. It finds all automatic sources in the repository. The same version can be built into the repository multiple times with different flavors, allowing users to receive the build that best matches their system flavor when they request a trove.
An emerge builds a transient changeset on the special local@local:EMERGE branch, and then commits it to (that is, installs it on) the local system. Like a repository cook, it takes the recipe and all sources from the repository, not the filesystem. (This is the only kind of cook that Conary allows to be done as the root user.)