PBUILDER-O-MATIC


Using your Debian computer to automate the process of building packages for multiples distributions (stable, unstable, testing) in a chrooted environment is a relatively easy task with pbuilder. This memo will help preparing and using automated scripts to achieve compilation in the same architecture of your installed Debian distribution. It was written in september 2010, using a Debian Lenny 5.0.6 stable.

Install packages


You'll just need to install pbuilder, it will download 83 more packages (build-essential, debhelper, lintian, ...):
$ aptitude update
# aptitude install pbuilder



Environment


Path

To ease access on your scripts, you can put them in your $HOME/bin directory. This way you can call them directly:
$ mkdir ~/bin

Add this folder to your $PATH variable if it's not already declared at the end of your .profile :
$ pico ~/.profile

# set PATH so it includes user's private bin if it exists
if [ -d "$HOME/bin" ] ; then
    PATH="$HOME/bin:$PATH"
fi

If you changed your .profile file or if the $HOME/bin directory was not created, you'll have to logout and login again to reload your environment variables.

Sudoers

The script will use the sudo command to have access to such directories like /var/cache/pbuilder, so please verify that you are declared at the end of the sudoer's file:
# visudo

your_login    ALL=(ALL)  NOPASSWD:  ALL



Script


Save the following script in your fresh $HOME/bin directory, as pbuilder-dist-arch.sh :
$ pico ~/bin/pbuilder-dist-arch.sh

#!/bin/bash                                                  
# Copyright (C) Jamin W. Collins <jcollins@asgardsrealm.net>
#                and Jordan Mantha <mantha@ubuntu.com>      
# Copyright 2007 (C) Siegfried-A. Gevatter <siggi.gevatter@gmail.com>
# Copyright 2008 (C) ludomatic <ludomatic@gmail.com>                
# License: GPLv2 or later                                            
#                                                                    
# This script is a wrapper to use pbuilder with many different      
# distributions / versions. (It was originally created because of    
# bug #255165 in Debian.)                                            
#                                                                    
# If you want to use this copy of the script only for a single distribution
# / version, rename it to 'pbuilder-dapper', 'pbuilder-feisty', 'pbuilder-gutsy',
# or whatever it is. If you have an amd64, you can also use names like          
# 'pbuilder-feisty-i386', etc.                                                  
# You could also create symlinks on this file :                                  
#   ~/bin/@pbuilder-lenny-i386 -> pbuilder-dist-arch                            
#   ~/bin/@pbuilder-squeeze-i386 -> pbuilder-dist-arch                            
#   ...                                                                          

# Base directory where pbuilder will put all the files it creates
# Must be set, usually "/var/cache/pbuilder"                    
BASE_DIR="/var/cache/pbuilder"                                  

# Enable additional repositories by default? (universe and multiverse in Ubuntu,
# contrib and non-free in Debian.)                                              
# Set to (0|1)                                                                  
EXTRACOMP=0                                                                    

# Save the log of the last operation in a dot-file? ('.lastlog' in BASE_DIR)
# Set to (0|1)                                                              
SAVELOG=1                                                                  

# Allow this script to use /var/cache/apt/archives/ when possible
# Set to (0|1)                                                  
SYSCACHE=0                                                      

# Use a proxy ?
# Set to (""|"http://hostname_or_ip:port")
USEPROXY=""                        


################################

ARCH=`dpkg-architecture -qDEB_HOST_ARCH`
SYSDIST=`lsb_release -cs 2>/dev/null`  

help()
{    
  echo "Insufficient number of arguments!"
  echo "Usage: $0 "$( [ "$1" != 'show-dist-flag' ] || echo "<distribution> " )$( [ $ARCH != "amd64" ] || echo "[i386|amd64] " )"[mainonly|allcomp] [withlog|nolog] <operation>"                                                                          
  echo "       with <operation> in create, update, build, clean, login or execute."                                          
  exit 1                                                                                                                    
}                                                                                                                            

if [ ! -z `echo \`basename $0\` | grep -- '-'`  ] && [ `basename $0` != 'pbuilder-dist' ]
then                                                                                    
  if [ $# -lt 1 ]                                                                        
  then                                                                                  
    help                                                                                
  fi                                                                                    

  BINARCH=`basename $0 | cut -f3 -d '-'`
  DISTRIBUTION=`basename $0 | cut -f2 -d '-'`
else                                        
  if [ $# -lt 2 ]                            
  then                                      
    help show-dist-flag                      
  fi                                        

  DISTRIBUTION=$1
  shift 1        
fi              


if [ $1 = "i386" ]  || [ $1 = "amd64" ]
then                                  
  if [ $ARCH = "amd64" ]; then        
    BINARCH=$1                        
  else                                
    echo "Warning: Architecture switching is not supported on your system; ignoring '$1', using '$ARCH'."
  fi                                                                                                    

  shift 1
fi      


if [ $1 = "mainonly" ]; then
  EXTRACOMP=0              
  shift 1                  
elif [ $1 = "allcomp" ]; then
  EXTRACOMP=1                
  shift 1                    
fi                          


if [ $1 = "withlog" ]; then
  SAVELOG=1                
  shift 1                  
elif [ $1 = "nolog" ]; then
  SAVELOG=0                
  shift 1                  
fi                        


distdata()
{        
  if [ "$1" = "debian" ]
  then                  
    # Set Debian specific data
    ARCHIVE="http://ftp.debian.org/debian"
    COMPONENTS="main"$( [ $EXTRACOMP = 0 ] || echo " contrib non-free" )
  else                                                                  
    # Set Ubuntu specific data                                          
    ARCHIVE="http://archive.ubuntu.com/ubuntu"                      
    COMPONENTS="main restricted"$( [ $EXTRACOMP = 0 ] || echo " universe multiverse" )
  fi                                                                                  
}                                                                                    


case $DISTRIBUTION in
  #warty|hoary|breezy)
  dapper|edgy|feisty|gutsy|hardy|intrepid|jaunty)
    distdata ubuntu                              
  ;;                                            

  oldstable|stable|testing|unstable|experimental|etch|lenny|squeeze|sid)
    distdata debian                                                          
  ;;                                                                          

  *)
    echo "Warning: Unknown distribution '$DISTRIBUTION'."
    echo -n "Continue [y/N]? "                          
    read continue                                        
    if [ "$continue" != 'y' ] && [ "$continue" != 'Y' ]; then
      echo "Aborting..."                                    
      exit 1                                                
    fi                                                      

    distdata
  ;;        
esac        

FOLDERBASE="${DISTRIBUTION}-$( ([ "$BINARCH" != "" ] && echo $BINARCH) || echo $ARCH )"

OPERATION=$1

case $OPERATION in
  update|build|clean|login|execute)
    shift 1                        
  ;;                              

  create)
    if [ -s $BASE_DIR/$FOLDERBASE-base.tgz ]
    then                                    
      echo "Warning: file '$BASE_DIR/$FOLDERBASE-base.tgz' already created."
      echo "You will replace an existing $DISTRIBUTION installation of PBuilder!"
      echo -n "Continue [y/N]? "                                                
      read continue                                                              
      if [ "$continue" != 'y' ] && [ "$continue" != 'Y' ]; then                  
        echo "Aborting..."                                                      
        exit 1                                                                  
      fi                                                                        
    else                                                                        
      shift 1                                                                    
    fi                                                                          
  ;;                                                                            

  *)
    echo "Unrecognized argument. Please use one of those:"
    echo "    create"                                    
    echo "    update"                                    
    echo "    build"                                      
    echo "    clean"                                      
    echo "    login"                                      
    echo "    execute"                                    
    exit 1                                                
  ;;                                                      
esac                                                      

if [ ! -d $BASE_DIR/${FOLDERBASE}/result ]; then
  sudo mkdir -p $BASE_DIR/${FOLDERBASE}/result  
fi                                              

if [ $SYSCACHE != 0 ] && [ "$SYSDIST" = "$DISTRIBUTION" ] && [ "$ARCH" = "$BINARCH" -o -z $BINARCH ]
then
  DEBCACHE='/var/cache/apt/archives/'
fi

#EXTRAPACKAGES="pentium-builder gcc-4.2 g++-4.2"

sudo pbuilder $OPERATION \
    --basetgz $BASE_DIR/$FOLDERBASE-base.tgz \
    --distribution $DISTRIBUTION \
    $( [ -z $DEBCACHE ] || echo "--aptcache $DEBCACHE" ) \
    $( [ $SAVELOG = 0 ] || echo "--logfile $BASE_DIR/.lastlog" ) \
    $( [ -z $USEPROXY ] || echo "--http-proxy $USEPROXY" ) \
    --buildresult $BASE_DIR/${FOLDERBASE}/result \
    --othermirror "deb $ARCHIVE $DISTRIBUTION $COMPONENTS" \
    --override-config \
    $@


Before you start playing with this script, just update ARCHIVE variable to your default repository address (line 112). Thanks to Nautile, I can use an official local mirror in New-Caledonia:
ARCHIVE="http://ftp.nc.debian.org/debian"


Make your script executable:
$ chmod +x ~/bin/pbuilder-dist-arch.sh


Now declare as many distributions as you like by creating symlinks directly in the ~/bin folder. Let's declare a Lenny and Squeeze:
cd ~/bin
$ ln -s  pbuilder-dist-arch.sh pbuilder-lenny-i386
$ ln -s  pbuilder-dist-arch.sh pbuilder-squeeze-i386


Note on the architectures : I can only use the i386 architecture as my processor is a i386. If you are on a amd64 architecture, you can create amd64 and i386 chrooted environments!


Play with your dists


To first initialize the Squeeze distribution, simply use create option:
$ pbuilder-squeeze-i386 create
ludo@ludo-vbox:~$ pbuilder-squeeze-i386 create
W: /home/ludo/.pbuilderrc does not exist
  -> Logging to /var/cache/pbuilder/.lastlog
Distribution is squeeze.
Building the build environment
 -> running cdebootstrap
/usr/bin/cdebootstrap
P: Retrieving Release
P: Retrieving Release.gpg
P: Validating Release
I: Good signature from "Debian Archive Automatic Signing Key (5.0/lenny) <ftpmaster@debian.org>"
P: Parsing Release
P: Retrieving Packages.gz

[...grab some RSS news while pbuilder does it's job...]

update-alternatives: using /usr/bin/aptitude-curses to provide /usr/bin/aptitude (aptitude) in auto mode.
Copying back the cached apt archive contents
 -> new cache content libxapian22_1.2.3-2_i386.deb added
 -> new cache content libept1_1.0.3+b1_i386.deb added
 -> new cache content libsigc++-2.0-0c2a_2.2.4.2-1_i386.deb added
 -> new cache content libcwidget3_0.5.16-3_i386.deb added
 -> new cache content aptitude_0.6.3-3.1_i386.deb added
 -> new cache content libsqlite3-0_3.7.2-1_i386.deb added
 -> new cache content libncursesw5_5.7+20100313-3_i386.deb added
 -> new cache content libboost-iostreams1.42.0_1.42.0-4_i386.deb added
 -> unmounting dev/pts filesystem
 -> unmounting proc filesystem
 -> creating base tarball [/var/cache/pbuilder/squeeze-i386-base.tgz]
 -> cleaning the build env
    -> removing directory /var/cache/pbuilder/build//8564 and its subdirectories


Ok, now log into the new distribution with login option:
$ pbuilder-squeeze-i386 login
W: /home/ludo/.pbuilderrc does not exist
  -> Logging to /var/cache/pbuilder/.lastlog
Building the build Environment
 -> extracting base tarball [/var/cache/pbuilder/squeeze-i386-base.tgz]
 -> creating local configuration
 -> copying local configuration
  -> Installing apt-lines
 -> mounting /proc filesystem
 -> mounting /dev/pts filesystem
 -> policy-rc.d already exists
Obtaining the cached apt archive contents
 -> entering the shell
File extracted to: /var/cache/pbuilder/build//10794

cat /etc/debian_version
squeeze/sid

exit
Copying back the cached apt archive contents
 -> unmounting dev/pts filesystem
 -> unmounting proc filesystem
 -> cleaning the build env
    -> removing directory /var/cache/pbuilder/build//10794 and its subdirectories


Everything you will do in this chrooted environment won't be saved to keep a system as sane as possible, without packages that wouldn't be present on a fresh install. You will then be sure that if your package builds correctly with pbuilder, it will do the same on "real" environments.


Everyday usage


Now that everything is set, here is the method I use to build new versions of packages:
$ pbuilder-squeeze-i386 update                            
W: /home/ludo/.pbuilderrc does not exist                                  
  -> Logging to /var/cache/pbuilder/.lastlog                              
Upgrading for distribution squeeze                                        
Building the build Environment                                            
 -> extracting base tarball [/var/cache/pbuilder/squeeze-i386-base.tgz]
 -> creating local configuration
 -> copying local configuration
  -> Installing apt-lines
 -> mounting /proc filesystem
 -> mounting /dev/pts filesystem
 -> policy-rc.d already exists
Refreshing the base.tgz
 -> upgrading packages
Get:1 http://ftp.nc.debian.org squeeze Release.gpg [835B]
Ign http://ftp.nc.debian.org/debian/ squeeze/main Translation-en
Get:2 http://ftp.nc.debian.org squeeze Release [89.9kB]
Get:3 http://ftp.nc.debian.org squeeze/main i386 Packages/DiffIndex [2038B]
Hit http://ftp.debian.org squeeze Release.gpg
Ign http://ftp.debian.org/debian/ squeeze/main Translation-en
Hit http://ftp.debian.org squeeze Release
Get:4 http://ftp.debian.org squeeze/main i386 Packages/DiffIndex [2038B]
Fetched 94.8kB in 3s (29.4kB/s)
Reading package lists...
dpkg: warning: ignoring request to remove lilo which isn't installed.
Obtaining the cached apt archive contents
Reading package lists...
Building dependency tree...
Reading state information...
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Reading package lists...
Building dependency tree...
Reading state information...
apt is already the newest version.
aptitude is already the newest version.
build-essential is already the newest version.
dpkg-dev is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Copying back the cached apt archive contents
 -> unmounting dev/pts filesystem
 -> unmounting proc filesystem
 -> creating base tarball [/var/cache/pbuilder/squeeze-i386-base.tgz]
 -> cleaning the build env
    -> removing directory /var/cache/pbuilder/build//10951 and its subdirectories

$ cd packagexx_version
$ debuild -S -sa -i.git -I.git

$ cd ..
$ pbuilder-squeeze-i386 build packagexx_version.dsc

$ cd /var/cache/pbuilder/squueze-i386/result/
$ lintian -iv packagexx_version_i386.changes
$ debsign -S packagexx_version_i386.changes


Debug


Pbuilder reads your configuration in the .pbuilderrc file. Here we can specify a hook script that will allow you to log into the chrooted environment if a package build fails.
$ pico ~/.pbuilderrc

# https://wiki.ubuntu.com/PbuilderHowto
HOOKDIR="/var/cache/pbuilder/hook.d"


Then create the script /var/cache/pbuilder/hook.d/C10shell
# pico /var/cache/pbuilder/hook.d/C10shell

#!/bin/sh
# invoke shell if build fails.
apt-get install -y --force-yes vim less bash mc
cd /tmp/buildd/*/debian/..
/bin/bash < /dev/tty > /dev/tty 2> /dev/tty

Note that pbuilder can install the packages you like just before you login (line 3), just add the one you want.

Valid XHTML :: Valid CSS: :: Powered by WikkaWiki