The i3 buildbot setup

Michael Stapelberg
<michael@i3wm.org>
September 2012
Table of Contents
WARNING!

This document is outdated and kept for “historical” reasons. i3 uses Travis as a CI instead.

This document explains the buildbot setup we use to provide up-to-date documentation and debian packages at http://build.i3wm.org/. We publish these information so that our setup is well-documented (thus decreasing future maintenance effort) and because it might be interesting for other projects.

1. Introduction

What we are doing in i3 is called Continuous Integration (see http://en.wikipedia.org/wiki/Continuous_integration): we publish the changes we make on our local machines as often as possible. In order to maintain a continuously high quality, each time any developer pushes changes to the official git repository, a number of quality assurance tools start running automatically:

  1. Latest documentation is generated and provided at http://build.i3wm.org/docs/. This makes it easy to link to documentation for features which are only in the current git version, not in the released version.

  2. The source code is compiled and it is automatically posted to the IRC channel whether there were any compiler warnings. While developers should notice compiler warnings, this mechanism creates a bit of public pressure ("Oh, Michael introduced warnings with this commit!"). More importantly, since this mechanism builds a dist tarball and then compiles that tarball, any changes to the source which would result in an uncompilable dist tarball are immediately obvious. Therefore, we could cut a release from the current git version at any point in time.

  3. The clang static analyzer runs and the resulting report is provided at http://build.i3wm.org/clang-analyze/. While every developer needs to compile his code before committing, he doesn’t necessarily use clang (so we catch build failures when using clang) and he also probably doesn’t run a static analyzer as part of his normal workflow. By just being available without any friction, this mechanism encourages developers to look at the report and fix problems.

  4. Debian (and Ubuntu) packages are built. This not only ensures that we don’t change anything in the source code which would lead to an FTBFS (Fails To Build From Source) when building a Debian package, it also goes a long way to encourage end users to test i3. To remove the need and resource requirements for them to compile their own version of i3 regularly, we provide packages that integrate conveniently with a normal Debian system (e.g. that are automatically upgraded).

2. Why buildbot?

Previously, I was unsatisfied with the current state of FOSS CI tools like Jenkins, Tinderbox and others. They either seemed bloated, hard to use, outdated or unappealing for some other reason.

Then I discovered buildbot and was impressed by its flexibility. It let me implement everything I wanted from a CI tool and (in my opinion) it is light-weight, easy to deploy and well maintained.

The only downside of buildbot is its configuration and documentation: You need to spend quite a bit of time (I needed multiple days) until it works the way you want it to and oftentimes, the documentation is far too sparse. This is one of the reasons why I’m publishing the i3 setup.

3. Configuration

See the next section for a complete, copy & pasteable configuration file. This section covers the most important aspects without covering every line.

This document assumes you are running buildbot 0.8.6p1.

3.1. Change sources

Since i3 uses a central git repository, we use the official buildbot git post-receive hook that sends the change information to the buildbot master.

3.2. Schedulers

There are two things (called "builders" in buildbot-language) which happen whenever a new change in the next branch of i3 occurs:

  1. The "docs" builder builds and uploads the latest documentation. This happens directly from the git repository with a custom asciidoc configuration which indicates that these docs refer to the git version. Therefore, this builder does not benefit from having a dist tarball available (contrary to the other builders).

  2. The "dist" builder prepares a dist tarball and then triggers the remaining builders. This ensures that building the dist tarball (an operation which takes about one minute due to documentation generation) only happens once.

Here is the relevant configuration part:

Schedulers:

c['schedulers'] = []

c['schedulers'].append(SingleBranchScheduler(
    name = 'dist',
    branch = 'next',
    treeStableTimer = 10,
    builderNames = [ 'dist', 'docs' ],
))

c['schedulers'].append(Triggerable(
    name = 'dist-tarball-done',
    builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
))

3.3. Building the dist tarball

This builder clones the i3 git repository and runs "make dist", which creates a tarball that could be named "i3-4.2.tar.bz2" for example. This tarball is then renamed to dist-%(gitversion).tar.bz2 (so that we can work with a predictable name in the next steps) and uploaded to the buildbot master (since we can have multiple buildslaves, we cannot just let it rest on the buildslave that built it). Afterwards, old dist tarballs are cleaned up and the remaining builders are triggered:

Building a dist tarball:

factories = {}

f = factories['dist'] = BuildFactory()

# Check out the git repository.
f.addStep(s_git)

# Fill the 'gitversion' property with the output of git describe --tags.
f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))

# Build the dist tarball.
cmd(f, name = 'make dist', command = [ 'make', 'dist' ])

# Rename the created tarball to a well-known name.
cmd(f,
    name = 'rename tarball',
    command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'),
)

# Upload the dist tarball to the master (other factories download it later).
f.addStep(transfer.FileUpload(
    slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
    masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
))

# Cleanup old dist tarballs (everything older than tree days).
f.addStep(master.MasterShellCommand(
    command = "find distballs -mtime +3 -exec rm '{}' \;",
    name = 'cleanup old dist tarballs',
))

# Everything worked fine, now trigger compilation.
f.addStep(Trigger(
    schedulerNames = [ 'dist-tarball-done' ],
    copy_properties = [ 'gitversion' ],
))

Three things are noteworthy about this part of the configuration:

  1. For convenience, we call each factory f (just like the global buildbot config uses c for the top-level configuration) and add it to a dictionary. Factories in that dictionary are later automatically configured for each buildslave.

  2. We have a shared step called s_git so that we only have one location in the configuration file where we specify the git repository URL and branch.

  3. We have a custom function called cmd which is a shortcut for defining a ShellCommand with haltOnFailure=True (since each step is critical) and logEnviron=False (for brevity).

Here are their definitions:

cmd:

def cmd(factory, **kwargs):
    factory.addStep(ShellCommand(
        haltOnFailure = True,
        logEnviron = False,
        **kwargs
    ))

s_git:

s_git = Git(
    repourl = 'git://code.i3wm.org/i3',
    branch = 'next',

    # Check out the latest revision, not the one which caused this build.
    alwaysUseLatest = True,

    # We cannot use shallow because it breaks git describe --tags.
    shallow = False,

    # Delete remnants of previous builds.
    mode = 'full',

    # Store checkouts in source/ and copy them over to build/ to save
    # bandwidth.
    method = 'copy',
)

3.4. Compiling the dist tarball

For this builder to work, you obviously need to install all the build-dependencies for your software on each buildslave. In the case of i3, this can be done with apt-get build-dep i3-wm.

The compilation is pretty straight-forward since it uses the builtin Compile step. We call make with -j4 (we don’t have enough buildslaves to make figuring out the amount of cores at build-time worthwhile) and DEBUG=0 to simulate release build conditions. Also, we pass the preprocessor flag -D_FORTIFY_SOURCE=2 and the compiler flags -Wformat and -Wformat-security to enable additional warnings.

Compiling the dist tarball:

f = factories['compile'] = BuildFactory()
unpack_dist_tarball(f)
f.addStep(Compile(
    command = [ 'make', 'DEBUG=0', '-j4' ],
    warningPattern = '.*warning: ',
    warnOnWarnings = True,
    workdir = 'build/DIST',
    env = {
      'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
      'CFLAGS': '-Wformat -Wformat-security'
    },
))

f.addStep(WarningsToIRC())

Again, we use custom functions (and a custom buildstep) to make our lives easier. Here is the definition of unpack_dist_tarball which adds three steps to the factory that download and unpack the dist tarball to the DIST/ directory:

unpack_dist_tarball:

def unpack_dist_tarball(factory):
    factory.addStep(transfer.FileDownload(
        mastersrc = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
        slavedest = 'dist.tar.bz2',
    ))

    factory.addStep(slave.MakeDirectory(dir = 'build/DIST'))

    cmd(factory,
        name = 'unpack dist tarball',
        command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
    )

The WarningsToIRC build step is a custom build step which sets a property called "ircsuffix" that is used by our custom IRC bot. This is covered later in more detail. This property gets set to a green or red message, depending on whether there were any warnings:

WarningsToIRC:

class WarningsToIRC(buildstep.BuildStep):
    def start(self):
        warnings = self.getProperty("warnings-count")
        if warnings is not None and int(warnings) > 0:
            warnings = int(warnings)  # just to be sure
            self.setProperty("ircsuffix", ("\0037 with %d warning%s!" %
                (warnings, "s" if warnings != 1 else "")))
        else:
            self.setProperty("ircsuffix", "\0033 without warnings")
        self.finished(SUCCESS)

3.5. Static code analysis

For this builder to work, you additionally need the clang compiler on each buildslave: apt-get install clang.

This builder uses only custom functions which you already know by now. It runs scan-build, then moves scan-build’s output from a date-based directory directly into the CLANG/ directory and uploads that to the buildmaster.

On the buildmaster, a webserver is configured which has a symlink to /home/build/i3-master/htdocs/clang-analyze in its document root.

static code analysis:

f = factories['clang-analyze'] = BuildFactory()
unpack_dist_tarball(f)
cmd(f,
    name='analyze',
    command = [
        'scan-build',
        '-o', '../CLANG',
        '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
        'make', '-j8',
    ],
    workdir = 'build/DIST',
)

# remove the subdirectory -- we always want to overwrite
cmd(f, command = 'mv CLANG/*/* CLANG/')

f.addStep(transfer.DirectoryUpload(
    slavesrc = 'CLANG',
    masterdest = 'htdocs/clang-analyze',
    compress = 'bz2',
    name = 'upload output',
))

f.addStep(ClangToIRC())

The ClangToIRC custom step is even simpler than WarningsToIRC. It simply sets the ircsuffix property to a static message:

ClangToIRC:

class ClangToIRC(buildstep.BuildStep):
    def start(self):
        self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
        self.finished(SUCCESS)

3.6. Generating documentation

This builder is the one which is the least clean of all. It uses the Debian packaging information to decide which docs to publish and which manpages to generate. Additionally, it uses a for loop instead of calling a script. I recommend including a script to do this in your repository instead.

Apart from these concerns, the builder is straight-forward: It clones the git repository, generates the documentation and then uploads the documentation to the buildmaster:

Generating documentation:

f = factories['docs'] = BuildFactory()
f.addStep(s_git)
# Fill the 'gitversion' property with the output of git describe --tags.
f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")

f.addStep(transfer.DirectoryUpload(
    slavesrc = 'COPY-DOCS',
    masterdest = 'htdocs/docs-git',
    compress = 'bz2',
    name = 'upload docs'))

f.addStep(DocsToIRC())

Just as ClangToIRC, DocsToIRC appends a static message:

DocsToIRC:

class DocsToIRC(buildstep.BuildStep):
    def start(self):
        self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
        self.finished(SUCCESS)

3.7. Building Debian/Ubuntu packages

This is the most complex builder of all. It uses pbuilder-dist, debchange, dpkg-buildpackage and reprepro to generate a Debian repository with a cleanly compiled package for amd64 and i386. In order for it to work, you need to install the following packages: apt-get install devscripts dpkg-dev reprepro ubuntu-dev-tools pbuilder. Afterwards, you need to allow the user as which the buildslave runs to execute pbuilder via sudo without needing a password, so add a config file like this one:

sudoers.d:

echo 'build    ALL= NOPASSWD: SETENV: /usr/sbin/pbuilder' > /etc/sudoers.d/build

Then, as the user as which your buildslave runs, setup the pbuilder environments (you only need to do this once):

pbuilder preparation:

sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-amd64
sudo ln -s pbuilder-dist /usr/bin/pbuilder-sid-i386
pbuilder-sid-amd64 create
pbuilder-sid-i386 create

Also, you will need a GPG key to sign these packages.

The debian builder starts by unpacking the dist tarball, copying the Debian packaging from git, creating an empty Debian repository with the i3-autobuild-keyring contents in it. It then adds a new changelog entry to reflect the git version and the fact that this package was built automatically, builds a source package with dpkg-buildpackage and adds it to the repository. Afterwards, it updates each pbuilder and builds binary packages for each architecture (amd64 and i386). After adding the resulting packages to the repository, it uploads the repository to the buildmaster:

Debian builder:

distributions = [ 'sid-amd64', 'sid-i386' ]
gpg_key = 'BE1DB1F1'

f = factories['debian-packages'] = BuildFactory()
# We need the git repository for the Debian packaging.
f.addStep(s_git)
unpack_dist_tarball(f)
cmd(f, name = 'copy packaging', command = "cp -r debian DIST/")

# Add a new changelog entry to have the git version in the package version.
cmd(f,
    name = 'update changelog',
    workdir = 'build/DIST',
    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
)

cmd(f,
    name = 'source pkg',
    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
    workdir = 'build/DIST',
)

for dist in distributions:
    f.addStep(slave.MakeDirectory(dir = 'build/RESULT-' + dist))

# Create debian sid repository
f.addStep(slave.MakeDirectory(dir = 'build/REPO-sid/conf'))
f.addStep(transfer.StringDownload(
    """Codename: sid
Suite: unstable
Architectures: i386 amd64 source
Components: main
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release . .gz .bz2
SignWith: %(gpg_key)s
""" % { "gpg_key": gpg_key },
    slavedest = 'REPO-sid/conf/distributions',
))

# add source package to repository
reprepro_include(f, 'i3-wm*_source.changes', 'dsc')

# Add keyring to the repository. We need to run git clone on our own because
# the Git() step assumes there’s precisely one repository we want to deal with.
# No big deal since the i3-autobuild-keyring repository is not big.
cmd(f,
    name = 'clone keyring repo',
    command = 'git clone git://code.i3wm.org/i3-autobuild-keyring',
)
reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')

for dist in distributions:
    # update the pbuilder
    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')

    # build the package for each dist
    f.addStep(ShellCommand(
        logEnviron = False,
        name = 'pkg ' + dist,
        command = 'pbuilder-' + dist + ' build --binary-arch \
--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
        warnOnFailure = True
    ))

    reprepro_include(f, 'RESULT-' + dist + '/*.changes')

# upload the sid repo
# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
# prefer providing old packages over providing no packages at all :).
for directory in [ 'pool', 'dists' ]:
    f.addStep(transfer.DirectoryUpload(
        slavesrc = 'REPO-sid/' + directory,
        masterdest = 'htdocs/debian/sid/' + directory,
        compress = 'bz2',
        name = 'upload sid ' + directory,
        haltOnFailure = True,
    ))

f.addStep(master.MasterShellCommand(
    command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
    name = 'cleanup old packages',
))

# We ensure there is an empty i18n/Index to speed up apt (so that it does not
# try to download Translation-*)
f.addStep(master.MasterShellCommand(
    command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
    name = 'create i18n folder',
))
f.addStep(master.MasterShellCommand(
    command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
    name = 'touch i18n/Index',
))

The reprepro_include command is defined as follows:

reprepro_include:

def reprepro_include(factory, path, debtype='deb', **kwargs):
    cmd(factory,
        name = 'reprepro include',
        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
        **kwargs
    )

Running such a builder for Ubuntu works exactly the same way, but you need to replace "sid" with "precise" in all places (see the full configuration file for an example).

3.8. Status targets

We don’t advertise the HTTP status target. Instead, status is posted to IRC via a custom bot. This bot provides an HTTP end point and buildbot is configured to push status changes to that endpoint:

http status target:

c['status'].append(buildbot.status.status_push.HttpStatusPush(
    serverUrl = 'http://localhost:8080/push_buildbot',
))

You can find the source code of that bot at http://code.stapelberg.de/git/go-buildbot-announce/. As the name suggests, it is written in Go. Also, it is quite specific to i3, so you might be better off implementing such a bot (or plugin) on your own. It might make for a nice example, though, especially back when its only feature was announcing the build status:

http://code.stapelberg.de/git/go-buildbot-announce/tree/src/i3build.go?id=eeebf1a546454c8a0d82ca623886bb835cd32ba0

3.9. Creating the buildslave

One more thing to note is that when creating the buildslave, you should use the --umask argument to configure the umask for all generated files:

Creating the buildslave:

buildslave create-slave --umask=022 i3-buildslave buildbot.i3wm.org build-1 <password>

4. Full configuration file

This is the full configuration file, as tested and currently in use (except for the passwords, though):

master.cfg:

# -*- python -*-
# -*- coding: utf-8
# vim:ts=4:sw=4:expandtab:syntax=python
#
# i3 buildbot configuration
# © 2012 Michael Stapelberg, Public Domain
# see http://i3wm.org/docs/buildbot.html for more information.

from buildbot.buildslave import BuildSlave
from buildbot.changes import pb
from buildbot.schedulers.basic import SingleBranchScheduler
from buildbot.schedulers.triggerable import Triggerable
from buildbot.process.properties import WithProperties
from buildbot.process.factory import BuildFactory
from buildbot.steps.source.git import Git
from buildbot.steps.shell import ShellCommand
from buildbot.steps.shell import Compile
from buildbot.steps.trigger import Trigger
from buildbot.steps import shell, transfer, master, slave
from buildbot.config import BuilderConfig
from buildbot.process import buildstep
from buildbot.status import html
from buildbot.status import words
import buildbot.status.status_push
from buildbot.status.web import auth, authz
from buildbot.status.builder import SUCCESS, FAILURE

c = BuildmasterConfig = {}

c['slaves'] = [BuildSlave('docsteel-vm', 'secret')]
c['slavePortnum'] = 9989
# Changes are pushed to buildbot using a git hook.
c['change_source'] = [pb.PBChangeSource(
    user = 'i3-source',
    passwd = 'secret',
)]

################################################################################
# schedulers
################################################################################

c['schedulers'] = []

# The first scheduler kicks off multiple builders:
# • 'dist' builds a dist tarball and starts the triggerable schedulers
#   'compile'
# • 'docs' builds the documentation with a special asciidoc configuration
#   (therefore, it does not profit from a dist tarball and can be run in
#    parallel).
c['schedulers'].append(SingleBranchScheduler(
    name = 'dist',
    branch = 'next',
    treeStableTimer = 10,
    builderNames = [ 'dist', 'docs' ],
))

c['schedulers'].append(Triggerable(
    name = 'dist-tarball-done',
    builderNames = [ 'compile', 'clang-analyze', 'debian-packages', 'ubuntu-packages' ],
))

################################################################################
# Shortcuts for builders
################################################################################

# shortcut for a ShellCommand with haltOnFailure=True, logEnviron=False
def cmd(factory, **kwargs):
    factory.addStep(ShellCommand(
        haltOnFailure=True,
        logEnviron=False,
        **kwargs
    ))

# Shortcut to add steps necessary to download and unpack the dist tarball.
def unpack_dist_tarball(factory):
    factory.addStep(transfer.FileDownload(
        mastersrc=WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
        slavedest='dist.tar.bz2',
    ))
    factory.addStep(slave.MakeDirectory(dir='build/DIST'))
    cmd(factory,
        name = 'unpack dist tarball',
        command = [ 'tar', 'xf', 'dist.tar.bz2', '-C', 'DIST', '--strip-components=1' ],
    )

# Includes the given path in REPO-sid using reprepro.
def reprepro_include(factory, path, debtype='deb', **kwargs):
    cmd(factory,
        name = 'reprepro include',
        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include sid ' + path,
        **kwargs
    )

def reprepro_include_ubuntu(factory, path, debtype='deb', **kwargs):
    cmd(factory,
        name = 'reprepro include',
        command = 'reprepro --ignore=wrongdistribution -T ' + debtype + ' -b REPO-sid include precise ' + path,
        **kwargs
    )

################################################################################
# Custom steps
################################################################################

# Adds the ircsuffix property to reflect whether there were warnings.
class WarningsToIRC(buildstep.BuildStep):
  def start(self):
    warnings = self.getProperty("warnings-count")
    if warnings is not None and int(warnings) > 0:
      warnings = int(warnings)  # just to be sure
      self.setProperty("ircsuffix", "\0037 with %d warning%s!" % (warnings, "s" if warnings != 1 else ""))
    else:
      self.setProperty("ircsuffix", "\0033 without warnings")
    self.finished(SUCCESS)

# Adds a link to the automatically generated documentation.
class DocsToIRC(buildstep.BuildStep):
  def start(self):
    self.setProperty("ircsuffix", ", see http://build.i3wm.org/docs/")
    self.finished(SUCCESS)

# Adds a link to the clang report.
class ClangToIRC(buildstep.BuildStep):
  def start(self):
    self.setProperty("ircsuffix", ", see http://build.i3wm.org/clang-analyze/")
    self.finished(SUCCESS)

################################################################################
# Shared steps, used in different factories.
################################################################################

s_git = Git(
    repourl='git://code.i3wm.org/i3',
    branch='next',

    # Check out the latest revision, not the one which caused this build.
    alwaysUseLatest=True,

    # We cannot use shallow because it breaks git describe --tags.
    shallow=False,

    # Delete remnants of previous builds.
    mode='full',

    # Store checkouts in source/ and copy them over to build/ to save
    # bandwidth.
    method='copy',

    # XXX: In newer versions of buildbot (> 0.8.6), we want to use
    # getDescription={ 'tags': True } here and get rid of the extra git
    # describe --tags step.
)

################################################################################
# factory: "dist" — builds the dist tarball once (used by all other factories)
################################################################################

factories = {}

f = factories['dist'] = BuildFactory()
# Check out the git repository.
f.addStep(s_git)
# Fill the 'gitversion' property with the output of git describe --tags.
f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
# Build the dist tarball.
cmd(f, name = 'make dist', command = [ 'make', 'dist' ])
# Rename the created tarball to a well-known name.
cmd(f, name = 'rename tarball', command = WithProperties('mv *.tar.bz2 dist-%(gitversion)s.tar.bz2'))
# Upload the dist tarball to the master (other factories download it later).
f.addStep(transfer.FileUpload(
    slavesrc = WithProperties('dist-%(gitversion)s.tar.bz2'),
    masterdest = WithProperties('distballs/dist-%(gitversion)s.tar.bz2'),
))
# Cleanup old dist tarballs (everything older than tree days).
f.addStep(master.MasterShellCommand(
    command = "find distballs -mtime +3 -exec rm '{}' \;",
    name = 'cleanup old dist tarballs',
))
# Everything worked fine, now trigger compilation.
f.addStep(Trigger(
    schedulerNames = [ 'dist-tarball-done' ],
    copy_properties = [ 'gitversion' ],
))

################################################################################
# factory: "compile" — compiles the dist tarball and reports warnings
################################################################################

f = factories['compile'] = BuildFactory()
unpack_dist_tarball(f)
f.addStep(Compile(
    command = [ 'make', 'DEBUG=0', '-j4' ],
    warningPattern = '.*warning: ',
    warnOnWarnings = True,
    workdir = 'build/DIST',
    env = {
      'CPPFLAGS': '-D_FORTIFY_SOURCE=2',
      'CFLAGS': '-Wformat -Wformat-security'
    },
))

f.addStep(WarningsToIRC())

################################################################################
# factory: "clang-analyze" — runs a static code analysis
################################################################################
# $ sudo apt-get install clang

f = factories['clang-analyze'] = BuildFactory()
unpack_dist_tarball(f)
cmd(f,
    name='analyze',
    command = [
        'scan-build',
        '-o', '../CLANG',
        '--html-title', WithProperties('Analysis of i3 v%(gitversion)s'),
        'make', '-j8',
    ],
    workdir = 'build/DIST',
)

# remove the subdirectory -- we always want to overwrite
cmd(f, command = 'mv CLANG/*/* CLANG/')

f.addStep(transfer.DirectoryUpload(
    slavesrc = 'CLANG',
    masterdest = 'htdocs/clang-analyze',
    compress = 'bz2',
    name = 'upload output',
))

f.addStep(ClangToIRC())

################################################################################
# factory: "docs" — builds documentation with a special asciidoc conf
################################################################################

f = factories['docs'] = BuildFactory()
f.addStep(s_git)
# Fill the 'gitversion' property with the output of git describe --tags.
f.addStep(shell.SetProperty(command = 'git describe --tags', property = 'gitversion'))
cmd(f, name = 'build docs', command = [ 'make', '-C', 'docs', "ASCIIDOC=asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf" ])
cmd(f, name = 'build manpages', command = "for file in $(sed 's/\.1$/.man/g' debian/i3-wm.manpages); do asciidoc -a linkcss -a stylesdir=http://i3wm.org/css -a scriptsdir=http://i3wm.org/js --backend=xhtml11 -f docs/asciidoc-git.conf \"$file\"; done")
f.addStep(slave.MakeDirectory(dir='build/COPY-DOCS'))
cmd(f, name = 'copy docs', command = "cp $(tr '\\n' ' ' < debian/i3-wm.docs) COPY-DOCS")
cmd(f, name = 'copy manpages', command = "cp $(sed 's/\.1$/.html/g' debian/i3-wm.manpages | tr '\\n' ' ') COPY-DOCS")

f.addStep(transfer.DirectoryUpload(
    slavesrc = 'COPY-DOCS',
    masterdest = 'htdocs/docs-git',
    compress = 'bz2',
    name = 'upload docs'))

f.addStep(DocsToIRC())

################################################################################
# factory: "debian-packages" — builds Debian (sid) packages for amd64 and i386
################################################################################

distributions = [ 'sid-amd64', 'sid-i386' ]
gpg_key = 'BE1DB1F1'

f = factories['debian-packages'] = BuildFactory()
# We need the git repository for the Debian packaging.
f.addStep(s_git)
unpack_dist_tarball(f)
cmd(f, name='copy packaging', command = "cp -r debian DIST/")

# Add a new changelog entry to have the git version in the package version.
cmd(f,
    name = 'update changelog',
    workdir = 'build/DIST',
    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
)

cmd(f,
    name = 'source pkg',
    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
    workdir = 'build/DIST',
)

for dist in distributions:
    f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))

# Create debian sid repository
f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
f.addStep(transfer.StringDownload(
    """Codename: sid
Suite: unstable
Architectures: i386 amd64 source
Components: main
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release . .gz .bz2
SignWith: %(gpg_key)s
""" % { "gpg_key": gpg_key },
    slavedest = 'REPO-sid/conf/distributions',
))

# add source package to repository
reprepro_include(f, 'i3-wm*_source.changes', 'dsc')

# Add keyring to the repository. We need to run git clone on our own because
# the Git() step assumes there’s precisely one repository we want to deal with.
# No big deal since the i3-autobuild-keyring repository is not big.
cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
reprepro_include(f, 'i3-autobuild-keyring/prebuilt/*.changes')

for dist in distributions:
    # update the pbuilder
    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')

    # build the package for each dist
    f.addStep(ShellCommand(
        logEnviron = False,
        name = 'pkg ' + dist,
        command = 'pbuilder-' + dist + ' build --binary-arch \
--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
        warnOnFailure = True
    ))

    reprepro_include(f, 'RESULT-' + dist + '/*.changes')

# upload the sid repo
# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
# prefer providing old packages over providing no packages at all :).
for directory in [ 'pool', 'dists' ]:
    f.addStep(transfer.DirectoryUpload(
        slavesrc = 'REPO-sid/' + directory,
        masterdest = 'htdocs/debian/sid/' + directory,
        compress = 'bz2',
        name = 'upload sid ' + directory,
        haltOnFailure = True,
    ))

f.addStep(master.MasterShellCommand(
    command = "find htdocs/debian/sid/pool -mtime +3 -exec rm '{}' \;",
    name = 'cleanup old packages',
))

# We ensure there is an empty i18n/Index to speed up apt (so that it does not
# try to download Translation-*)
f.addStep(master.MasterShellCommand(
    command = [ 'mkdir', '-p', 'htdocs/debian/sid/dists/sid/main/i18n' ],
    name = 'create i18n folder',
))
f.addStep(master.MasterShellCommand(
    command = [ 'touch', 'htdocs/debian/sid/dists/sid/main/i18n/Index' ],
    name = 'touch i18n/Index',
))

################################################################################
# factory: "ubuntu-packages" — builds Ubuntu (precise) packages for amd64 and i386
################################################################################

distributions = [ 'precise-amd64', 'precise-i386' ]
gpg_key = 'BE1DB1F1'

f = factories['ubuntu-packages'] = BuildFactory()
# We need the git repository for the Debian packaging.
f.addStep(s_git)
unpack_dist_tarball(f)
cmd(f, name='copy packaging', command = "cp -r debian DIST/")

# Add a new changelog entry to have the git version in the package version.
cmd(f,
    name = 'update changelog',
    workdir = 'build/DIST',
    command = [ 'debchange', '-m', '-l', WithProperties('+g%(gitversion)s'), 'Automatically built' ],
)

cmd(f,
    name = 'source pkg',
    command = [ 'dpkg-buildpackage', '-S', '-us', '-uc' ],
    workdir = 'build/DIST',
)

for dist in distributions:
    f.addStep(slave.MakeDirectory(dir='build/RESULT-' + dist))

# Create debian sid repository
f.addStep(slave.MakeDirectory(dir='build/REPO-sid/conf'))
f.addStep(transfer.StringDownload(
    """Codename: precise
Suite: unstable
Architectures: i386 amd64 source
Components: main
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release . .gz .bz2
SignWith: %(gpg_key)s
""" % { "gpg_key": gpg_key },
    slavedest = 'REPO-sid/conf/distributions',
))

# add source package to repository
reprepro_include_ubuntu(f, 'i3-wm*_source.changes', 'dsc')

# Add keyring to the repository. We need to run git clone on our own because
# the Git() step assumes there’s precisely one repository we want to deal with.
# No big deal since the i3-autobuild-keyring repository is not big.
cmd(f, name='clone keyring repo', command = 'git clone git://code.i3wm.org/i3-autobuild-keyring')
reprepro_include_ubuntu(f, 'i3-autobuild-keyring/prebuilt/*.changes')

for dist in distributions:
    # update the pbuilder
    cmd(f, name = 'update builder', command = 'pbuilder-' + dist + ' update')

    # build the package for each dist
    f.addStep(ShellCommand(
        logEnviron = False,
        name = 'pkg ' + dist,
        command = 'pbuilder-' + dist + ' build --binary-arch \
--buildresult RESULT-' + dist + ' --debbuildopts -j8 i3-wm*dsc',
        warnOnFailure = True
    ))

    reprepro_include_ubuntu(f, 'RESULT-' + dist + '/*.changes')

# upload the sid repo
# Since the next step is cleaning up old files, we set haltOnFailure=True -- we
# prefer providing old packages over providing no packages at all :).
for directory in [ 'pool', 'dists' ]:
    f.addStep(transfer.DirectoryUpload(
        slavesrc = 'REPO-sid/' + directory,
        masterdest = 'htdocs/ubuntu/precise/' + directory,
        compress = 'bz2',
        name = 'upload precise ' + directory,
        haltOnFailure = True,
    ))

f.addStep(master.MasterShellCommand(
    command = "find htdocs/ubuntu/precise/pool -mtime +3 -exec rm '{}' \;",
    name = 'cleanup old packages',
))

# We ensure there is an empty i18n/Index to speed up apt (so that it does not
# try to download Translation-*)
f.addStep(master.MasterShellCommand(
    command = [ 'mkdir', '-p', 'htdocs/ubuntu/precise/dists/sid/main/i18n' ],
    name = 'create i18n folder',
))
f.addStep(master.MasterShellCommand(
    command = [ 'touch', 'htdocs/ubuntu/precise/dists/sid/main/i18n/Index' ],
    name = 'touch i18n/Index',
))


c['builders'] = []

# Add all builders to all buildslaves.
for factoryname in factories.keys():
    c['builders'].append(BuilderConfig(
        name = factoryname,
        slavenames=['docsteel-vm'],
        factory=factories[factoryname],
    ))


####### STATUS TARGETS

c['status'] = []

authz_cfg=authz.Authz(
    gracefulShutdown = False,
    forceBuild = False,
    forceAllBuilds = False,
    pingBuilder = False,
    stopBuild = False,
    stopAllBuilds = False,
    cancelPendingBuild = False,
)

c['status'].append(html.WebStatus(http_port=8010, authz=authz_cfg))

c['status'].append(buildbot.status.status_push.HttpStatusPush(
    serverUrl = 'http://localhost:8080/push_buildbot',
))

####### PROJECT IDENTITY

c['title'] = 'i3'
c['titleURL'] = 'http://i3wm.org/'
# Removed so that search engines don’t crawl it
c['buildbotURL'] = 'http://localhost/'

####### DB URL

c['db'] = {
    # This specifies what database buildbot uses to store its state.  You can leave
    # this at its default for all but the largest installations.
    'db_url' : "sqlite:///state.sqlite",
}