Introducing debos, a versatile images generator

In Debian and derivative systems, there are many ways to build images. The simplest tool of choice is often debootstrap. It works by downloading the .deb files from a mirror and unpacking them into a directory which can eventually be chrooted into.

More often than not, we want to make some customization on this image, install some extra packages, run a script, add some files, etc

debos is a tool to make this kind of trivial tasks easier. debos works using recipe files in YAML listing the actions you want to perform in your image sequentially and finally, choosing the output formats.

As opposite to debootstrap and other tools, debos doesn't need to be run as root for making actions that require root privileges in the images. debos uses fakemachine a library that setups qemu-system allowing you to work in the image with root privileges and to create images for all the architectures supported by qemu user. However, for this to work, make sure your user has permission to use /dev/kvm.

Let's see how debos works with a simple example. If we wanted to create an arm64 image for Debian Stretch customized, we would follow these steps:

debootstrap the image
install the packages we need
create an user
setup our preferred hostname
run a script creating an user
copy a file adding the user to sudoers
creating a tarball with the final image

This would translate into a debos recipe like this one:

{{- $architecture := or .architecture "arm64" -}}
{{- $suite := or .suite "stretch" -}}
{{ $image := or .image (printf "debian-%s-%s.tgz" $suite $architecture) }}

architecture: {{ $architecture }}

actions:
  - action: debootstrap
    suite: {{ $suite }}
    components:
      - main
    mirror: http://deb.debian.org/debian
    variant: minbase

  - action: apt
    recommends: false
    packages:
      - adduser
      - sudo

  - action: run
    description: Set hostname
    chroot: true
    command: echo debian-{{ $suite }}-{{ $architecture }} > /etc/hostname

  - action: run
    chroot: true
    script: scripts/setup-user.sh

  - action: overlay
    description: Add sudo configuration
    source: overlays/sudo

  - action: pack
    file: {{ $image }}
    compression: gz

(The files used in this example are available from this git repository)

We run debos on the recipe file:

$ debos simple.yaml

The result will be a tarball named debian-stretch-arm64.tar.gz. If you check the top two lines of the recipe, you can see that the recipe defaults to architecture arm64 and Debian stretch. We can override these defaults when running debos:

$ debos -t suite:"buster" -t architecture:"amd64" simple.yaml

This time the result will be a tarball debian-buster-amd64.tar.gz.

The recipe allows some customization depending on the parameters. We could install packages depending on the target architecture, for example, installing python-libsoc in armhf and arm64:

- action: apt
  recommends: false
  packages:
    - adduser
    - sudo
{{- if eq $architecture "armhf" "arm64" }}
    - python-libsoc
{{- end }}

What happens if in addition to a tarball we would like to create a filesystem image? This could be done adding two more actions to our example, a first action creating the image partition with the selected filesystem and a second one deploying the image in the filesystem:

- action: image-partition
  imagename: {{ $ext4 }}
  imagesize: 1GB
  partitiontype: msdos
  mountpoints:
    - mountpoint: /
      partition: root
  partitions:
    - name: root
      fs: ext4
      start: 0%
      end: 100%
      flags: [ boot ]

- action: filesystem-deploy
  description: Deploying filesystem onto image

{{ $ext4 }} should be defined in the top of the file as follows:

1	{{ $ext4 := or .image (printf "debian-%s-%s.ext4" $suite $architecture) }}

We could even make this step optional and make the recipe by default to only create the tarball and add the filesystem image only adding an option to debos:

$ debos -t type:"full" full.yaml

The final debos recipe will look like this:

{{- $architecture := or .architecture "arm64" -}}
{{- $suite := or .suite "stretch" -}}
{{ $type := or .type "min" }}
{{ $image := or .image (printf "debian-%s-%s.tgz" $suite $architecture) }}
{{ $ext4 := or .image (printf "debian-%s-%s.ext4" $suite $architecture) }}

architecture: {{ $architecture }}

actions:
  - action: debootstrap
    suite: {{ $suite }}
    components:
      - main
    mirror: http://deb.debian.org/debian
    variant: minbase

  - action: apt
    recommends: false
    packages:
      - adduser
      - sudo
{{- if eq $architecture "armhf" "arm64" }}
      - python-libsoc
{{- end }}

  - action: run
    description: Set hostname
    chroot: true
    command: echo debian-{{ $suite }}-{{ $architecture }} > /etc/hostname

  - action: run
    chroot: true
    script: scripts/setup-user.sh

  - action: overlay
    description: Add sudo configuration
    source: overlays/sudo

  - action: pack
    file: {{ $image }}
    compression: gz

{{ if eq $type "full" }}
  - action: image-partition
    imagename: {{ $ext4 }}
    imagesize: 1GB
    partitiontype: msdos
    mountpoints:
      - mountpoint: /
        partition: root
    partitions:
      - name: root
        fs: ext4
        start: 0%
        end: 100%
        flags: [ boot ]

  - action: filesystem-deploy
    description: Deploying filesystem onto image
{{end}}

debos also provides some other actions that haven't been covered in the example above:

download allows to download a single file from the internet
raw can directly write a file to the output image at a given offset
unpack can be used to unpack files from archive in the filesystem
ostree-commit create an OSTree commit from rootfs
ostree-deploy deploy an OSTree branch to the image

The example in this blog post is simple and short on purpose. Combining the actions presented above, you could also include a kernel and install a bootloader to make a bootable image. Upstream is planning to add more examples soon to the debos recipes repository.

debos is a project from Sjoerd Simons at Collabora, it's still missing some features but it's actively being developed and there are big plans for the future!

Debconf15 and ... happy birthday Debian!

Debconf15 started yesterday and as expected, talk rooms are always fully crowded! I had to stand up in a couple of talks and I watched another couple of them from outside thanks to the real time streaming.

Thanks to the fantastic work of the video team, video recordings of the talks from yesterday have started to be available at http://meetings-archive.debian.net/pub/debian-meetings/2015/debconf15/.
I would heartily recommend you to watch "Debian's Central Role in the Future of Software Freedom" by Bradley M. Kuhn (video available).
I was expecting a good talk and Bradley exceeded my expectations.

I also got to meet again Simon Kainz and to get my DUCK branded lighter from the duck challenge :)

After dinner, we celebrated Debian's 22 birthday, that's exactly today. We had a wonderful cake made by DebConf attendees made by small pieces of pastry with fruits drawing a mosaic with a Debian swirl.
While the cake was a very nice detail, the best part of it was watching the people making the cake. Everybody had a great time and this kind of things are what make Debian (and DebConf) great. When people work together to make something wonderful.

Keysigning, it's never too late

I managed to sign all the keys from DebConf14 within a month after the conference ended, but I had still a pile of keys to sign
going back to DebConf11 from several minidebconfs and other small events. Today it was frakking cold and I finally managed to sign something close to 100 keys. I didn't sign any key with less than 2048 bits, expired or revoked.

If you got your key signed by me (or get it in the next hours), sorry it took so long!