Continuous integration for open source distros v 3.0

Continuous Integration

For

Open Source Distributions

Sriram Narayanan

sriram@belenix.org

@sriramnrn

About me

Belenix Tester + Developer

Thoughtworker

Why we work on distros

● Innovate● more versatile installation● faster boot up times● experimental drivers

● Learn package management (Rolling updates!)

● Learn how a distro works (LFS,BLFS + more)

Pain points for a Distro Maintainer

● Repeat Tasks

● Dependency Tracking and Management

● Manual Testing

Pain points for a Distro Maintainer

● Repeat Tasks● Track upstream changes and fixes● Regular Distro builds● Changelog and Release Notes● Documentation Updates

● Dependency Tracking and Management

● Manual Testing

Pain points for a Distro Maintainer

● Repeat Tasks

● Dependency Tracking and Management● Dependencies Expressed by not Tested● Insufficient Integration Tests● Heavily dependent on manual testing

● Manual Testing

Pain points for a Distro Maintainer

● Repeat Tasks● Dependency Tracking and Management● Manual Tests

● Package Integration● Installation● Desktop Application Functionality● Service Functionality

● Testing at too many levels● Not just exploratory (More exploratory is good)

Continuous Integration - Basics

● “Integrate changes continuously”● Checkout source, make, make test, package● Usual Implementation

● Automated Agents that perform tasks● Farm of Agents depending upon scale and budget

– Great for packaging multiple apps in parallel– Parallel Tests

Continuous Integration - Benefits

● Detect issues earlier● Package software earlier● Automate various tasks

● Packaging● Testing● Integration● Generate ChangeLog + Release Notes skeleton● Pending Documentation Hints to Documentation

Team● Build the ISO!

Continuous Integration Today

● Isolated build environments● Based on chroot jails● Used by Fedora, Debian, Suse, et al

● Continuous Integration servers● Jenkins/Hudson● Koji● Others (distro specific)

Continuous Integration – What's missing

● Dependency awareness● A “Green” upstream should trigger downstream

builds● Complex Dependency graph builds and tests● Prevents Automated Consolidation Releases

● A Variety of Testing● “make test” is not enough!

Continuous Integration – What's missing

● Dependency Awareness

apr

httpd

subversion

Image Credits: http://cd-with-thoughtworks-go.blogspot.in/

Why “make test” is not enough

● Tests the app only on the compiled system● No tests post installation

● Packaging occurs after make test● Packaging errors are not detected

– Incorrect placement, ownership and permissions– Conflicting requirements– Missing files

● Integration Tests needed

./configure makemake make test package

Testing

Unit

Service Layer(Integration)

UserInterface

The Test Pyramid – Mike Cohn

http://martinfowler.com/bliki/TestPyramid.html

About Unit Tests

● Unit Tests● Requirements in executable form● Detect what exact functionality has broken

About Integration Tests

● Service Level Tests● Is ComponentA correctly calling ComponentB?● Is ComponentA fulfilling its public API contract?

About UI Level Tests

● Test End to End Functionality● e.g. Sahi based tests

What Tests tell you

● UI Tests - “Something in this feature broke”

● Integration Tests - “Component B returned an unexpected response to Component A”

● Unit Tests - “When we introduce functionality requirement #13 into Component B, then our code causes requirements #7 and #9 to break”.

Post – Installation Integration Tests

● CentOS package level tests● A good start!

● Some useful tests for distro maintainers– Is httpd able to serve files ? (selinux issues!)– Is SSH accepting connections ?– Did the network subsystem initialize correctly?– Did the installer set the MBR + active partition correctly?– Is the DNS server accepting queries ?

● Is the application behaving as required?

CI in Belenix

● Distro tests + Consolidation level smoke testskernel

httpd

postgres

ssh

WebStack

nginx

python

httpd

vi Userland

shell

awkIntegration

Tests

kernel

AutomatedInstall

Post InstallTests

Distro + package levelsmoke tests

Roadmap - CI and Belenix

● Package early● Consolidations – kernel, userland, subsystems

(storage, firewall, load balancer)● Custom distros – CoreOS, storage, firewall,

load balancer● Each with its own functional test suite!

● CI for open source projects that we bundle● Potential inter-distro inter-project collaboration

Thanks!

www.belenix.org

@belenix

@sriramNRN