Tuesday, 21 October 2014

Why fake RPM's ?

In IT companies across the globe, infrastructure and servers are provisioned at the click of a button - this is the world the Devops engineer is creating.  The deployment tool we used at this particular organisation is called ansible.  Ansible allows the programmatic install of RPM's and config files on target servers - our whole estate was built off the back of ansible scripts.

We wanted to develop the ansible scripts using TDD (Test Driven Development), so that we could put the whole regression test and deployment into a pipeline.

Importantly for TDD to work you get yourself into a feedback loop:

1) Write specs
2) Update ansible scripts
3) Run specs  (this will start with an empty base server build then apply the ansible scripts)
4) Repeat 2 and 3 until specs pass

The problem we had was that running the specs could take up an hour.  The feedback loop was way too long!  It turned out that some of the RPM's which the ansible scripts install took the best part of 30 mins to complete.

So we made a compromise - we would install fake rpms for the longest running offenders (luckily these were "bloated 3rd party monitoring tools" so not critical to the servers functionality)


FPM is a ruby gem to create .rpm's (or .debs if running on ubuntu).  To install you will need ruby:

curl -sSL https://get.rvm.io | bash -s stable
rvm install 1.9.3
rvm use 1.9.3 --default
gem install fpm

Fake it

The idea, is to add the fake rpm to the yum repository (alongside the original), but give the fake a higher version number, so it gets picked up by the yum install command.

What we found was that in some of the in house rpm's, the epoch had been set to the unix epoch time and in fact whenever the rpm was regenerated the epoch would increase.  In this scenario, the highest epoch wins, irrespective of the version number.

So the strategy for creating the winning the version, is very much dependent on how the original RPM was versioned:
mkdir -p /tmp/fake_package
fpm -s dir -t rpm -n sysmontool -v 999999999 --epoch 2000000000 -f -a noarch -C /tmp/fake_package ./
fpm -s dir -t rpm -n patchmontool -v 0.0.1 --epoch 2000000000 --iteration 999 -f -C /tmp/fake_package ./

Strangely, you can not just set the epoch to 9999999999,   We actually set the epoch to `date +%s`


The fpm command will create a .rpm.  Copy this into your yum repository and do a yum update.  You make need to do a yum clean all if its not getting picked up.

From experience, for server builds which are automated, it might be a good idea to set the yum,conf metadata_expire field to a low value.  This avoids the confusing scenario of thinking that something has installed/updated when it hasn't.

No comments:

Post a Comment