Devops malarkey

Success. Failure. Cake.

Actually Just Testing Something Else

You’d think, after all this time, that I could bosh things together and have them perform some semblance of useful work, right?

You’d think…

… Argh. ‘-’ characters in directory names? Surely not…

Leveraged Neatness Is Not Always Your Friend

I must begin by apologising for using the word ‘leveraged’. It’s terrible marketing-speak. However, it mostly fits in context and anyway I am using it in an ironic sense. (not that that’s even a valid excuse for the likes of me, but there you go.)

There are certain aspects of the admin trades where semi-obsessive neatness and/or attention to detail are useful things to have. It’s one of the times where not being able to leave the house without checking that you’ve turned the gas off twice and locked the front door three times can be a positive thing to admit in an interview. [NB: Self-deprecating hyperbole.]

It can also lead to the sort of unfortunate failure-modes that if you’re lucky you’re not familiar with.

(This was all kicked off by one of the network guys asking me some hard questions about the nature of packaging and dependencies, which set me off on a bit of a ramble. Thus network types get it first. Beard-fondling admins get their kicking later.)


For reasons that seemed jolly sensible at the time (ie - someone dared me to do it) I gave a much-modified version of The Puppetcamp Talk at EMFcamp. Modified in that it was four months later and we’d made some things work better, abandoned others and had some mostly-bright ideas for new stuff.

Thus linked here somewhere should be the PDF version of the slides, which because I am that sort of forward-thinking sod (actually, I’d seen this be a requirement at 28c3, and it seemed sensible, so…) was the version I presented because the Nice People didn’t have the relevent podule to connect a Macbook to the big screen in the tent.

In Which There Is Silence and Excuses for Same

Hello. Where on earth did the last few months go? I guess I could blame it having been Grand Tour season (the Giro, the Tour and now the Vuelta) which has meant that I’ve been watching sweaty men in lycra pedal up the sides of mountains for three weeks at a time.

I guess I can also blame one of those periods when several mildly concerning things happen in rapid order like a conspiring mob of buses, and when the smoke clears you’re somewhere grim like Burnt Oak, Oldland Common or Quedgeley.

Anyway. Let’s see what happens next.

A Hazelnut in Every Bite

An obvious question is ‘Why on earth bother with all this message-wanging gubbins? Isn’t mail and/or SSH good enough for you?’ to which the glib answer is ‘No it isn’t.’

A longer answer involves spotting the really obvious problem in my last post. That is ‘Ok, so you’ve bodged in some code that’ll auto-update your live puppetmaster tree on a commit to that repository. What are the chances that said commit is b0rked and causes an epic cake-fail?’

Well, in theory you’re committing to a develop branch which only one or two of your machines are following, because that’s the entire point of having the dynamic branch rig in the first place. However, PEBCAK happens and perhaps you should have a Jenkins instance that sanity-checks (as much as is possible, anyway) the puppet code that’s just been committed.

Hurrah! Problem solved! Let’s have one of those!

I don’t know how you’d plumb something like that into another environment, but this is how it works here:

  • Configure the stomp-jenkins daemon to listen for puppet-environments commits on future.git.commits (c0dez available in the usual place)
  • Configure Jenkins for Puppet and puppet-lint
  • Configure Jenkins to emit a message on (say) future.jenkins.success if the tests pass. (c0dez for that available ditto)
  • Configure the stomp-git daemon on your puppetmaster to listen on future.jenkins.success
  • Er, profit.

Obviously enough, chaining in extra bits or constructing side-chains is relentlessly trivial.

You could probably do it with a massive make or rake file, too.

One Very Important Thought

I have been hacking on stomp-git over the past couple of days. Mostly because there may or may not be a lurking nasty with the way it sometimes stops listening on its, er listen topic, but also because it has needed a deal of de-Futuring and general fettling so it doesn’t make proper coders claw their own faces off in horror.

And because I had forgotten how I’d made it work when it came to the svn->git rollout of one of our major sites.

Thus it seems as good a time as any to explain how I think it should work and indeed why it works like that.

Java-based Diversionary Tactics

The other week, I stood in front of a room filled with my notional peers and allowed as how ‘ActiveMQ setup was a bit of a pig, but once you’d got it working it was pretty simple.’

[FX: Pause for hollow laughter]

In retrospect, that statement was obviously going to come back and bite me just as soon as it had located its special big false teeth. Thus it came to pass on Wednesday night that two of the brokers had a meltdown, which in turn broke some experimental Nagios-over-Stomp code and so kept the poor sod who was on call in a state of near panic as everything reportedly failed.


So when I pitch up on Thursday AM, I am welcomed by the whole team merrily grousing about ‘your effing brokers’ (Some software objects are like children and pets. When they’re well behaved and/or looking cute for visitors, they’re our children or pets. When they’ve just left a deposit behind the telly, they’re your children or pets). Indeed it was a right mess. One broker had used all the memory allocated and as if for spite had run out of filehandles too. It was all a bit odd. Some poking about revealed [AMQ-jira ticket] which more-or-less explains itself and the fact that it looked an awful lot like an experimental client wasn’t responding to messages as quickly as one might hope.

Much Todo About Some Things

In the wake of Puppetcamp Edinburgh, which was jolly nice, I emitted about half a screen of rambling on one of the work IRC channels as a set of notes for future selves (pun very much intended). This is something I do rather a lot, because obviously it’s impossible to perform any sort of useful thinking in an open-plan office filled with cheery sorts who Want Things Now. The drive/ride home is also a time for sudden and Hoffman-like inspirations.


Whatever I wrote has vanished off the end of the scrollback, so I’ll have to make it all up again.