Dealing with rapid change

It’s never going to be perfect

I know a number of sysadmins, myself included; who always are striving to get to the “perfect” system. It’s a wonderful goal to have and it should definitely be what you’re aiming to do, but you shouldn’t hold onto the perfect vision despite everything else, at the end of the day you are at work to provide value to the business not to hinder it. This is not to say that you shouldn’t have morals, and that there are not things that you need to stick to your guns on, as long as there’s a good business reason for it you can pretty much do that, but again, at the end of the day you are there for the businesses benefit.

The first part of dealing with rapid change is to realise that it will never be perfect, any designs you draw up now will be totally inaccurate a few months down the line or the business would have changed significantly where what you were aiming for has now moved. Goal post syndrome. So you have to be adaptable, your plan can not be strict in execution but it can be a guideline to what you are trying to achieve. Once you have accepted that it will never be perfect it makes life a lot easier when compromising on the little decisions and means you can focus on getting towards the goal rather than achieving the goal, as long as each change is a progression towards the end goal there is really no need to argue the point, you may want to go “above and beyond” to deliver a little more benefit but that should be off your own back.

The second part is all about knowing what is important to hold onto and what is not. It’s very easy to treat every decision as being vital to achieving the goal but normally they are not so important. As I mentioned before, it’s more important that you are moving towards the goal than achieving it, so any decision that moves towards the goal should not be contested, by all means try and add extra work in to get to the goal quicker but do not contest the task.

You should never really carry out tasks that make things worse, and I’d suggest it is these that you hold onto a little bit (not a lot…), make sure you point out why the task makes things worse, if needed make sure you get it in writing, but ultimately you need to get the task done if the business needs it, regardless of it making the overall system worse.

Making sure it doesn’t get un-manageable

So assuming that you have had to make some changes that are not in-line with the end goal you need a way of dealing with it, you have to track the fact that because of one change you have extra work to do. This is called “Technical Debt”, it’s what you accrue when you make a change that is not overall beneficial to the management of the environment or has a negative impact on the business in the long run. You’re always going to have some technical debt because you will never reach your goal, and if you do you will be dreaming up new ways to create technical debt.

These types of changes that produce technical debt can be summarised as “Tactical” typically because the business needs it now to remain competitive. The trick here is to not be objectionable to doing the tactical change, at the end of the day you have to deliver X by Y, so the scope of changing how you do that is finite, from the sysadmin point of view you can do an elaborate deployment if you wish, as long as you meet the time based deadline. This gives you the ability to try and carry out the task in a better way and hopefully reducing the technical debt.

This post was all about how to deal with rapid change, so far I’ve only really taked about making the decisions around what is important to hold on to and not arguing about every detail. This is important as it will make you less objectionable and hopefully minimise the technical debt, but you still need to deal with the technical debt.

Every time you have to make a tactical change that produces technical debt, spend a couple of mins logging tickets or making note of all of the individual actions that need to happen to get back out of the situation that has now been created. This is vital as the next time it comes to prioritising a project you have a shed load of evidence that says what you need to already fix to get back to a positive position, to make sure that is easy to plan in when you log the tickets make sure you capture the amount of time it will take to do the task.

Hopefully your boss will like the fact that the work has been captured and you can discuss how to reduce the technical debt to get “back on track” if you’re really unfortunate and your boss still has desires for tactical changes that eventually lead to fire fighting you have a nice list of tass that you have raised to say that you should be doing those to get back to a point of not fire fighting.

Summary

Don’t be too stuck on arguing the point of a task, eventually you’ll lose so just focus on minimising the technical debt that it causes and where possible only make changes that progress towards the end goal. I’ll write something else next week about dealing with technical debt, mainly because this was getting too long. Just take away from this that you don’t have to hold on to you technical bone to get your way, you just need to make sure that if there is a negative change the remediation work is tracked.

Google Apps – How easy is it?

A bit of context

Last week, very much inspired by the Internal IT team’s flawless switch over to Google Apps I decided it was about time I resurrected my old email account which was off of my personal domain. Now, I use to have a friend run a mail server for me and that worked okay, I use to run a mail server and that was okay also. Well, apart from the copious amounts of spam, I took drastic measures when I was hosting it with spamassassin and blacklisted domains etc etc but still spam made it through.

Spam was the main reason why I stopped hosting my own email there was just a lot of it and it was becoming much more of a chore than I would have liked. Like most people I’m busy I don’t really want to get home after work and find out that I’ve received 3000 emails, of which some may be legitimate, I also didn’t want to spend huge amounts of time trying various different tools to cut down on the emails, so I stopped hosting it and let it die.

It has been dead now for at least 3 years, and with this blog and seeing how easy google apps was I thought “Why not get my email back up and running, but this time pay someone to host it and take care of the crap” This was a good idea, I decided to do a bit of digging into Google Apps and the costs.

What I found out was…

I was surprised that google offered google apps for $5 per user per month, Not bad I thought, but I kept digging, I wanted to make sure I was getting good value for money and wanted to check I had the right plan for me. So, I went to the comparison page as a place to compare the different options.

Now up to this point the most annoying thing with the google apps was the focus on the business side, I may / may not use my domain for business, I have no income, I have no outgoings I just wanted my mail hosted for me, maybe my wife too so I was very pleased to see the individual option on the comparison page.

What I liked about this was that I was an Individual / Group / Entrepreneur, which automatically entitled me to a free account. This was purely by luck that I found this and I was surprised to find it, it was exactly what I wanted, free email hosting by a company that knows how to handle spam and not get me involved in the process. I have always been fond of GMails ability to filter spam, and now I had it for my personal domains!

Was it easy?

It definitely wasn’t easy to find the free email hosting, but it was really straight forward to set up. There’s a nice walk through that is really simple to follow, the hardest bit was verifying the domain, mainly as I don’t host my website any more, so I spent 15 mins setting up a site on www for my domain to realise it wanted it on the root of the domain, at which point I went for alternative ways to register the domain. A true winner came up, add a TXT record to DNS! Luckily I host my own domain, by that I mean I have the authoritative zone and I send it out to some public slaves to do the leg work thanks to Gridstar. So it was that simple a few step by step instructions on the setup; a bit of time to authorise the domain and bingo, working email hosted by Google.

Of course I had a Gmail account already, and I was able to use the multi account feature to login to both accounts and flip between my email accounts. So far it’s only been a couple of days but it works fine.

As I mentioned I was some what spurred on by my companies move to google apps, if they hadn’t have moved I wouldn’t have looked at google apps at all, either way I’m now pleased I have my domain hosting emails again, it won’t be long and I’ll have it hosting a www website as well!

Hiring in IT

The background

So for those of you that do not know me, I work at a Open Source company called Alfresco and we are hiring a lot of people this year. As a result we are suffering like all other IT companies trying to hire in the UK at the moment, there’s a skills shortage in IT and it is not something that we as an individual company can solve. If you are a person currently looking for work and struggling in IT I suggest you stop reading now.

What’s the real issue

It’s not a lack of candidates, we get a regular stream of good looking CV’s, but we’re not in the habit of looking for just anyone, we want people that are passionate about open source, people that want to make a difference. I suppose that by it’s self is not difficult, but when you combine that with a relatively decent job spec, and then the personal fit of the team etc it becomes a bit more challenging. We quite regularly get either very personable people that are lacking the technical skills to do the job or some very technical people that are missing the personal skills, needless to say it’s about compromise.

The above is based on a good candidate that just isn’t right for us, however in most cases (80% ish) the candidates are, how do we say this, overly optimistic with their CV’s? there’s been a number of people we’ve interviewed off of the strength of their CV and then they honestly expect us to not ask questions about their past experiences. Quite a number of CV’s have items on them where the person was part of a team doing the task and they claim (on the CV) to have been more or less the sole member, it all becomes very painful in the interview when you start asking questions on it. Do people really think we won’t ask?

So with that in mind, be weary of CV’s that have a lot of jargon on them, if possible aim for CV’s that have less on them, especially if that’s backed up with years of experience. it is more likely that that candidate is only listing things they are proficient in rather than things they have touched, or watched someone else touch.

The right compromise

If you think you won’t have to compromise on some element of the job spec / candidate persona then you should also probably stop reading this and go back to that wonderful place you come from where pigs fly and monkeys act as butlers.

It’s easier to say this than to live with the consequences, needless to say I have been unfortunate enough to live with some compromises that just weren’t quite right. Lets look at it in little chunks.

Technical ability is not everything, if you do have to compromise, compromise on this, you shouldn’t shoot your self in the foot, but choose a candidate that can do the job at hand and has room to grow. Hopefully the candidate will want to learn new things and develop their careers so it wouldn’t be the end of the world as long as they show signs of wanting to progress and develop.

Personality is key, there’s no point hiring someone that will come in and act as a rouge agent or isolated because they can’t gel with the team. I’d say the personality of the person is an area where you really don’t want to compromise on, if they only seem to slightly not fit with the team don’t worry about it, the team dynamic will change slightly but it should settle, you would need to identify which elements of the team’s “quirks’ you want to preserve and which ones would be benefited by having a slightly different personality in the team.

A little digression but probably valuable at this point, in the past I’ve had a sysadmin working along side me who seemed personable and would fit in personality wise (and they did) they seemed a little weak technically but we hoped that would develop; it turned out when they started they thought they were more technical than they actually were. They were the sort of person that you could give a task to and eventually they’d have something that worked, but they may not necessarily know how they got it to work. They also had issues arguing technical points, sticking hard and fast to their guns while not having a good reason to at the time and requiring a few days of googling to come back with a link that summarised that you could do it their way or the other way, so it also didn’t back up their ideals.

To be honest it was a real struggle for me personally as what I needed in the team was someone that was technical enough to take some of the workload off of me, but despite spending time training and explaining why things were done that way, the knowledge didn’t sink in. I’d feel bad that I couldn’t get the information in, but other people tried and failed as well, so the common problem is probably the cause. The biggest issue was that they were not willing to progress, maybe because they thought they knew it all, or because they just weren’t able to grasp the technical detail.

So if you have to compromise on technology skills make sure they are personable and willing to learn and progress, people can develop technical know how if they have the personality that allows them to, but changing someones personality is much much harder.

If you’re interested in working at Alfresco you should keep a regular eye on our Careers page, it doesn’t have all of our jobs on it so feel free to email your CV through just on the off chance, we’re always looking for good people.

Ruby, Pass by value or pass by reference?

I wish I knew for sure…

The only reason I’m writing this is I got bitten while writing some code for Sentinel I was expecting a value based on the fact I thought I was passing by value so the original object would remain unaffected.

Needless to say it was actually passing by reference, although if you read some articles it suggests that it actually only passes by value and those values just so happen to be object ID’s which point to objects… It get’s very complicated.

In an earlier post I talked about my programming ability, and I have written not to long ago about starting to code in Ruby So to be clear, I am not a programmer, I never will be, this stuff that i’m writing about now is what I’d class as ground breaking for me, it’s probably a really simple concept, but for some reason it has been really difficult to get my head around I struggled in C with it, more so in Java, and now Ruby; the difference? this time I actually went off did some reading and got some friends involved in a rant (both programmers who understood it all instantly even though they’d never seen ruby before, Annoying)

My quest started as most do, on Google where I came across this I’d highly recommend reading this page and the following two links that appear on that page One, two a combination of those and more reading and more talking I think I finally got my head around it.

Where did it go wrong?

Laziness is probably the answer here, I didn’t bother to read anything about it I just found a tutorial and went with it, got my head around the basics and started to dive in; which is fine as long as you don’t mind spending your evenings like I did, reading lots and coding.

I started out on an assumption that unless I specified otherwise it would be pass by value, mistake 1. In fairness it wasn’t a big mistake but annoying none the less. The other mistake I made was just not doing any research into how the langage worked, I could have spent a little time looking. So just now I did that a 2 min google and i came across this which would have saved me a little time, Granted I should have followed this straight after following Steve Litt’s it would have saved me a lot of hassle.

There is but one more thing I would have done differently, I would have read about the class I was about to use… Instead I was following examples and basically shooting in the dark and hoping it would get better, I wouldn’t do it with Sysadmin, so why do it with programming?

Just those few things would have made a big difference, not only to my understanding of Ruby but also would have ultimately saved me coding a lot of return’s that were not needed, so now I have to re-write some Sentinel code and make better use of methods I didn’t know existed. Well, at least it keeps me busy and I’m still learning new things.

Summary

I realise in the middle of this I digressed a little, but The links above are all very useful, especial those for Sentinel! If you are looking to get into Ruby I’d suggest the tips and links in “Where did it go wrong?” It would have saved me time and effor so no doubt it will save you time and effort; likewise if you were looking for the really techy explanation of Ruby pass by value or reference re-read “I wish I knew for sure…” all the information is there. I could try my best to explain it but that would be morally wrong, who knows what tripe could be produced based on a beginners understanding of a core piece of functionality.

How to get a spaceship

It’s all about aiming High!

I remember a while back I was reading a book about goal setting (I can’t remember which one now) and it was touching on the points of goal setting and achievement of goals. The bottom line for those who can’t be bothered to read on is that you typically will achieve 80% of a goal. I took this and had a bit of a think about it, if you have a goal that is large enough and you actively work towards it you’re going to achieve 80% of it, knowing that is good, aim 20% higher…

I’m not really sure if that will help in the long run but it is a good guideline, I’m not exactly known for my small understated targets and ambitions, part of the reason behind this is the 80% targets. I know that if I achieve 80% of what I set out to do I’m happy with that. What more can you ask for, it’s crazy to think that you’re going to achieve all your goals, so just be happy with 80% but if that means you need to aim 20% higher do that.

I’m not sure of the psychology involved but by having a higher goal you just try harder to get there, I’d suggest this is balanced up with a bit of reality and what is actually achievable, just like the spaceship, a small tiny bit of reality.

So where does the spaceship come into it?

Well this was jus a personal goal I set myself, I simple want a spaceship, nothing flashy just enough to get into orbit; have a bit of zero-G and back again. This all stemmed from a couple of years ago when I was having serious thoughts about setting up a small business which started off as a website but ended up with a spaceship, it’s a weird set of steps to go through to end up with a spaceship.

In my head the business plan was good, but as with these things a bit of planning and a bit of graft reveals that although it could be sustainable and provide an income I could live on it was not the grand scheme I was looking for so, in short I shelved the idea.

However I haven’t lost site of my goal, I have no delusions of grandure, I just want to get 80% of the way there, so maybe a small aeroplane or something similar. That’s what the spaceship is about, it’s about having a goal to stretch for.

This does mean I’m always looking for a big idea to get into to help me get my spaceship, but it is a goal, a goal I’m aiming for, it’s also one I have no real plan for, well other than the South park (contains adult content) school of planning.

I guess this also stands as a marker in the ground for everyone reading, let’s see how close I get to having a spaceship, note I reserve the right to just go out and buy a small plane if needed ;-)

Summary

I could spend a long time here going through different ways to plan things and how to get to the goal but it is kind of all pointless, this whole taks is more about setting expectations and having a positive outlook on what you can achieve. When you aim higher you get further, you need to have passion, drive and commitment to the goal.

By having a slightly higher goal that intended you will find your self getting closer to your original intention.

But I digress, this was about getting a spaceship, well, quite simply, aim for a space station.

Sentinel – An open source start

An open source start

Last week I introduced a concept of Self Healing Systems Which then lead me on to have a tiny tony bit of a think and I decided that I would write one, the decision took all of 5 mins but it gives me an excuse to do something a bit more complex than your every day script.

I created a very simple website here which outlines my goals, as of writing I have got most of the features coded up for the MVP, and I do need to finish it off which will hopefully be soon, which will hopefully be by the time this is published, but let’s see.

I decided to take this on as a project for a number of reasons:

1. More ruby programming experience
2. Other than Monit there doesn’t seem to be any other tools, and I had to be told about this one…
3. It’s a project with just the right amount of programming challenge for me
4. I like making things work
5. It may stop me getting called out as often at work if it does what it’s meant to

So a number of reasons, and I’ve already come across a number of things that I don’t know how to solve or what the right way of doing it is. Which is good I get to do a bit of googling and work out a reasonable way, but to be honest that is not going to be enough in the long run. hopefully as time goes on my programming experience will increase sufficiently that I can make improvements to the code as time goes by.

Why continue if there’s products out there that do the same thing?

Why not? Quite often there’s someone doing the same thing even if you can’t find evidence of it, competition should not be a barrier to entry, especially as people like choice.

I guess the important thing is that it becomes usefully different, Take a look at systems management tools, a personal favourite of mine, you have things like RHN Satellite, Puppet and Chef 3 tools, 1 very different from the other two and another only slightly different. People like choice, different tools work differently for different people.

I guess what I mean by that is that some people strike an accord with one or another application and as a result become FanBoys, normally for no good reason.

There’s also the other side of it, I’ve not used monit, I probably should, I probably won’t; but it doesn’t sound like where I want to go with Sentinel. Quite simply I want to replace junior systems administrators, I don’t want another tool to be used, I want a tool that can provide real benefit by doing the checks on the system, by making logical deterministic decisions based on logic and raw data, and not just by looking at the systems it’s on but by considering the whole environment in which it is part of. I think that is a relatively ambitious goal, but I also think it is a useful one, and hopefully it will get to a point where the application is more useful than the MVP product and it can do more than just look after one system.

Like any good open source product it will probably stay at version 0.X for a long time until it has a reasonable set of feature sin it that make it more than just a simple ruby programme.

A call for help

So I’ve started on this path, I intend to continue regardless at the moment and one thing that will help keep me focused is user participation either through using the script and logging bugs at the github site it’s hosted on.

I think at the moment what I need is some guidance on the programming of the project, it’s clear to me that in a matter of months if not weeks this single file application will become overly complicated to maintain and would benefit from being split out into classes. Although I know that, I do not know the right way of doing it I don’t have any experience of larger applications and the right way to do it so if anyone knows that would be good!

In addition to the architecture of the application there is just some programming issues which I’m sure I can overcome at some point but I will probably achieve the solution by having a punt and seeing what sticks. There’s a wonderful switch in the code for processor states which needs to change. I need to iterate through each character of the state and report back on it’s status where as at the moment it is just looking for a combination. To start with I took the pragmatic option, Add all of the processor states mys system has to the witch and hope that’s enough.

So if anyone feels like contributing, or can even see a simple way of fixing some dodgy coding, I’d appreciate it, I guess the only thing I ask is if you are making changes, See the README, Log a ticket in github and commit the changes with reference to the ticket so I know what’s happened and why.

So please, please, please get involved with Sentinel

Are you doing what you love?

Well, are you?

I am a firm believer in doing what you love todo, What I mean by that is that if there is something you like todo you should try to find ways to make it a central part of your life.

It doesn’t make sense to be doing a job that you don’t like or that you are uncomfortable with and if you want to get ahead and you want to be more than average you have to do something you are passionate about. Just look around the office and see how many people you perceive to be more than average I bet all if not most of them are passionate about what they’re doing…

So the question is are you? if you ever wake up and think “I just don’t want to do it today” Then probably not, and even if you do think that, work out is it because you don’t like what you’re doing or the environment around you?

If you are not doing what you love to do you need to first work out what that is and then work out how you can spend more time doing it.
I can’t tell you what you love to do, but you will have an idea about it, what do you do in your spare time, when you’re bored of sitting in front of the tv and you really feel like doing something, for me I always end up fiddling with one of the servers I have or writing a small application or doing something a little techy.

It’s not all roses

The only reason I’m a sysadmin right now is I like knowing how things work, I like diving into the nitty gritty detail and coming out with an understanding, I also know I hate getting bogged down in the detail, I really don’t like it when I feel like I’m learning all that can be learnt about a subject so I tend to start to switch off if I sense the end is in sight. So being in IT is good for me I get to play with all sorts of interesting stuff and I get to do it to an interim level and I never have to be considered an expert in anything, if only. Some times you have to do things that are not interesting, things that are not what you find enjoyable about your work, but work is work, the good with the bad, the trick is to minimise that work so you spend more time doing what you like.

As times goes on you can start to realise that there’s many elements of even the things you like doing that are not right for you. I hate documenting, I hate long drawn out problems, I hate tickets so what do I do to stop myself doing them? Well with documentation, I relaised that I hate spending hours in one chunk writing something that is useless, but I don’t mind throwing a structured wiki together and just writing small snippets of docs, so my documentation is now light not heavy, so it’s not that i hate documentation it’s more that i hated spending all that time in one go on it. Same principle with long drawn out problems, I tend to chunk them into smaller more manageable pieces and do a little it it often.

Summary

While doing all the things you love there will of course be bits you don’t love doing but still need to do, the trick is to find ways of doing them that make it more bearable. Try and avoid the temptation of just offloading the work to others, it doesn’t make you look good, and it is good to keep your hand in.

Also while in the job you love, don’t be afraid of looking for other jobs with different challenges or opportunities, change is good and helps us provide more value to an organisation. This doesn’t mean you need to look outside of your current organisation but by being a little astute with your conversations in work you may find other opportunities arise and help you out.

Also don’t be afraid of working out what you actually love, so I started out in IT because I love the technology and the playing with things, but as I’ve been in the environment for a while thats tuff is interesting but not so much a passion, I have realised over the years that I prefer helping people, Mentoring people planning projects etc, and i’m sure when i start doing those things more I’ll find other things that I like doing and I will always move towards doing more of the stuff I Love doing and less of the stuff I dislike, it’s better for me and it’s better for the organisation.

Self healing systems

An odd beginning

So I’m writing this having just spent the last 10 days on my back in pain and finally starting to feel better, it’s not come at a good time as another member of the same team as me decided they had a “better opportunity” This is the second person to have left this organisation without as much as a passing comment to myself that they were even leaving, how strage; but I digress.

Either way it opens up a void, a team of 2 and a manager now down to a team of one, with the one having back pain that could at any moment take me out of action. Unfortunately up to the day before I was not able to make it to work the system we look after has been surprisingly stable, rock like in-fact; as soon as I say “I’m not going to make it in” the system starts having application issues (JVM crashes).

Obviously the cause needs a bit of looking into and a proper fix etc etc, but in the mean time what do we do? I had an idea, A crazy idea which I don’t think is a fix to any problems but it at least a starting point.

Sentinel

I have spent a bit of time exploring Ruby a few weeks back so I started to look at ways of writing something that would do a simple check; is process X running? In the simple version I wrote it just checked that tomcat was running more than one instance (our application always runs 2) if it was 2, do nothing, if it was more than 2 do nothing (something else crazy has happened so it just logs to that affect) but if it was less than 2 it would try a graceful-ish restart of the service.

So this obviously works in the one specific case that we have, but isn’t extensible and it doesn’t do any better checks, which all got me thinking. Why isn’t there something to do this for us? I don’t know of anything that does this, if anyone does I’d appreciate knowing, there’s a number tools that could be muddled together to do the same sort of function.

Nagios would monitor the system, cucumber could monitor the application interactions, Swatch could monitor the logs, but in most cases these are monitoring, I’m sure there’s ways to get them to carry out actions based on certain outcomes but why use so many tools.

Yes, the individual tools probably do the job better than a single tool, but as a sysadmin, I’d rather have one tool to do everything but that isn’t practical either. So can we some how get the benefits of monitoring with nagios but have a tool that is specifically monitoring the application performance nagios is gathering information about and then making decisions based on that?

The big Idea

So I wonder if it’d be possible to write a simple ruby application that every now and then did a number of actions:

1. Check the service on the box, right number of processes, not zombied etc, etc
2. Check the disk capacities
3. Check the CPU utilisation
4. Check the memory utilisation
5. Probe the application from the local box, a loopback test of sorts
6. Integrate with nagios or another monitoring tool to validate the state it thinks the box is in compared witht he locally gathered stats
7. Depending on the outcome of all the checks carry out a number of actions
8. Hooks int ticketing systems

When I was thinking this through the other day, it seemed like a good idea, the biggest issue I have is not being a programmer, So I have a steep learning curve, it’s a complicated application, so requires some thought. I would also probably have to ignore everyone that thinks it is a waste of time, which isn’t too hard to do.

I guess what I’m thinking of is something like FBAR. As a system scales up to hundreds of servers the up time and reliability becomes more important, it is sometimes necessary to take a short term view to keep a system working. The most important thing is that those short term views are totaled up and then logged as tickets, 1% of your severs crashing and needing a restart isn’t an issue, but if that 1% becomes 5% and then 10% it’s panic stations!

Summary

I think my mind is made up, a sentinel is needed to keep watch over a solution, and what’s crazy is that the more I think of it the more useful it seems and the more complicated it seems to become. As such I think I’m going to need help!