Helping others

A while back

I was volunteered to help our finance team customise their accounting system which finally I’ve got to the end, or at lest the end is now in sight. It was an interesting situation, a few years back the accounting use to be done in Access and a couple of years back they migrated to a solution provided by Netsuite. I have to say, when I started I spent ages complaining about how awkward and painful it was to do anything in Netsuite, but as I’ve done the journey I’ve come to understand a bit more and it’s not bad, it’s still massively complicated but it is at least not that bad!

One of the most frustrating things I found was using web services, I’d never done it before and on paper it sounded like the best solution, but it was a nightmare, the most annoying thing is the provided wsdl had errors in it, so I ended up having to spend some time finding the bad lines and fixing it from within vi thanks to this link. Needless to say after a few weeks of struggle I started down the right path which was to use more of Netsuite to do more and to write a simpler tool to do the rest.

Time for a rest

I found the rest-let’s which was a big help, use their documentation to work out roughly what to do and then hook it in and hope for the best. Its worth noting the Documentation is excessive and hard to follow, makes you wonder why they bother. Either way After writing that then spending some time trying to get that up and working it was there, just needed the other 7 pieces of the puzzle to make it work, Luckily you can call searches remotely to get a list of results and then manipulate them and send them back. I came up (with a lot of help from the finance team) a search that only returned entries I hadn’t yet changed and then took the details from that and manipulate it into the format I needed to post back to my rest-let to do the updating.

After sorting out the entry of data I then had to just work out how to invalidate the entry if anything was changed, this was harder, it took a bit of reading and then it was clear what to do, just impossible to find out how to do it! I must have spent more time on this project working out how the system worked and just using the tool rather than coding.

Bigger and better things

This initial step was an important one it sets the foundation for a raft of other changes that will hopefully be simpler and easier to put in place, but like anything unknown, we won’t know until we know. I’m looking forward to taking a system that already provides value and making it provide more, with reporting and customisations, hopefully over the coming weeks it progress to produce more useful information.

Summary

In short, Help people, sometimes it starts off being a pain but at least you help someone and you also learn something, yes it may lead to a flurry of other bits but, ask your self this. If you needed help, would you want someone to help you?

Let’s Chat

It’s good to talk

Over the last few weeks I’ve been playing with some forum based technology, trying to find an ideal platform to take and add lots of modifications too to integrate fully with another app. The idea is that in said other app you will click “help” and it will take your code and post it directly to a forum, where people can offer up code solutions which you as the original poster can click the “try it” button that then puts it back in the other app and allows you to test it against your test systems.

There’s a whole other bunch of features and gamification that needs to happen to help build up a desire to help people with rewards for those that help the most, needless to say there will be a lot of customisations so I spent a little bit of time playing around with a few technologies and the one we’re going to use is NodeBB. It’s also worth mentioning they have a nice indiegogo campaign going and more support is required!

Everyone we’ve shown the forum to likes it, it is still quite immature, they only started in mid May (2013) and the code has come a long way, and it is continuously improving. With help from me ;)

#NodeBB got it's first pull request today, from @soimafreak ! Guess that makes us a "real" #OpenSource product now… #github #nodejs #redis

— NodeBB (@NodeBB) June 27, 2013

Building a stable platform for the future

I have some constraints around what I’m doing, one is I’m still new to Javascript, learning, but new; luckily I have people I can bug if I get stuck which really helpful. I also have to make the discussion boards easy to support for the people that will be looking after it and easy to integrate it with our exiting app as and when needed.

The next biggest issue with building a solid platform is a good choice of frameworks, using a frame work like PassportJS rather than writing individual login methods, implementing configurable logging with something like Winston and using templating solutions like Dust (Dust seems really god but not maintained any more, I guess it’s perfect?) or Jade (Jade is only server side but ClientJade fills that niche)

All of these frameworks will just make my life easier in the log run, I’ll hopefully be able to work quicker and it should give other sone thing less to learn assuming I choose libraries they are familiar with anyway.

So there’s a few things that need to be looked at from a supportability point of view and to enable easier development by us and then there’s all the extras we need to work on, because of the level of integration and needing to poke almost all elements it will be interesting to see how the plugins will work for NodeBB. I’m desperately trying to work out how I can maintain heavy customisation of the code and still contribute back shared goals, I’d prefer not to fork and become so separated that we can’t push back contributions, hopefully we’ll be able to work out how we do that!

What challenges you?

Over the last few weeks

I have been wondering what most people find challenging in the “modern” IT world. There’s been a recent upsurge in tools and technology that address most problems which only leaves me to wonder what is filling that gap? What is the current big annoying problem, maybe it’s not being able to push your architecture into multiple clouds, or having to live with the constraints of small root disk volumes; Who knows? Hence the poll :)

Configuration management alone is not the answer

Everything in one place

Normally when businesses start out building s product, especially those that don’t have the pre-existing knowledge of configuration management, tend to just throw the config on the server and then forget what it is. This is all fine, it’s a way of life and progression and sometime just bashing it out could prove very valuable indeed, but typically this becomes a nightmare to manage. Very quickly when there is then 100 servers all manually built it’s a pain in the arse so then everyone jumps into configuration management.

This is sort of phase 1, everything has become too complicated to manage, no one knows what settings are on what boxes and more time is spent working out if box 1 is the same as box 2. This leads to the need to have some consistency which leads to configuration management, the sensible approach is to move an application at a time into configuration management fully, not just the configuration files.

During this phase of execution it is critical to be pedantic and get as much as possible into configuration management, if you only do certain components there will always be the question of does X affect Y which isn’t in configuration management? and quite frankly, every time you have that conversation a sysadmin dies due to embarrassment.

Reduce & Reuse

After getting to Phase 1, probably in a hack and slash way, the same problems that caused the need for Phase 1 happen. 100 servers in configuration management lots of environments with variables set in them, and servers, and in the manifests themselves and the question starts to be come well is that variable overriding that one, why is there settings for var X in 5 places, which one wins? Granted in configuration management systems there are hierarchies that determine what takes precedence but that requires someone to always look through multiple definitions. On top of having the variables set in multiple locations, it is probably becoming clear that more variables are needed, more logic is needed, what was once a sensible default is now crazy.

This is where phase 2 comes in, aim to move 80%+ of each configuration into variables, have chunks of configuration turned on or off through key variables being set and set sensible defaults inside a module/cookbook. This is half of phase 2, the second half and probably the more important side is to reduce the definitions of the systems down to as few as possible. Back in the day, we use to have a server manifest, an environment manifest and a role manifest each of these set different variables in different places, how do you make sure that your 5 web servers in prod have the same config as the 5 in staging? that’s 14 manifests! why not have 1? just define a role and set the variables appropriately, this can then contain the sensible defaults for that role, all other variables would need to be externalised in something like hiera, or you would need to push them into Facter / ohai.

By taking this approach to minimising the definitions of what a server should be and reducing it down to one you are able to reuse the same configuration so all of your roleX servers are now identical except what ever variables are set in your external data store which can now easily be diff’d.

build, don’t configure

By this point, phase 1 & 2 are done, all is well with the world but still there’s some oddities Box X has a patch level y and box A has a patch level z, or there’s some left over hack to solve a prod issue which causes a problem on one of the servers. Well treat your servers as configurable and throw-away-able, There’s many technologies to help with this be it cloud based with Amazon and OpenStack or maybe VMWare, even physical servers with cobbler. This is Phase 3, build everything from scratch every time, at this point the consistency of the environment is pretty good leaving only the data in each environment to contend with.

Summary

Try and treat configuration management as something more than just config files on servers and be persistent about making everything as simple as possible while trying to get everything into it. If you’re only going to manage the files you might as well use tar’s and if that sounds crazy it’s the same level as phase 1 which is why you have to get everything in and I realise it can seem a massive task but start with the application stack you’re running and then cherry pick the modules/cookbooks that already exist for the main OS components like ntp, ssh etc

To IDE or not

What’s in an IDE!

Over the last 15 years or so I have used a few IDE’s for programming “proper” applications in Pascal, Delphie, C/C++ or Java but since moving to the darker side of computing, the linux world I never really needed an IDE and they were just bloated text editors. I don’t think much has changed but there are some good points and I’m at a cross roads as to what to do.

Should I spend some time making Vim, the best text editor the world will ever see better or give up and use an IDE, it’s not as straight forward as I would like to think or as anyone else would probably think so I wonder how it will turn out.

The basics

I’m currently, and probably most likely to be doing web based development and Ruby / python. Up until recently I had just used Vim to get any Ruby programming I needed done, and to be honest it worked okay. I typically just opened a new terminal tab and then changed to the repo directory and started opening files, So I had to cd to a directory before opening the file it wasn’t killing me. I was able to just open and get on with making changes with no real hassle and with syntax high lighting it worked out quite well. Recently I have started to do more web based things and have started using an IDE for that and that’s what’s lead to this blog, there’s a few features of an IDE that I think I’m missing from Vim.

1, Being able to see the entire project directory easily and visually navigating to supporting files to have a quick view / edit is nice.
2, Refactoring, simple things like changing a variable name can be a real bugger in Vim, yes you can find and replace but that is not the same as refactoring, to refactor like an IDE does Vim would need to understand to some degree the codes structure and not to treat it all like text
3, Hiding chunks of code, being able to rol up / down sections of code can make it a lot easier to focus on what’s actually going on
4, Word completion, IDE’s know about the libs being used and the other variabels so can offer hints to complete the words being typed which is very handy
5, Short hand, being able to type short hand to do lots of things, like starting a html tag and having it auto completed speeds up development which must be good

I think if I could get the above features in Vim I probably would struggle to justify not using Vim, lets see what happens.

Vim IDE

1, This is seemingly an easy oen to address with something like NERD tree Initially this looks good so I’m saying point 1 is solved, just need some better aliases (:help 40.2)

2, In short, no, I’ve only seen normal find / replace or basically using grep, There does seem to be language specific ones like This but who wants to find a new refactoring plugin (if someone wrote it) for every new language…

3, This is again easy if you know what to look for Folding is not new and from my quick play seems good.

4, Again language specific it seems possible: this one for ruby, but what about python? javascript? etc etc

5, This one is a simple one that vim does do: Here but it’s almost like you have to hand crank it all!

Summary

So from what I can see I can spend some time making Vim do what I need, and no doubt there’s some stuff there for Ruby and Python that would make it a lot more useful to use as an IDE but it still doesn’t make it easy and will be rather language specific in terms of making it work well out of the box. It does seem that IDE’s are better at those specific tasks above, not to mention debugging and break points etc, maybe Vim can do more of that too and I just don’t know how.

Either way it coped better with the 5 original points than I thought it would and I have found some things out that suggest it may be worth tweaking my current vim to make it better. In the mean time I think I will continue to persevere with learning the IDE and it’s short cuts as it appears that it could dramatically speed up my development by just simply using the tools already there.

I wasn’t expecting to find vi to be as good as it was but in my head even though I prefer Vim at the moment I think the IDE will cope better in the long run, time will tell.

Angular.js – starting something new

Why not!?

Last week I played with Python / Django the natural extension of this was to not do python but to look at using something like Angular and bootstrap, the main reason for this is that just using python / django gives you server side manipulation but why do it all on the server? Ideally the server would generate relavent data and sent the data back to the client for processing, this is how the internet is meant to work after all. Think about simple HTML and CSS, HTML is just a data structure and CSS formats that to make it look pretty for you, now, with modern computers being so powerful, why not have it do the formatting in CSS and also manipulate the data into the right structure.
This way the server just generates a response of say JSON and the client takes this and manipulates it into tables / lists or what ever you need. Another advantage to this approach is all of a sudden you have an ajax driven website with not too much effort which is more efficient on bandwidth, combined with local client side caching of information the user experience should be pretty quick and seamless with or with ut the server, as long a sit pulls the dat to start with it should be okay.

So that covers why Angular, Bootstrap on the other hand is more or less just CSS but quite frankly, I can’t do CSS it never looks right so by using bootstrap I can then programatically set out hierarchy and classes etc while leaving the pretty styling to a theme generator Like this or download one Like this

Tasklist

Okay this is not really fully featured.. You can’t delete stuff from the list it just gets bigger and bigger but was only done to use the combination of applications in the right way and see if it worked, well it works, you can add tasks and it displays them so lets have a look at how pretty it is:

I was only able to achieve this level of prettyness because of bootstrap my past sites have all looked terrible…
The home page just includes some example code from the AngularJS website to prove I managed to get Angular working before butchering it with my typos.

Here’s a picture of it listing the tasks…

So for starters if you’d like to criticise the code the best way of doing that is on GitHub I have no intention of making it any better but if you want to play with Angular, Mongo, Python and Django where python/Django provides a rest api to the DB and Bootsrap/AngularJS provides the front end then this has it all and the set up is pretty straight forward and some brief notes on install / settings in the README. Maybe you could add the ability to edit or delete tasks maybe put a checkbox next to them and track a done status who knows…

Overall…

I quite like Angular and bootstrap as a combo, I think it’s got some real potential to do some powerful stuff and i’m sure people out there are doing some really crazy things with it, hopefully one day that will be me and there will be something worth while coming out of it. I’m not convinced on python/django combination, I’m sure it’s okay but I really need to write some more of it and see how it goes; I’m fairly sure it’ll be better than RoR but that’s not a good framework for ruby anyway.

My next task having played with those is to do something even more whacky, maybe some sort of node.js backend, angular boot strap front end and some sort of queue… The next project I’m working on with a larger team will be pretty good and once we have something up and working it can be shared, the oddest thing will be the “button driven development” approach to development, which is basically top down, where you prototype the interface to look as you want it to and then fill the data with dummy values or hard coded values. Once that’s done write a backend that provides the data. In theory by the time you get to do the technical nitty gritty there’s a very clear idea on what the output should be.

Python – A Django experience

For a little while

The team here have been looking at rapid development of web applications and have been trying python with Django to deliver some nice projects in a short period of time and with good results thus far. Up until the other day I really hadn’t done anything with Python and I had no intention to, but as these little skunk works projects have come to an end and the results look okay I figured it was worth learning a little.

Now, there’s always two approaches to this, start slow, learn python, learn some django and keep building on it until it all works and is fantastic or just dive in, I chose to dive in. The downside of diving in is you end up with some stupid errors where you don’t know the syntax, how the frame work works what should be done etc etc so you spend a lot of time googling! The advantage of this approach is you tend to learn more quicker, especially if you’re like me and struggle to read the doc and then apply, i’m only good with example sand then making mistakes, but with examples you’re on auto pilot.

So I had a bit of a chat today about the structure of Django where to put certain things and it was suggested to do something simple like a task list so that’s what I did, I made a simple task list, I decided to complicated it by using mongo as the database which the only example I had of it did not make use of the models in Django but tbh, I dived in i’m not sure what i was meant to to do there anyway… Although…. I do know that with MySQL I can use the models to outline the database structure and have the manage.py create the db for me, but alas I chose a harder route.

Some bad code

My view’s looks a little like this, I had started out with good intentions of using csfr but it was causing problems so I ended up removing it, in fact I removed quite a bit to simplify this to make it easier to debug, and it worked as this now works…

# Create your views here.
from pymongo import Connection
from django import forms
from django.shortcuts import render_to_response
from django.template import RequestContext
from django.http import HttpResponse, HttpResponseRedirect
from django.core.urlresolvers import reverse


def add_task(request):
    return render_to_response('add.html')

def add(request):
    if request.POST['task'] and request.POST['detail']:
        databaseName = "test"
        connection = Connection()

        db = connection[databaseName]
        todo = db['tasks']
        task = {"task" : request.POST['task'], "detail" : request.POST['detail']}

        todo.save(task)
        
        return HttpResponseRedirect(reverse('task.views.list'))

def list(response):
    databaseName = "test"
    connection = Connection()

    db = connection[databaseName]
    todo = db['tasks']
    tasks = []
    for task in todo.find():
        tasks.append({ 'task': task['task'], 'detail': task['detail']})
        #tasks.append(task['task'])
    print tasks
    return render_to_response('list.html', {"tasks":tasks})

The template for the add is very simple, it’s just a html form really not templating involved and it posts to the add method (which may have been the issue with the csfr stuff)

<html>
<head>Test</head>
<body>

    <p>Testing django</p>

    <h1>list</h1>
    <table align="centre">
    <th>
        <td>Task</td><td>Details</td>
    </th>
    {% for task in tasks %}
    <tr>
        <td>
            {{ task.task }}
        </td>
        <td>
            {{ task.detail }}
        </td>
    </tr>
    {% endfor %}
    </table>

</body>
</html>

As you can see the template is straight forward, the bigest issue I had with the template is I was generating a json string so

tasks.append("{ 'task': '"+task['task']+"', 'detail': '"+task['detail']+"'}")

Although this looked okay it was really a problem, either way, so far from what I can tell Django is okay, python isn’t too bad and I will do some more over the coming weeks to help me understand it better but it seems okay, nothing to sing about just yet though.

Automate to survive

Everyone has a choice

Automate or die, That is pretty much it, you can automate everything or you can keep working with manual processes that slow you down. If you don’t think you have the time to automate, you’re wrong; you need to automate and do it quickly before you get even more busy and even further behind. Maybe you think that you can’t automate because you don’t have the time to do it justice? maybe you can’t automate because the task is to big? Too complicated? well it’s all rubbish.

Start small

This is a bit like eating an elephant, You have to start somewhere, you have to start small, by all means try and start big if you want, but smaller is better. Maybe you have a task to check for new packages from a site once a month, that is a good place to start, pull third party packages from vendors sites into your yum repo or maybe every time you build a server you need to do x, y & z. These sort of tasks are achievable for everyone even those without a good background of programming background which leads on to language choice, not all are equal but knowing two or three is better than just one. At a minimum some sort of terminal language, so Bash,ash,sh or ksh and a ‘Proper’ sort of language that is object orientated like, Ruby, Python or Perl. The terminal languages are good for re-producing what you do on a terminal into a reproducible and consistant format but are terrible for manipulating multiple data sources, mangling data, although with that said you can do some complicated things.

Once you start building up many smaller components of automation start looking at ways of linking it all together so that a series of tasks becomes one. It is this constant cycle of simplifying the process to automate the small chunks and then linking the small chunks together that make an automated system.

Grow large

Over a year ago we use to deploy our environments with Puppet and cloud formations and it use to take about 2.5 days to complete and get it working, that whole process is now down to 10 mins thanks to automation. It required many leaps of faith, many poor decisions and a lot of bug fixing but it got there though simplifying everything down and then automating each component. Other than building the servers and tagging them with appropriate keys in Amazon the whole process is controlled by bash to the point of a working system and is typically very robust. That is a massive time saving, but to get there we had to fail, we had to try and we had to persist.

As a result we now automate large portions of the architecture to a point where all of our time is split between incidents or project work to implement new features hardly any daily grind. Recently I have been working on our DR strategy to take it to the point of clicking a button to deploy a clean environment built from the ground up and automatically pulls the latest backups to restore to the environment but it is now done and saves hours of time building out a DR which makes the recovery time shorter and the process is easier. So larger projects are perfectly achievable with the right attitude!

Summary

Give it a go, start small and work up to it but be un-relenting and do what ever it takes, no matter how much you disagree with it, just do it to get it automated, once more is automated you’ll have some time to fix it up properly or you’ll need to extend it and you can make a small part better then.

Oh no, not java

How strange

Over the past few months I’ve been writing more and more applications to help maintain and deliver the services we run, from metric gathering to regional dr and anything in the middle. For A while now one of the developers at Alfresco has been writing a framework that makes it easier to write selenium tests for Alfresco share which takes a lot of the hassle out of looking for certain elements or class id’s or updating everything if the UI changes. So we have been talking about it for a couple of months and today I decided to get some time to look at it and ask loads of silly questions about eclipse and maven and so on and so forth.

It took about 3 hours to get everything set up and working, most of the time was just spent learning to use eclipse and maven with a walk through of what it can do, how to extend it and how to do stuff with it. Considering I hadn’t done any Java for 6 years it wasn’t that bad and within 15 mins of being left to it I had made a class that loged in and searched for content inside the repo.

One of the reasons we’re so interested in it is because as DevOps we like simple things and it takes a lot of the hassle out, it means we also get to do some complicated things with Share and we only have to worry about what we want to test or measure. All of this got me thinking about the languages we use and the problems they solve.

Right tool, Right job

Currently in our team we are using bash, ruby, python and java. Bash is simple and can achieve some good results although typically quite slow, typically if it is a short script it will end up in bash, although we do our orchestration in bash and it manages the bear metal to working OS by triggering what ever apps we need or setting config.
Ruby is the language of choice for me when I need to do something that requires data to be manipulated or retrying actions or anything that is more than procedural and you can rely on it to do a good job in a reasonable time.
Python is new to the team, it feels a gap which is that it’s as easy to write as Ruby but is more scalable at size, I haven’t done any python yet so I can’t really comment but the web app that has been built with it in a couple of weeks is quite impressive. Java is more complicated harder to write but can offer more complex apps, but typically I’m not sure that you need to make apps that complex.

So I’m not a fan of Java, but mainly because I think it takes a long time to get anything of any value out of it, especially on a small task. If I had to write an application to manage backups I would not go to Java as it’s like using a bazooka to hit a fly, likewise using Bash is like using a feather duster where as ruby and python fit nicely in the middle. Well after todays experience I’m glad I’m doing it in Java, I would have spent weeks making something half as good in Ruby to just avoid using Java and I guess it’s not really that bad.

I could have wasted time doing it all from scratch or just take what’s already written, so I stole like any good DevOps guy would.

Summary

I’m probably going to spend some more time in Java over the next week writing something a bit more useful than todays experiment so hopefully I will still be optimistic about it all, and maybe I’ll remember why I don’t like Java or maybe I’ll change my opinion, who knows!

You have to stick with these things

Hang in there

Just over a year ago I decided that I had, at best a mediocre online presence, Sure if you search for my handle all sorts of stuff turns up and most of it is me but in this age where the internet is ever lasting I didn’t want my previous 10 years of internet to be the defining pattern I left on the world. So with that I decided to annoying people, originally I planned to do two blogs a week, one techy and one non-techy, well quite frankly that’s hard work but it lasted for several months, now I have a much more leisurely one a week post and that seems to be working out better.

When I first started I remember being relatively happy with a small number of users visiting, well over the last year I have grown my blogging empire quit significantly and the best of all of this is statistics, it is nicely measurable. Having that feedback on what articles work and which ones don’t is handy, it doesn’t stop me writing the ones no one likes but at least i’m aware they will be less liked before I write them.

Statistics

Here’s some numbers to make things more interesting

Visits per month

Month	visitors
Feb 2012	132
Mar 2012	167
Apr 2012	167
May 2012	284
Jun 2012	387
Jul 2012	407
Aug 2012	460
Sep 2012	491
Oct 2012	728
Nov 2012	1323
Dec 2012	1115
Jan 2013	1572

Up until September I was thinking how the progress was slow but steady, a little disappointed, and then bang, much better! I was talking to my boss a couple of months back and he was mentioning how google seem to sit you in a sandbox for a few months before they really let things go, and that’s sort of what it seems like here, I didn’t write any killer articles that all of a sudden had a spurt, I may have got a link back from Alfresco.com but I didn’t know that, until November, my boss decided to tell everyone, oh well better than employing a marketing person :)

I now hover around 300 visitors a week to the site which is still not a large number but if only 1 percent of those actualy reads an article it maks it worth while!

One of my articles rhn satellite vs puppet, a clear victory? has almost managed 200 views by it’s self. but there’s a few others in there doing okay, and some I thought would do better, so here’s a chance to show some of those that I thought would do better.

The 5 posts that should have done better!

1. Self healing systems
2. What university forgot to mention about programming
3. Cloud deployment 101 – Part3
4. Cloud deployment 101 – Part1
5. Who burnt down the building?

There are quite a few more stats that could be shared but they are not that interesting really… so back to the point of why I started the blog, to leave a bit more of an impression well I think I have done that, I have some godo referrals that make it back to me, here’s a couple that always surprised me

• savannah.gnu.org
• Puppet labs

And one that came up to day is me being quoted! Here. So when I look back over the year and I see what has been achieved, despite a slow start I’m glad I stuck it out, hopefully when I’m doing this next year I’m talking more about the news article I had to write or tv show I appeared on… well We still need to have dreams else what’s the point!

So I wonder what next year will bring if I keep on blogging, and you the people keep on reading, as let’s face it if it wasn’t for you people I would just be watching tv or something, so Thanks!