AWS best practice – Introducing Amazon

Introducing Amazon

Last week I introduced the Cloud, if you missed it and feel the need to have a read you can find it Here Now on with Introducing Amazon…

I’m not really going to introduce all of Amazon, Amazon release a lot of new features each month but I will take you though some of the basics that Amazon offer so when you’re next confronted with them it is not a confusing list of terms, I won’t go into any of the issues you may face as that is a later topic.

EC2 Elastic Compute Cloud, this is more than likely your entry point, it is in short a virtual platform to provide you an OS on, they come in various shapes and sizes and different flavours. For more information on EC2 click here

ELB Elastic Load Balancer, this is used to balance web traffic or tcp traffic depending on which type you get (layer 7 or Layer 4) an ELB is typically used to front your web servers that are in different Availability Zones (AZ) and they can do SSL termination.

Security Groups These are quite simply containers that your EC2 instances live in and you can apply security rules to them. However, two instances in the same security group will not be able to talk to each other unless you have specifically allowed them to do so in the security group. It is this functionality that separates a security group from a being considered a network, that and the fact each instance is in a different subnet.

EIP Elastic IP, These are public IP addresses that are static and can be assigned to an individual EC2 Instance, they are ideal for public DNS to point to.

EBS Elastic Block Storage, In short, a disk array attached to your EC2 Instance. EBS volumes are persistant disk stores, most EC2 instances are EBS Backed and are therefore persistant. However, you can mount ephemeral disk drives that are local storage on the virtual host, these disk stores are non-persistant so if you stop / start an instance the data will be lost (they will survive a reboot)

S3 Simple Storage Service, S3 is a simple key value store, but one that can contain keys that are folders, and the value can be anything, text files, word docs, ISO’s, html pages etc. You can use S3 as a simple web hosting service if you just upload all of your html to it and make it public. You can also push S3 data into a CDN (Cloud Front). There are some nice security options around accessibility permissions and at rest encryption for your S3 buckets. An s3 bucket is just the term to describe where your data ends up and is the name of the S3 area you create.

IAM Identity and Access Management, This is a very useful service that will allow you to take your original account you used to signed up to amazon with and lock it away for eternity. You can use IAM to create individual accounts for users or services and create groups to contain the users in, with users and groups you can sue JSON to create security policies that grant the user or group specific access to specific services in specific ways.

VPC Virtual private Cloud, This is more or less the same service you get via EC2 but private. There are some interesting elements of it that are quirky to say the least, but you can create your own networks making your services private from the greater amazon network but you can still assign EIP’s if you so wish. Most services, but not all are available with a VPC and some features are only available in VPC’s such as security groups on ELB’s.

AZ Availability Zones, are essentially data halls, or areas of racks that have independent cooling and power but are not geographically disperse. i.e. an AZ can be in the same building as another. Amazons description is as follows “Availability Zones are distinct locations that are engineered to be insulated from failures in other Availability Zones and provide inexpensive, low latency network connectivity to other Availability Zones in the same Region” This will be touche don later.

Region A region is a geographically disperse amazon location, it could be in another country, it could be in the same country, I’d imagine that all are at least 30 miles apart but amazon are so secretive about everything it could be that building behind you.

If you want to know more about the products I would read the product page here. In next weeks post I’m going to start going into a bit of detail about architecting for the cloud and some design considerations that you should be aware of.

Not putting in the time and effort

Last week…

I realised I basically was running out of ways to say “always put in extra time and effort when you can, it will reward you”, now if that is too concise you can read the full blog here. As a result I thought I’d drag it out to a whole new post about not putting in the time and effort, after all your time is valuable.

How much time do you have?

Really, how much? I use to think I had lots of time, then I realised that I don’t have any time really, the only thing that changed was I got in a relationship and it was more important to be doing other things than sitting at work late into the night or watching tv all the time.

I then started to realise that something had to give, I could keep working or I could retain a relationship, I chose the relationship. I then realised that I actually had more time than I originally thought, I was just wasting hours each week, I like watching tv like the rest of us, but I did this amazing thing that I believe Women take for granted, I started to multi task, only ever giving half of my attention anything, but quite frankly, it’s tv…

So, again, how much time do you have? probably more than you realise.

where to spend your time and effort

So as hopefully we all came to the same conclusion, you can probably find time to do some things, certainly you can find time to do important things. As with last week I was touching on using your time smartly then but to invest the time and effort into improving your career, well what happens if you chose to take that drive and energy and apply it internally? So rather than just looking at your career, look at making time to spend on doing things you want to.

This year I decided that I would do a few things, one, maintain a blog, which I hadn’t done for a long time and to make things worse I thought writing two articles a week was a good idea, based purely on the fact that someones articles I read on the register writes two a week. So In addition to the blogging I took on number two, create an open source project, I had always wanted to do one / contribute to one and I didn’t ever want to ruin a proper project so just did my own.

If there’s something you’ve always wanted to do then why don’t you do it ? You don’t have to slave away at work if there’s no benefit to it, especially if you could spend that time doing something else. I’m not saying these projects will progress rapidly but you will be able to progress them now and then. What more do you want?

So other than being distracted…

The projects can still benefit you career wise, if nothing else it’s a talking point to discuss with people and show that you do put in the time and effort into projects, work related or not.

Let us not forget that a lot of businesses start off as a project for fun and enjoyment and not purely a business venture, Google was a research project called BackRub Facebook was pretty much created to prove a point that it could be done quicker and simpler by Mark than allowing harvard to take 2 or 3 years over it ( Here ). I could go on, but in short the best companies around are created out of a need / desire to do something different and outside of the norm, all squeezed into those little chunks of time that you currently don’t have.

Summary

You have the time, you just need to put the effort in.

AWS best practices – Introducing cloud

Overview

With this series of posts over the next few weeks I am attempting to help those new to Amazon Web Services (AWS) get a step up and to help you avoid some of the pitfalls that I have encountered, the sort of guide I would have been biting peoples hands off for when I was in the same boat, but before we go any further, a picture, people like pictures, you are people so here’s a picture.

It’s a picture of a tree trunk which over the years has been embedded with coins, people walk by, they see that someone has pushed a coin in, so they do the same, rinse and repeat for 10 years or so and you end up with several trees like this one. This is essentially what the Cloud is to people, 5 years ago, no one knew what the cloud was, no one cared, 3 years ago, people said “Hey, look at this”, 2 years ago people said the cloud was going to change the world, a year ago people said big business was adopting the cloud and today I tell you not to with out reading through this.

Although I am going to focus AWS the topics covered are more than likely relevant to other cloud providers and I would encourage you to read through this to cement the foibles of the largest cloud provider Amazon, so you can better understand the contraints they place on you and those of other cloud providers may place on you. Now on with the post.

What is the Cloud

Apologies for the history lesson, feel free to breeze over…

In the beginning there was only one way, build a data centre, source your own power, cooling and network and start building out a data centre full of disk arrays, high performance servers and networking equipment, I would label these the “Golden age” but truth be told running your own data centre from the ground up can not be easy.

As with everything progression; some smart people noticed an opportunity and started to take over the management so all you had to do was turn up with your server, disk arrays, and networking, this is co-location and is a good way of doing things, this is not as cheap as doing your own but takes a lot of hassle out of it.

Leading on from this companies began to form that went one step further, they would provide your equipment for you so all you have to do now is login, all the disk worries, network were taken care of and they would help you on your way, of course charging a premium for the service. Moving on from this but in the same area of hosted services are the almost fully managed solutions where they do everything, you give them an application and they make it happen, great if you don’t have an IT team.

So getting onto more recent times, virtualisation has really taken off in the last 10 years despite being around for longer, I believe the big drive for this was after the “.com bubble” burst back in the early 2000’s and companies were looking for ways to save costs on their hosted or co-location services. One of the ways this was achieved was through virtualisation such as Xen and VMWare. In most cases the equipment was run and managed by the company and people over allocated or under allocated memory and CPU depending on their use profiles with all sorts of redundancy.

As you can see from all of this the constant push is to reduce costs, granted running your own data centre is the cheapest way, but you will need a few thousand servers to make that so. Faced with a problem a company called Amazon, who you have probably heard of, they run a web shop by the way; noticed that even with all of their virtualisation technology they still needed a large percentage of servers just for 2-3 weeks of their business each year, the rest of the time te boxes sat idle; but what are you to do, you have to have the capacity for your peaks. Well they worked it out and we have the cloud, i’m not 100% sure if they had the big idea but they certainly took the idea and ran with it.

The idea behind Cloud computing is a utility based cost $X per hour, this comes across as a very cheap modle but as we’ll find out later in the following posts it’s not that cheap and it depends heavily on your use modle. With the Cloud you now have the ability to choose how much disk you want and for how long and how much CPU time you need. This is the joys of Cloud computing.

Summary

That was a rather long introduction to the cloud but with this understanding of the history behind it and how it was born you will now hopefully appreciate where it is going. I wouldn’t be surprised if most of the the features Amazon release are just new ways of them making better use of their own applications and architecture and then working out how to do that more times to cover the costs and offer it as a service.

Putting in time and effort

Every now and then…

A decision needs to be made, do you value your time or your effort? I know when I first started out in work I had little choice but to put in the hours. At some point we all have to just put in the hours for one reason or another. For me putting in the hours was about working 30 hours a week and doing a full time college / university course and still finding time to educate myself out side of college or university. During those days I had time on my hands I didn’t have a CV full of skills or jobs that made me look attractive to employers so I had to put in hours to make myself achieve what I wanted to and to ensure I could continue achieving more.

Occasionally there will be those moments, rare moments where you look at something that is working, no real issues with it and you realise it’s not “right”; you identify some issue or room for improvement and you give it some thought come up with a few ideas and then just implement the best one. This is putting in some effort, there was nothing really wrong but you decided it needed to be better, it needed to be improved some how and then it was done.

When I talk about effort I am not talking about slogging away for hours at a time I’m talking about people thinking about improving a situation on their own. It is incredibly easy to just sit there and solve problems that are presented, it’s something different to try to solve problems that no one is screaming about or worrying about but provide huge benefits to your daily work.

When to use each

Let’s start by clearing up something, you as an individual work for a company to earn money, the company works for the share holders to get them money. At no point is the company working for you, sure companies will pay you a salary and throw bonuses at you if you successfully navigate some hoops and give you a raft of benefits on the side. Now going back to earlier, at which point was any of this putting in extra time or effort to make you happy? give you better experiences or to increase your value.

Bear that in mind, I will probably contradict it in a moment.

In short, you care about you and you need to make a call whether investing the extra time into a work related task is worth while or that putting in some effort is worth while, no one else can do that, and anyone that tries should be shot down vigorously.

If you have some spare time where maybe your wife is out or you were just going to sit down and watch tv and you decide that on this day trading a couple of hours of home time to for work time would provide some benefit to you then you should do it. It’s not about putting in 12 hours days, it’s about using the time you have better, and when needs must and you want to, never feel pressured.

As for effort, you should be doing this all the time, yes you can have a easy time do the job and move on and scrape by doing the bear minimal, and sometimes this is a necessity due to time constraints, so don’t feel bad about it for starters. Normally to go that little bit further and add extra functionality to something is only a small amount of actual effort so it is worth identifying some “quick wins” in terms of effort, it may take away a bit of that extra time you put in but it is always valuable.

Now, what about if you work in an environment where they are rewarding you in ways that are “non-standard” ? Maybe after a hard month of effort they let you expense a meal for you and your wife (I’ve had this happen twice and it’s really nice mainly for my wife who is a work widow) or maybe they decide to fly you and everyone in the company to a Mediterranean island and let you do a treasure hunt across an island on a jeep (this has also happened). Well, I’d say firstly, if you’re being rewarded for the time and effort you put in then grab it and go with it, take what you can from the experience you never know where it will take you, but for sure, set targets for your self and only rarely re-address them and only if the company is working for you.

So if you are that lucky and even if you are not, choose to invest time and effort into the things you do if for no other reason than so you can say you did it.

What university forgot to mention about programming

Programming at uni…

So for starters I actually learnt to program at college, which it turns out was really lucky as at university they seem to struggle to teach people the basics. If it wasn’t for learning programming in a more structured way at college; university could have been a real nightmare, luckily it was a breeze, which is some of the problem as I’ll get onto.

So what happens at university is people turn up from studying Art or History or some other subject and are given a language such as Java to jump right on into. The first module is something like use if statements and loops, 2nd or 3rd will be something like here’s a class. By the time you leave university you are pretty much at the ability of a mouse, you can write if else ladders and you know what a loop is. In short I learnt far more at college than university, at least at college we were introduced to OOP after learning basic programming, we were then introduced to concepts such as linked lists and binary trees. For some reason at university this was all missing, which is the reason for this rant.

I’ve spent a couple of weekends lately learning some basics about programming which makes me think that really a lot of this should have been where university picked up, but instead because of those not up to speed with programming we all reset and lower the grade yay for higher… education.

What did I learn

I literally learnt some basics. Up until now the majority of the programming I have needed to do (all of it) has not been of significant depth or complexity to warrant having any structure to classes or even necessarily having classes. In addition to this I introduced myself to testing, turns out when you have large complex programs having some basic tests on classes can give you some confidence that the program will actually work.

So I’ve been re-structuring code into classes, which to be fair I was taught at uni, but it leads me on to realise that I wasn’t taught about testing code or writing test cases, all of which would have been useful to touch on at university.

This got me thinking, what else did I not learn at university that could have been useful? Well how about programming concepts such as working with pluggable modules, creating interfaces to API’s so they can be versioned or providers.

I still don’t know what a lot of this stuff is, and as always it seems the best way to learn is to do it and hit problems. So I am busily refactoring my code into what I currently know, knowing full well that I don’t have a clue how I’m going to tackle any of the problems that may be presented. All the time wondering if university had spent less time bringing people up to the same speed if we would have touched on some of these subjects.

Does make you wonder how graduates these days are meant to come out of university and get a job in programming, I only hope that with the new push for schools to teach programming that the gradual bar will increase all the way across.

Summary

If I wanted to go into programming right now I’d be doing as much programming as possible out side of university / college as they just don’t teach you enough. I would also bite off more than I could chew, not knowing how to solve the whole problem is half the fun, so break the big problem into little ones and keep tackling them. Worse case scenario you learn a thousand ways not to do something.

When is enough, enough?

Every now and then…

There’s a time, once in a while when someone get’s on your nervers, if you’ve met me, I’m probably that guy ;-)
Well this is called life, and there’s not much point getting worked up about annoyances, or people that get on your nerves, unless you have spare time to waste on such activities anyway, personally I’d rather just move on and wait for sunny times. That isn’t always possible, either due to your own state of mind or for some reason the ‘annoyance’ (yes I just referred to a person as an annoyance not granting them any dignity or space or time or energy ) won’t give up and move on, again, this is called life.

Is it a long term or short term affect

From my experience I would say that there tends to be two types of ‘annoyance’ either it is short term or long term; it is quite important to identify this as it means you can take appropriate action. For example, if there is someone in your team that is traditionally a happy go lucky sort of person and one day they seem to just be an arse, sorry ‘annoyance’; then you’d probably be right to let it slide and do nothing about it, this is very low impact for you effort wise. If there is a member of your team that seems to take constant glee in pushing your buttons or pointing out your flaws, creeping behind your back and so on, this is slightly more longer term.

Short term ‘annoyance’

So with those that you would classify now as short term annoyances there’s always multiple options depending on how much you like the person. If you’re not too keen on the person or you don’t have to spend much time interacting with them you can just brush off the odd comment, if it is someone you have to work with closely you could simple ignore it and wait for an opportune moment to ask if everything is okay. Be warned, if you ask someone “is everything okay?” and you do it disingenuously this will fall back and hurt you, if you dont really care don’t ask, you’ll make it worse.

Long term ‘annoyance’

I hate these, mainly because it takes a lot of effort, either positive or negative depending on the brutality of the ‘annoyance’

If there’s someone in your team you just don’t see eye to eye with it’s a struggle, you’ve had a few handbags, disagreements etc already and you both pretty much agree you hate each other. By this point it doesn’t matter if the other person is being nice, in your eyes it’s two faced. There’s only really two ways to deal with these types of people, either be really nice, or be really devious in both cases be meticulous, and because I don’t want to let people in on how to “eliminate” these problems we’ll focus on the nice tact, it also drives them crazy!

I’ve been in a position where there’s been someone I really couldn’t get on with, At first it was handbags, but then I started to ask myself what causes someone to be that obnoxious. I started to look at their history, who they had worked for, where they had come from, how they had gotten to where they are, what structure their personal life had, how they dealt with pressure, what influences their decisions, what is important to them, how do they communicate, what are their tells and so on, the list is endless. I believe if you have a long term ‘annoyance’ you have to know them inside and out, this is great for the being nice approach, you have no choice but to get to know them as an individual. It will take weeks of being nice and ignoring snipes and odd remarks but you should see a genuine improvement in the relationship, once that happens you will find that it no longer bothers you what they do you can just be nice and say things like “I noticed you forgot this, so I did it for you” or “how was your weekend” see this is caring and sharing…

Summary

As much as you may want to change someone else you can’t, the only thing you have any control over is yourself so you can make yourself be nice, you can make yourself ignore comments and eventually things will get better. On a side note while being all nice and kind and not being mean it certainly wouldn’t hurt to have a good record of what is said in meetings, who committed to what and if you have a ticketing system, use it. Hopefully you’ll never need it, but if you do you have it, and because you wrote it down and the other person saw you doing it they have even less to go on.

So when is enough, enough? at the point you decide to start changing yourself and not others.

AWS Outage

Not again

Yes, again, another Amazon outage, in fact their reports are a little more miss leading and much more forgiving than the truth. For some background on the rant look Here and the official words: Here

So on the 29th we saw a small minor issue with a couple of serves EBS volumes suffering, luckily we identified and fixed the issue quickly by removing the nodes from the clusters. Well with that problem dealt with on with a restful weekend… not so much.

During Saturday we had a single minor incident but on the whole we seemed to survive. At some point in the early hours of Sunday on the 1st July in the UK but I guess 30th June for the Americans, another issue, here *sigh* To be honest I typically wouldn’t mind, it’s a data centre, they provide multiple regions for a reason so you can mitigate this and rest assured the availability zones are all separate well separate ish, either way they provide availability zones which are meant to be fully isolated form each other.

Wakey wakey, rise and shine

So luckily I was not on call and some other team members dealt with the initial fallout. 5 hours later my phone starts ringing, which is fine it will do that as an escalation. On a side note, about 11pm the night before my PC just stopped working and suffered a kernel panic, so I lost DNS / DHCP so no internet access easily. I rolled downstairs more or less right away and started playing with my mobile phone to set up a wireless hotspot, wonderful 2 bars of GPRS, thankfully it was enough for a terminal (or five).

It turns out that almost all of our servers were completely inaccessible, now we very much divide and conquer with our server distribution, so each node is in a different AZ (Availability Zone) on the assumption that they should be fine. On a side note I will write down some information I’ve learnt about Amazon for those starting a hosting experience with them so they can avoid some of the pitfalls I’ve seen.

Anyway, back to the point I keep procrastinating from. We managed to bring the service back up and working which wasn’t difficult but it did involve quite a bit of effort on our part to get it back working. What I was able to spot was a high amount of IO wait on most instances or at least on the ones I could get onto. In some cases a reboot was enough to kick it on its way but on others a more drastic stop / start was needed.

The annoying thing for me

Is that they have had outages of this kind in the US-East region in the past Time and Time Again. They obviously have issues with that region, and there are some underlying dependancies between AZ’s that aren’t clear, like power, EBS, RDS and S3 obviously these services need to be shared at some point to make them useful but if they are integral to what you are doing then simply putting your servers in another availability zone won’t be good enough. For example if you are using any instance… you are probably EBS backed, as we know they are not infallible.

I do hope that they are looking to make some serious improvements to that region, we are certainly considering other options now and trying to work out the bets way to mitigate these types of issues. If you are not heavily tied into US-East I would suggest abandoning ship apart from your most throw-away-able servers. I’m sure the other regions have there own issues as well, but at the same frequency?

The other thing that is puzzling, is that when we saw the issues Amazon claim all is well. There is certainly a miss understanding somewhere, I guess it could be that we were hit by the Leap second bug, but all of our devices are on new enough kernels that they shouldn’t have been affected but alas they were, either way something happened and it may remain a mystery for ever.