Puppet with out barriers -part two

Magnificent manifests

Back in the day, we use to have maybe 7 node manifests per environment and 6 or 7 environments, which is a lot of individual nodes to manage. As a result not everything was always the same.

We got to the point where each environment, although using the same modules was not always quite right, we used variables to set each environment up, but each variable was in each environment and different, yes you can keep it in check with diff and good process, but that’s time consuming. Why not just have one manifest for each tier of the application? that’s what we have now every tier is identical in terms of its use of the manifest, but obviously every environment has its own settings. So how do you have 1 manifest 100s of environments and guarantee that everything is the same across each environment except a small number of differences which can be easily seen in one file.

Now you could try and manage the different sets of vars through node inheritance but you then end up with a quite complex set of inheritance which can be a pain to track through. At first I was a little confused at how we would reduce our manifests down and still be able to set variables in the right places. Before going to far, lets look at a typical node inheritance set up.

[12:56]msmith@pinf01:/etc/puppet/manifests$ tree
.
|-- company
|   `-- init.pp
|-- dev
|   `-- init.pp
|-- site1
|   |-- init.pp
|   `-- nodes.pp
|-- site2
|   |-- dev_nodes.pp
|   |-- init.pp
|   |-- nodes.pp
|   `-- test_nodes.pp
|-- site3
|   |-- init.pp
|   `-- nodes.pp
|-- prod
|   `-- init.pp
|-- site.pp
`-- test
    `-- init.pp

In this set up, The company init.pp sets up company wide details, domain names, IP addresses for firewall rules or anything that is applicable to multiple sites. The site init.pp inherits the company one and sets up any variables needed for that specific site like DNS servers. In some of the sites we have additional configuration which inherits that site’s init.pp and sets more specific information for say the test or dev environment and finally there is an environment init.pp which sets variables for that type of environment such as environment prefix or other settings that are specific to that environment.

With this set up it gives you plenty of room to grow but can be harder to work out where the variables come from as you will have to track them back through many levels of inheritance; this can make it hard for new people to the team to identify where the variables should be set which is made harder with parameterised classes and defaults.

Now with classes you can create a class/subclass that is used to set a majority of your defaults and this can be useful if you have multiple companies or distinct groups which always insist on having different settings, the advantage of this is that your node declarations then remain minimal. However, this can also make things complicated as you have meta classes that set some of the params needed for a class but not all, so you’ll still end up passing in params within your nodes to the meta class which in turns passes them into your modules and so on. This is typically more inheritance and paramaters than most sysadmins can handle.

So we had the same issue, it was never a 5 min talk to explain what is set where and when because it could be set in 3 or 4 nodes, passed as a param and this was the same for each of the 6 environments. After much talking and disbelief we eventually started on a journey to reduce 7 environments and what was 50+ node files (each controlling multiple nodes via regexp) down to a handful.

[msmith@puppet manifests]$ tree
.
├── account.pp
├── roles
│   ├── app1.pp
│   ├── app2.pp
│   ├── default.pp
│   ├── app3.pp
│   ├── app4.pp
│   ├── app5.pp
│   └── app6.pp
└── site.pp

So to start with, the modules are still parameterised as before, heavily so to make it easy to turn features on and off and where possible everything is set as a default within the module within a params.pp file, so there is one place within a module to change the defaults for all subclasses. Each appX.pp will load which ever sets of classes it needs and where possible it will set the params to make use of various internal variables that are needed.

Each appX.pp is inherited from the default which sets most of the classes we want on each app such as ssh, mcollecitve, aws tools. This inherits the account.pp which sets variables that are relavent to each account we use, or classes that are relavent to each account, the account in this case is an amazon account. We set different yum servers, infrastructure servers and so on.

Now we still have environments, and we choose to load the variables for this via hiera, but we have gone to great lengths to restrict the variables in the environment down to those that are absolutely necessary as such we have not yet needed to make sue of hiera’s hierarchy but we understand the necessity of keeping that option open to us.

The new structure means we have at the moment no more than 5 places to edit vars for every app / env / node. This particular structure works for us because we run a small set of node types but in multiple environments. As time goes on the roles manifests directory will grow to incorporate new servers and if we need to deal with nodes of a similar role type apart from option X/Y/Z we will make use of hiera to add another layer to the system.

Next week I’ll go through the node manifests in some detail which should hopefully cement some of this blurb.

Puppet with out barriers -part one

Structure is good

Like everyone that has used puppet and is using puppet; the puppet structure we use to use to manage our nodes was relatively straight forward. But before getting into that lets go into the two or three ways you would know of already.

1. Node inheritance, Typically you define a number of nodes that define physical and logical structures that suite your business and inherit from each of these to end up with a host node that has the correct details relavent to the data centre it is in and its logical assignment within it.
2. Class inheritance, Similar to node inheritance, you create your modules which are typically agnostic and more than likely take a number of parameters; then create a module that contains a set number of node types, for example, “web” nodes. So the web node would include the apache module and would do all of the configuration you expect of all apache nodes, this can be done to the point of each node manifest simple including a web, or wiki type class.
3, By far I imagine the most common, is a mixture of them both.

Either of these methods is fine, however what will essentially happen is duplication and you’ll be faced with situations where you want 90% of a web node but not that last 10%; you then create a one of module or node and before you know it you have several web nodes or several roles defining the same task.

It’s fair to say we do not do any one of those, node inheritance a little to save duplication of code but that’s about it, I’ll touch on this more next week, but this week is about foundations, and in puppet that means modules.

Parameterise everything

Parameterised classes help with this level of flexibility but if your module isn’t fully parameterised it’s a bit of a pain, so a slight detour based on experience.

I imagine like everyone out there, I write puppet modules that are half arsed on most days. I parameterise the variables I need for my templates and stick to a simple module layout to stop my classes getting to busy. This isn’t a bad way of doing things, you get quite a lot of benefit with out a lot of investment in time, and I for one will continue to write modules that are half arsed, but with one difference, more parameters.

Occasionally I have to extend my module, some times I don’t have time to re-factor and make it perfect so I work around my module, if you’re looking to build technical debt, this is a good way of going. and if you continue down this route you will end up with modules that are complicated and hard to follow.
Initially when introducing module layouts to people they assume they will make life more complicated because you have split a module into classes, but the reality is rather than having 1 file with 100 lines you have 10 files with 10 lines, and of those 10 files you may only be using 2 files in your configuration which can make life a lot simpler than sifting through code.

A typical module layout I use will consist of the following

[root@rincewind modules]# ls example/
files  manifests  README  templates
[root@rincewind modules]# ls example/manifests/
install.pp  config.pp  init.pp  params.pp

For those not familiar with such naming schemes… init is loaded by default and will typically call include the install class and the config class. So if I want to install / configure an entire module that has had some reasonable params set I just “include modulename”. You can read more about modules and how I typically use them Here It was rather experimental at the time but with only a few tweaks it is now in a production environment and is by far the easiest module we have to maintain.

The main reason this structure works for us is that we are able to set a number of sensible defaults in params which reduces the number of params we need to pass in while allowing us to still have a lot of parameters on each class. typically it means our actual use of the classes even though they may have 40+ params won’t normally go past 10.

The big thing we did different with this module is to parameterise about 90% of all the configuration options, now you may think that’s fine for small apps but what about large ones, well we did it with our own product Alfresco, which is quite large and has a lot of options. Granted we can’t account for every possible configuration but we gave it a shot. Consequently we now have a puppet module that out of the box should work with multiple application containers (tomcat, jboss etc ) but also allows for the application to be run in different containers on the same machine.

The advantage of this up front effort on a module which is core to what we are doing is that we are now able to just change options with out having to do a major re-factor of code, adding more complexity in terms of configuration is simpler as it has been split into multiple classes that are very specific to their tasks. If we want to change most of the configuration it is simply a matter of changing that parameter.

Be specific
So you have a module that is nicely parameterise and that’s it, well not quite, you have to make sure your module is specific to it’s task, and this is one that was a little odd to get over. Puppet is great at configuration management, it’s not so good at managing dependancies and order, so we made the decision to simplify our installation process so rather than having the puppet module try and role out the application and multiple versions of it, with different configurations for different versions, it simply set’s up the environment and sets the configuration with no application deployed.

So yes, you can install within the module if you wish, and we use to do that and some complicated expanding / re-zipping of the application to include plugins, but we also have a yum repo, or git, or a ftp server where we can just write a much more specific script to deal with the installation.

By separating out the installation we have been able to cut a lot of code from our module that was trying to do far more than was sensible.

Be sensible
When you look through configuration there are sometimes options that will only ever be set if X or Y is set, in this case write more complicated modules to cope with that logic and don’t just add everything as a parameter. When you look through your module and you start seeing lots of logic, you’ve gone wrong.

Typically now when writing modules the only logic that ends up in the pp file is is feature enabled or disabled, if you try to make a manifest in a module all things to all people you end up with a lot of logic and complications that are not needed. To tackle this, after writing a base for the module I then cloned the params file and the application manifest which does most of the config and made it more specific to the needs of the main task that I’m trying to achieve. It is 90% the same as the normal application one but with a few extra params and allows me to add specific files that are 100% relavent to what we do in the cloud with out ruining the main application one incase we decide to change direction or if we need a more traditional configuration.

The point is more that you should avoid over complicating each manifest in the module, create more simple manifests that are sensible for the task in hand.

This module layout will give you something to work with rather than against, we spent a lot of time making our whole puppet infrastructure as simple as possible to explain, this doesn’t mean it’s simple it just means everyone understands what it does, me, my boss, my bosses boss and anyone else that asks “How does it work?”

GNU Parallel – When bash just isn’t quick enough

Bash not cutting it for you?

We were looking at using nrpe to do smoke tests of our product after a deployment to give us a feel for how well the system faired with the deployment. Well, we wanted it to run quicker, anyone that has performance tuned bash scripts knows that you strip out as much as possible to and use the in built functions as they are quicker.

Well we had done that and still it wasn’t quickenough. My boss came across parallel Luckily the documentation was pretty good but we still had fun playing with it.

Imagine a situation where you want to run the following commands

echo 1 1
echo 1 2
echo 1 3
echo 2 1
echo 2 2
echo 2 3

Typically to do this you may write some bash loops to achieve this as follows

for i in {1..2}
do
    for x in {1..3}
    do
        echo "echo $i $x"
    done
done

This is a quite simple example, so lets look at something more useful, how would you do the following commands?

service tomcat6 stop
service tomcat6 start
service httpd stop
service httpd start
service nscd stop
service nscd start

Well, if you didn’t realise it’s the same loop above just with slightly different numbers… but what if I said you coud do that all on one line with no complicated syntax? Well this is where parallel fits in, it’s quite useful.

parallel service {1} {2} :::tomcat6 httpd nscd ::: stop start

Now it’s worth mentioning, Never do the above and in fact read on…

So having done Parallel a great disservice by reducing it to argument replacement it is probably worth mentioning you can do some other funky stuff. Which you can find on the documentation page, and it may be more applicable to your needs.

The other useful feature of parallel which I sort of started this blog on is the fact that it is run in parallel…. so if you consider the 6 service commands above, to execute all 6 could take 45 seconds, but by using parallel you could significantly reduce the time it takes to execute.

A real world example

Here is a sample of the smoke test script we’re running which makes use of MCollective with the nrpe plugin, this is a quick script my boss knocked up and I decided to steal some of it as an example, Thanks Steve.

#!/bin/bash

env=$1

function check
{
  echo -e "\nRunning $1 nagios check\n" 
  mco nrpe $1 -I /^$2/ 
}

# Arrays!

Common[0]='check_disk_all'
Common[1]='check_uptime'
Common[2]='check_yum'
Common[3]='check_ssh_processes'
Common[4]='check_load_avg'
Common[5]='check_unix_memory'
Common[6]='check_unix_swap'
Common[7]='check_zombie_processes'

Web[0]='check_apache_processes'

for i in "${Common[@]}"
do
  check $i $1server1
  check $i $1server2
  check $i $1server3
  check $i $1server4
done

# Check the Web nodes

for i in "${Web[@]}"
do
  check $i $1server1
done

Now the script above is abridged but the time is from the full script, and it takes 2 min 36 seconds, Well the parallel script is a little faster at 1 min 25 seconds, not bad.

Here is a sample of the same code above but in parallels

#!/bin/bash

env=$1

# Common checks
parallel mco nrpe {1} --np -q -I /^${env}{2}/ ::: check_disk_all check_uptime check_yum check_ssh_processes check_load_avg check_unix_memory check_unix_swap check_zombie_processes ::: server1 server2 server3 server4

parallel mco nrpe {1} --np -q -I /^${env}{2}/ ::: check_apache_processes ::: server1

It is not really pretty but it is quick.

parallel has all of the features of xargs too but with the added bonus of being in parallel so you can get these massive time savings if needed, the only thing to bear in mind is that it is parallel, sounds like a silly thing to mention but it does mean the service example above would have to be done in two parts to ensure the stop was done before the start.

Hopefully that’s proved a little interesting, enough to play with and make something truly amazing happen, Enjoy.

Bind DDNS updates

DDNS

A little while ago I started working on some DDNS on a RHEL6 box where a client would be able to update it’s own IP address after a reboot to remove the need for us to do it our selves. I was pretty sure this would be a walk in the park, but I hit one very odd issue with nsupdate which looks to be caused by a change recently in the way the keys are generated but this is getting ahead.

DDNs is actually really easy to set, and there’s a few sites out there that will tell you how easy it is, in fact a majority of them do this. However most of them have a section that will mention ddns-keygen, they all go through and tell you to do the same thing.

dnssec-keygen -a HMAC-MD5 -b 512 -n HOST domain.com

After you have generated the two files (.key and .private) Most are un clear as to what to do next, the answer is you use the .key file for everything and you might as well forget the .private file was created.

You have to take the secret from the key file and place it in your named.conf, either as an include or directly but the code you want is as follows:

key domain.com {
algorithm HMAC-MD5;
secret "SAdhkjhkjashdkjhasdkjbasdkbk8768/+sadfnasd asdkjahsdasdhkjhasdasd";
};

Worth noting I keyboard mashed the key…. but from the .key file you want to copy the whole key (spaces and all), the key above is in fact on one line with a space, so it may have line wrapped.

Once you have this in your named.conf you can start securing your zones with the allow-update command

zone "domain.com" {
type master;
file domain.com;
allow-update { key domain.com; };
};

Now, I got to this stage with about 20 different websites and all was fine. The issues followed on trying to get nsupdate to work with the key. A lot, if not all, tell you to take the private key and pass that into nsupdate with the -k option.

Well this failed for me.

[root at host etc]# nsupdate -d -k ./Kdomain.com.+157+27613.private
Creating key...
16-Jul-2012 18:17:45.111 ./Kdomain.com.+157+27613.private:1: unknown option 'Private-key-format:'
16-Jul-2012 18:17:45.111 ./Kdomain.com.+157+27613.private:14: unexpected token near end of file
could not read key from ./Kdomain.com.+157+27613.{private,key}: unexpected token


It took me a while to figure out that it was in fact the key file that was causing the problem, one way i helped work it out is with the -y option

nsupdate -y domain.key:SAdhkjhkjashdkjhasdkjbasdkbk8768/+sadfnasdasdkjahsdasdhkjhasdasd

This helped prove that there was an issue with the key format, not there was no space above but the key does normally have one. now, a quite important thing here is that everyone thats ays to use the private key also has a key with this line in it: Private-key-format: v1.2, well mine said Private-key-format: v1.3 and failed. I’m not even sure if that’s relavent.

The fix for this issue was in fact really simple, if you read through the nsupdate man page sufficiently you’ll find that it will take the key in the bind format so create a file containing the say info as…

key domain.com {
algorithm HMAC-MD5;
secret "SAdhkjhkjashdkjhasdkjbasdkbk8768/+sadfnasd asdkjahsdasdhkjhasdasd";
};

Try again and watch it work.

AWS best practice – Architecting the cloud

Architecting the Cloud

In this post I will go over some best practice to help you architect a solution that will hopefully survive most amazon incidents. To start with, let’s look at a single region and how to make the best use of a region.

Instances

Starting with the most basic steps first, you want to have each instance created be as stateless as possible and as light weight as possible. Ideally you would use instance-store backed instances as these do not rely on EBS to be working, so you are reducing your dependancy on the Amazon infrastructure and one less dependancy is one less thing to go wrong. If you can not avoid the use of an EBS backed instance then you will want to be ensuring that you have multiple instances providing the same service.

Also consider the use of your service, S3 is slow for you to download data and then share out again, but you could push the handling of the access off to Amazon helping make your environment a bit more stateless. it is also worth noting that there have been far fewer issues with S3 than EBS. Obviously if you need the capacity of EBS (S3 has a single file size limit of 5TB) then RAID the drives together for data storage. You can not do this for your instance storage but at least your data will be okay.

On a side note out of 200+ volumes during a recent outage we only had one with issues so they are quite reliable, although some times slow, however if your aim is ultimate uptime you should not rely on it.

Storage

As I pointed out before, your main storage types are EBS and S3, EBS is block device storage and as a result is just another hard drive for you to manage, you can set RAID on them or leave them as single disks. Then there is S3 which is a key value store which is accessed via a REST API to get the data.

With EBS and S3 it is never stated anywhere that your data is backed up. Your data is your responsibility, if you need a backup you ned to take snapshots of the data and if you want an “off site” equivalent you would need to make sure you have the EBS snapshot replicated to another region, the same applies for S3.

A big advantage of EBS is the speed to write and read from it, if you have an application that requires large amounts of disk space then this si your only real option without re-architecting.

S3 is Simple, hence the name, as a result it very rarely goes wrong but it does have a lot more limitations around it compared to EBS. One of them is down to the reliability, it won’t send a confirmation that the data has written until it has been written to two AZs, for light usage, and non time dependant work this is probably your best choice. S3 is ideal however for data that is going to be read a lot, one reason is it can easily be pushed into cloud front (A CDN) and as a result you can start offloading the work from your node.

In short where possible don’t store anything, if you do have to store it try S3 so you can offload the access if that is not adequate then fall back to EBS and make sure you have decent snapshots and be prepared for it to fail miserably.

Database Storage

RDS is a nice database service that will take care of a lot of hassel for you and I’d recommend that is used or DynamoDB. RDS is can be split across multiple AZs and the patch management is taken care of for you which leaves you to just configure any parameters you want and point your data to it. There are limitations with RDS of 1TB of database storage but in most cases I’d hope the application could deal with this some how else you are left trying to run a highly performant database in Amazon at which point you are on your own.

Unless of course you can make use of a non-rational database such as DynamoDB which is infinitely scalable and performant and totally managed for you. Sounds too good to be true, well of course, it is a non rational database and the scalability speed is limited, at the present moment in time you can only double the size and speed of your dynamoDB once per day, so if you are doing a marketing campaign you have to take this int account days in advance possibly.

Availability Zones

Hopefully by this point you have chosen the write type of instance and storage locations for any data, leaving you the joys of thinking about resilience. At a bear minimum you will want to have servers that provide the same functionality spread out across multiple AZs and some sort of balancing mechanism, be it an ELB, round robin DNS or latency based DNS.

The more availability zones your systems are in the more likely you are to be able to cope with any incidents, ideally you would take care of this through the use of auto scaling, that way if there is a major incident it will bring nodes back for you.

Having instances in multiple AZs will protect you in about 80% of cases, however, EBS and S3, although spread across multiple AZ’s are a single point of failure and I have seen issues where access to EBS backed instances is incredibly slow across a number of servers, in my case 50% of servers across multiple availability zones were all affected by accessibility of the data. So your service can not rely on a single region for reasons like this. One of the reasons I believe for this is when EBS fails there is some sort of auto recovery which can flood the network and cause some disruption to other instances.

A little known fact about AZs is that every client’s AZ is different. If you have 2 accounts with Amazon you may well get presented different AZs but even those with the same name may in fact be in different AZs and visa-versa.

Regions

With all of the above you can run a quite successful service all in one region with a reasonable SLA, if you are lucky to not have any incidents. At the very least you should consider making your backups into another region. multiple regions much like multiple data centres are difficult, especially when you have no control over the networking, this leaves you in a bit of a predicament. You can do latency based routing within Route53 or weighted Round Robin, in this case, assume a region is off line your traffic should be re-routed to the alternative address.

Things to watch out for

Over the months we’ve been hosting on AWS there’s been a number of occasions where things don’t work the way you expect them too and the aim of this section is to give you some pointers to save you the sorrow.

Instance updates
There has been a number of occasions where an instance has stopped working with no good reason, all of a sudden the network may drop a few packets, the IO wait may go high or just in general it is not behaving the way it should. In these situations, the only solution is to stop and start the instance, a little known fact is that the stop and start process will ensure that your instance is on hardware with the latest software updates. However, I have been told by AWS support that new instances may end up on hardware that is not optimal so as a result you should always stop and start new instances.

In severe cases Amazon will mark a node in a degraded state, but I believe they will only do this after a certain percentage of instances have migrated over or it has been degraded for a while.

Scaling up instance size

This is an odd one, predominantly because of a contradiction. You can easily scale up any instance by stopping it in the web gui and changing it’s size on the right click menu. This is good, you can have a short period of downtime and have a much larger instance, the downside being your IP and DNS will change as it is a stop and start. However, if you had deployed your instance via Cloud formations it would be able to scale up and down on the fly with a cloud formations script change.

Security with ELBs
With security groups you can add TCP, ICMP or UDP access rules to a group from another security group or from a network range thus securing instances in the same way a perimeter firewall would. However, this doesn’t guarantee security specifically if you then add an ELB to the front end. With ELB’s you do not know what the network would or could be for them so you ultimately would need to open up full access just to get the ELB to talk to your host. Now, amazon will allow you to add a special security group that will basically grant the ELB’s full access to your security group and as a result you have guaranteed that access is now secured, in the most part.

However, ELB’s are by their nature publicly accessibly, so what do you do if you’re in EC2 and want to secure your ELB which you may need to load balance some traffic. Well Nothing. The only option available for you in this situation is to use a ELB within a VPC which gives you that ability to apply security groups to the ELB.

There are ways to architect around this using apache but this does depend on your architecture and how you intend to use the balancer.

Everything will fail
Don’t rely on anything to be available, if you make use of the API to return results expect it to fail or to not be available. One thing we do is cache some of the details locally and add some logic around the data so that if it’s not available it continues to work.. The same principle aplies to each and every server / service you are using, where possible just expect it to not be there, if it has to be there at least make sure it fails gracefully.