Puppet inheritance, revisited

A calmer approach

A few weeks ago I wrote an article which was more of a rant than necessary. I was trying to drastically alter the way we right one of our puppet modules, and not being a simple module like ntp it required a bit of flexibility. To give you an understanding of what our puppet module does, we deploy Alfresco, including the automation of upgrades and configuring varios aspects of the application including bespoke overrides here and there and all sorts of wizzardy in the middle. The original puppet module was written by Ken Barber and then re-written by Adrian and myself. Needless to say there was some big puppet style brains working on it and the time had come to make some fundamental changes, mainly to make it easier to deploy directly from built code. So a lot of the work I’ve been doing on the module meant that it was more important to have the module deploy code form the build servers as well as automatically update itself.

Well, I achieved that within the old module, but quite frankly, the module was about 10 times larger than it needed to be specifically for an operational deployment into the cloud. With this in mind I thought it would eb a good use of time to re-write it and make easier to maintain and hopefully easier to extend, hence the rant around inheritance; in my head I had the perfect solution, which due to some bugs didn’t work. Well Time has moved on and as always progress must continue.

Puppet modules, the easy way.

Okay, so What did I do to overcome the lack of inheritance yet not have all of the duplication that was in the old modules. Simples! combine the two. I gave it some thought and I realsed that the best way out of the situation is to make it so that the variables were set from one place, params, so even though there is some duplication, you still only have to set the variables in one place. As a result I wrote a simple puppet module which has not been tested…. as a demonstration, this is the same structure as a live module so in theory will work the same way.

Manifests

[matthew@rincewind manifests]$ ll
total 20
-rw-r--r--. 1 root root 1981 Feb 17 21:26 config.pp
-rw-r--r--. 1 root root 1757 Feb 17 21:06 init.pp
-rw-r--r--. 1 root root 1578 Feb 17 21:12 install.pp
-rw-r--r--. 1 root root  921 Feb 17 21:51 params.pp
-rw-r--r--. 1 root root 2480 Feb 17 21:57 soimaapp.pp

Let’s start at the top,

init.pp

# = Class: soimasysadmin
#
#   This is the init class for the soimasysadmin module, it loads other required clases
#
# == Parameters:
#
#   *application_container_class*             = Puppet application container class deploying soimasysadmin i.e. tomcat
#   *application_container_cache*             = Location of the tomcat cache directory
#   *application_container_home*              = Application container home directory i.e. /var/lib/tomcat6/
#   *application_container_user*              = User for application container i.e. tomcat
#   *application_container_group*             = Group for application container i.e. tomcat
#
# == Actions:
#   Include any classes that are needed i.e. params, install etc
#
# == Requires:
#
# == Sample Usage:
#
#   class	{ 
#			"soimasysadmin":
#				application_container_class		=> 	"tomcat6",
#				application_container_home		=>	"/srv/tomcat"
#		}
#
class soimasysadmin  ( 	$application_container_class     =	$soimasysadmin::params::application_container_class,
			                  $application_container_cache     =	$soimasysadmin::params::application_container_cache,
                  			$application_container_home      =	$soimasysadmin::params::application_container_home,
                  			$application_container_user      =	$soimasysadmin::params::application_container_user,
                  			$application_container_group     =	$soimasysadmin::params::application_container_group,
		                ) 	inherits soimasysadmin::params {

include soimasysadmin::install

# The application_container variables are set here, but the defaults are all in params, which is available thanks to inheritance, Because this is the top of the tree, these variables can be accessed by other classes as needed

}

install.pp

# = Class: soimasysadmin::install
#
#   This class will install and configure directories and system wide settings that are relavent to soimasysadmin
#
# == Parameters:
#
#   All Parameters should have defaults set in params
#
#   *logrotate_days*      = Number of days to keep soimasysadmin logs for
#
# == Actions:
#   - Set up the application_container
#   - Set up soimasysadmin directories as needed
#
# == Requires:
#   - ${::soimasysadmin::application_container_class},
#
# == Sample Usage:
#
#   class	{ 
#			"soimasysadmin":
#				application_container_class		=> 	"tomcat6",
#				application_container_home		=>	"/srv/tomcat"
#		}
#
#

class soimasysadmin::install 	(	$logrotate_days = $soimasysadmin::params::logrotate_days,
                        			) inherits soimasysadmin::params	{

  File  {
    require =>  Class["${::soimasysadmin::application_container_class}"],
    owner   =>  "${::soimasysadmin::application_container_user}",
    group   =>  "${::soimasysadmin::application_container_group}"
  }

  #
  # Create folders
  #
  define shared_classpath ( $application_container_conf_dir = "$::soimasysadmin::application_container_conf_dir") {

    file {
      "${::soimasysadmin::application_container_home}/${name}/shared":
      ensure  =>  directory,
      notify  =>  undef,
    }
    file {
      "${::soimasysadmin::application_container_home}/${name}/shared/classes":
      ensure  =>  directory,
      notify  =>  undef,
    }
	}

	file {
		"/etc/logrotate.d/soimasysadmin":
		source		=>	"pupet:///modules/soimasysadmin/soimasysadmin.logrotate",
	}

}

params.pp

# = Class: soimasysadmin::params
#
# This class sets the defaults for soimasysadmin parameters
#
# == Parameters:
#
#
# == Actions:
#   Set parameters that are needed globally across the soimasysadmin class
#
# == Requires:
#
# == Sample Usage:
#
#   class soimasysadmin::foo inherits soimasysadmin:params ($foo  = $soimasysadmin::params::foo,
#                                                 					$bar  = $soimasysadmin::params::bar) {
#   }
#
#
class soimasysadmin::params  ( ) {

#
# Application Container details
#

  $application_container_class            = "tomcat"
  $application_container_cache            = "/var/cache/tomcat6"
  $application_container_home             = "/srv/tomcat"
  $application_container_user             = "tomcat"
  $application_container_group            = "tomcat"


#
#	soimapp config
#

	$dev_override_enabled   = "false"
  $custom_dev_signup_url	= "http://$hostname/soimaapp" 

}

Okay that’s all that’s needed to do the install, Notice that each class inherits params. Interestingly enough Adrian informs me this probably shouldn’t work and is possibly a bug, as params is being inherited multiple times. I think the only reason it works is that we have no resources in the params.pp if we add a file resource in ti will fail. but nonetheless, this works (for now)

config.pp

# = Class: soimasysadmin::config
#
#		This class configures the soimasysadmin repository
#
# == Parameters:
#
#		All Parameters should have defaults set in params
#
#		*db_name*												= The Database name 
#		*db_username*										= User to connect to the DB with
#		*db_password*										= Password for the user
#		*db_server*											= The DB server host 
#		*db_port*												= DB port i.e. 3306
#		*db_pool_min*										= Min DB connection pool
#		*db_pool_max*										= Max DB connection pool
#		*db_pool_initial*								= Initial DB connection pool size
#
# == Actions:
#   Install and configure soimasysadmin repository component into appropriate
#   application container
#
# == Requires:
#   - Class["${soimasysadmin::params::application_container_class}"],
#
# == Sample Usage:
# 
#   class	{ 
#			"soimasysadmin::config":
#				db_password 	=>	"Hdy^D7D6fvndsakj(*80",
#		}
#
#

class soimasysadmin::config	(	$db_name					=	$soimasysadmin::params::db_name,
															$db_username			=	$soimasysadmin::params::db_username,
															$db_password,
															$db_server				=	$soimasysadmin::params::db_server,
															$db_port					=	$soimasysadmin::params::db_port,
															$db_pool_min			=	$soimasysadmin::params::db_pool_min,
															$db_pool_max			=	$soimasysadmin::params::db_pool_max,
															$db_pool_initial	=	$soimasysadmin::params::db_pool_initial,
														)	inherits soimasysadmin::params	{
	File {
    owner   =>  "${::soimasysadmin::application_container_user}",
		group		=>	"${::soimasysadmin::application_container_group}",
		notify	=>	Service["${::soimasysadmin::soimasysadmin::application_container_service}"],
	}

	#
	# properties file
	#

	# If extra config set do concatinate
  file {
    "${::soimasysadmin::application_container_home}/${::soimasysadmin::application_container_instance}/properties":
    content	=> template("soimasysadmin/properties"),
    mode		=> 0640,
  }
}

NB In this instance I use config.pp as a way of setting up default config for the application, things common, for example the DB, where as application specific config is done separately, this obviously depends on your application…

soimaapp.pp

# = Class: soimasysadmin::soimaapp
#
# This class installs the soimaapp front end for the soimasysadmin repository
#
# == Parameters:
#
#		All Parameters should have defaults set in params
#
#		*dev_override_enabled*						= Enable dev configuration override
#		*custom_dev_signup_url*						=	Custom dev signup url
#		*application_container_instance*	= Application container instance i.e. tomcat1, tomcat2, tomcatN
#		*application_container_conf_dir*	= Application container conf directory i.e. /etc/tomcat6/conf/
#		*application_container_service*		= Service name of application container i.e. tomcat6
#
# == Actions:
#   Install and configure soimaapp into appropriate application container
#
# == Requires:
#		- Class["${soimasysadmin::params::soimasysadmin_application_container_class}",
#						"soimasysadmin::install"],
#
# == Sample Usage:
# 
#		include soimasysadmin::soimaapp
#
class soimasysadmin::soimaapp	(	$dev_override_enabled								$soimasysadmin::params::dev_override_enabled,
															$custom_dev_signup_url							=	$soimasysadmin::params::custom_dev_signup_url,
															$application_container_instance			= $soimasysadmin::params::application_container_instance,
															$application_container_conf_dir			= $soimasysadmin::params::application_container_conf_dir,
															$application_container_service			= $soimasysadmin::params::application_container_service
															) inherits  soimasysadmin::params  {

	# Set Default actions for soimasysadmin files
	File	{
		notify	=>	Class["${::soimasysadmin::application_container_class}"],
		owner		=>	"${::soimasysadmin::application_container_user}",
    group   =>  "${::soimasysadmin::application_container_group}",
	}


	#
	#	Install / Upgrade War
	#
	
	# Push war file to application container
  file {
    "${::soimasysadmin::application_container_home}/${application_container_instance}/webapps/soimaapp.war":
    source	=> "puppet:///modules/soimasysadmin/soimaapp.war"
    mode		=> "0644",
  }

	#
	#	Shared class created
	#
	soimasysadmin::install::shared_classpath {
		"${application_container_instance}":
			application_container_conf_dir	=>	"${application_container_conf_dir}"
	}
	
	if	( $dev_override_enabled == "true" ) {	
		file {
			"${::soimasysadmin::application_container_home}/${application_container_instance}/shared/classes/soimasysadmin/web-extension/soimaapp-config-custom.xml":
			content => template("soimasysadmin/soimaapp-config-custom.xml"),
		}
	}	
}

Okay that’s the basics, in the examples above, soimaapp does application specific config, any generic settings are in params.pp, any global settings are in init.pp; hopefully the rest is self explanatory.

I haven’t included any templates or files, you can work that out, you are smart after all.

Now the bad news

I will say the above does work, i’ve even tested it with inheriting soimaapp.pp and applying even more specific config over the top and all seems well, so what exactly is the bad news…

I tested all of this on the latest version of puppet 2.7.10 and then came across this, luckily 2.7.11 is out and available but I just haven’t tried it yet. let’s hope it works!

The perfect solution myth

The Myth

There’s a new product roll out, the solution has been designed, everyone has added their own unique and valuable points to the big picture, it looks perfect. Unlikely.

Why is it a Myth? Well What ever you design is perfect only at the moment it was designed, from that point onwards it falls quietly into obsoleteness. Once you’ve experienced this a few times you’ll understand that the value in any solution is more about the flexibility and being realistic, expect any solution you put in place to have a shelf life of 6 months.

You have to consider the possibility that once you’ve wasted 6 months building out the worlds most beautiful solution that it was 3 months late to market and therefore no longer attracted the audience you wanted it to. With that in mind it’s a good idea to work out what is vital to be done in the immediate future and to have a plan about where you want to go, after all if you don’t know where you are going you will never get there.

What can you do?

Unless you are designing the solution maybe not much, but most likely you are talking to the people that are doing it or you are fortunate enough to be at the helm of the design ship and you can enact the real change needed.

Once you understand that the solution is only really going to last a few months the rest of it is irrelevant. Make sure that what ever you are putting in place meets the requirements, but above all else when faced with a decision on a slightly more flexible approach or a less flexible one, go for the more flexible one. This won’t always be possible but you want to mitigate the foot shooting as much as possible, after all you will be the one that has to re-build the solution if it is not flexible.

Not too long ago I was in the un-fortunate position of watching a colleague implement a puppet server, luckily they had experience of doing this in the past and although it was something I was really quite keen to do I had to bow down and let them continue. I only had one concern and that was that we structure our node manifests in a way that supporting multiple environments would be easier in the long run.
My idea was to have the site.pp include any pp file under puppet/manifests/something/*.pp unfortunately this didn’t happen and we stuck to one nodes.pp which is easier to manage, however we do now have a node manifest which is some 2000 lines long. Needless to say 5 months on we now have to implement that structure and migrate the nodes from the central nodes.pp into more site specific ones. I’d go into more detail but quite frankly it’s not that interesting and a little off topic for right now. In short, More flexibility is good!

This works in other situations too, just as you won’t ever design the perfect solution you won’t ever get it right first time but above all else, don’t worry about it. Don’t get frustrated when you see a solution going pear shaped or someone is implementing something that is not perfect, as long as it does what it needs to do now. Ideally it will do what it needs to do now and have the flexibility to be extended in the future.

So back to the point, a little frustrated, but it didn’t really cause any harm, we avoided a conflict over the issue and they now have the task of implementing the environmental structure, it is a more iterative approach. In this case it was more than likely we would need the environmental support so we should have done it sooner.

So we should have done that and we didn’t, however we spent no time at all worrying about a lot of other things, How would we cope with multiple sites in multiple countries? How will we transfer terabytes of data out of a hosting provider? I still have no idea on these; Why? it’s not that important, we haven’t got that far yet. In the large scheme of things the first 6 months of the project was more about meeting very specific business goals and making sure had done enough due diligence, more importantly we needed to implement things that really mattered.

We needed the tools to allow us to do the funky multi site environment, we needed the understanding of the product. With any new service it’s easy to get caught in the bigger picture, and be thinking that you have to worry about the multi site environment now because it will cause big issues if you don’t. in some cases that’s true.

So next time you’re faced with that situation do this.

Research alternatives and understand there impact if they were implemented

Check there’s nothing stopping you doing something. For example can you ship a portable hard drive to put the data on to move it?

Write it down some where

Move on, Job done

It’s more important that you understand the impact of inaction that doing the action, so what you backups are all manual, the important thing is you have backups, You can automate them later if necessary (although to be honest, that one should be high on the list!)

Summary

Work out what is important to be done now, work out what the impact of the inaction could be. Don’t worry about designing the perfect solution, it will be wrong.

RHN Satellite vs Puppet, A clear victory?

Is It Clear?

No! It never is, it is all about what your environment looks like and what you are trying to achieve. Both solutions provide configuration management, both have a paid for and a free version, although you’d be forgiven for not realising you can get RHN Satellite (or close to it) for free, At the end of the day they are different tools aimed at different audiences and it is important that you, yes you the sysadmin, works out what your environment is, not what it might be, not what think it is, but actually what it is.

Step 1

Before we even consider the features and the pro’s and con’s of each solution we (and I mean you) have to work out what the solution needs to provide, and more importantly the skill set of those managing it. There’s a quick way to do this and a slightly more laborious one, lets look at what technology is used to run each solution.

It’s worth noting the following skills are what I would say is needed to produce an effective system, that is not to say that you could do it with less or that you shouldn’t have more skills….

RHN Satellite required skills

Understanding of python – Especially if you want to interact with the API’s or configure templates; if you are not going to use it for templating configuration files you will need no skills in this.

Basic sysadmin skills, so RHCT or higher

Understanding of networking

That’s it, it is designed to be easy to use, the installation and configuration is not overly easy, if you know what you are doing / As good as the RH Consultants, You’re looking at 3 days, if you have no idea what you’re doing, allow 2 weeks. Personally I would get Red Hat in to set it up and train you at the same time you’ll be in a better position for it.

Puppet

Good Ruby skills or Very good programming skills

Understanding of object orientated programming languages

Good sysadmin skills, RHCE, if you can’t set up DHCP, DNS, Yum Repos, Apache, NFS, SVN or basic web applications from the CLI then this is not you.

Understanding of networking

A slightly higher set of skills, but achievable by most, it is not going to be difficult to install and get working, I’d think if you had an RHCE you’d have a working puppet server in a matter of mins (you would need to add a yum repo and install a package, done)

Who’s running the service?

Okay so you, the super dooper sysadmin, manage to convince yourself that you are the right person to run puppet, You get it installed set up and configured, write a few basic modules and then pass it off to some other team to “run with it”, their skill set is closer in line to those needed for RHN Satellite. In short, Bad sysadmin!

You have to consider who will be using the system, the skill set and the aptitude to learn and progress, just because you want to do puppet doesn’t make it the right solution. You could always go with the more simple RHN Satellite server to start with and as skills develop look back at something like puppet in a couple of years.

Step 2

What features do you need? Not what features you want… So, what does that mean… Do you need to have dynamic configuration files, files where depending on what the state of the node or configuration around the node change their configuration using if statements, loops, case statements etc?
Do you want to easily be able to build and deploy servers from “bare metal”?

Hopefully by this point you will have a good understanding of the skill set to support it and what the business actually needs, now you’ve done this I can happily tell you that either solution is able to do what ever you are thinking of (in reason) but it was important to get a fuller understanding of what was needed.

Puppet Features, a totally non-exhaustive list

Dynamic configuration of files through very powerful ruby templating, if you can think of it, it can do it

Powerful node inheritance for ease of managing set’s of servers

Classes to manage applications with parameters and basic inheritance (See Last Weeks post)

I did say it was non-exhaustive, for a full list look here but be warned, just because it say’s it can do something doesn’t mean it can do it as well as you might think, Doubly true of RHN Satellite server!

The important thing for puppet is the community behind it and the fact it is extensible, you can do anything with it. You can create your own providers, resource types, facter variables etc etc there’s always new developments so you really do need to subscribe to their Blogs

You can get an Enterprise version, which comes with support, a fancy gui and all that warm fuzzy stuff, you can even get the “bare metal” build by using something like Foreman

Enough of Puppet what about the RHN Satellite!

RHN Satellite Features

Repository management through Channels

Organisational management – You can create sub-organisations that have access to specific nodes or profiles to apply builds, so they appear to have their own RHN

Security control – You can easily manage access to the web interface, nodes, access keys

Easy to use – Really it is, anyone with a little tech savvy and some time on their hands could work it out

A few more features can be found here, But the real benefit with the RHN satellite system is the ease of use, if the people running the service are more RHCT than RHCE then it’s worth considering.

I will say it’s easier to manage your patch release cycle, although in reality it isn’t; the RHN Satellite does allow anyone to manage the flow, move systems between different channels etc etc.

One of the features I liked the most was the ability to group servers together and apply patches to those groups and manage a group of servers as single server, and migrate them from dev to staging etc etc.

The ups and the downs

So we’ve looked at what you need and what you want and who should be looking after it and even touched on a few features. With all this in mind I never once mentioned the downsides of either.

The biggest downside with puppet is its flexibility and its pure focus on configuration management, as a result it doesn’t fire up a PXE boot service or easily integrate OS install to configuration with out additional tools, it just does configuration. As a result you have to provide all of these ancillary services in addition to the configuration management to achieve the same completeness of service that you get from the RHN Satellite. It is for this reason that you need the additional skills and experience to cope with it or Foreman

So what about the RHN Satellite server? The biggest let down with this is the configuration management, if you want to push a static file out it’s really straight forward, and when I was looking at this a few years back you could put variables into the files but from memory the set of options were limited, like you could add the local IP address or the hostname of the server, but you couldn’t pass in custom settings.

The biggest benefit of puppet is the combination of generic modules and well written template files, the principles behind it is that you may very well have a complicated module, but you should be able to switch the configuration at a moments notice. This provides a very flexible approach to delivering the configuration. For example You can have a simple apache module which you can add additional complexity to through inheritance, parameters and defines. With RHN Satellite you just won’t get that unless you re-package your apache into its own RPM, for each type of web service.

With the RHN Satellite the biggest advantage is purely the easy of use, it is a jack of all trades and a master of none, but if your aims are simple and your staff not quite up there on the pillars of excellence it is a good solution that you will be able to do most of what you want with.

Summary

For me I’d boil it down to the simple way of determining this.

If your company is predominately Microsoft Windows, or the sysadmin’s are not dedicated (and yes that’s plural sysadmin’s…) to Linux then I would recommend RHN Satellite, unless you have a very specific use case that can not be solved by the RHN satellite it is worth giving up some flexibility. For example, if you need to manage Red Hat and Debian, rule out the RHN Satellite, or if you know that you are going to be growing a team (know not think…) skills and numbers to be dedicated Linux sysadmins.

If you are an open source company, or have dedicated Linux Sysadmins who have been there, done that, brought the T-shirt, ruined the T-shirt redecorating the house and know the difference between nc -z -t and nc -z -w 10. Then I would consider puppet your first choice, it is young and upcoming, it’s easy to forget that it has not been around 5 years. It has some rough edges but they are getting better, and with support from puppet it makes total sense.

It’s worth touching on the training and availability of skills, Good Luck RHN satellite skills are not well-known and mainly retained within Red Hat. Puppet skills are in very high demand and people claiming the experience and understanding may be pulling your leg (a lot in some cases). However training is available for both the RHN Satellite and Puppet

These are not two products to compare evenly, both can be done for free, both can be very complicated, the only recommendation is to not choose either one based on their technical merit alone but more so on which one fits best with the aims of the project, the hearts and minds of those using the system and good ol’fashioned gut-feel.

For me, having used both, I would lean more towards Puppet, but I’m lucky, Where I am we have a lot of very technical people who are able to understand and work with puppet.

Who burnt down the building?

Fire!

There comes a time in every Sysadmin’s life where some management bod comes rushing over red faced and out of breath, From my personal experience I highly recommend not replying with “It is working okay for me” That does not get brownie points.

Having learnt from that experience I do highly recommend showing a bit of understanding instead, It is far far better to reply with “Let me look at this right away”. That doesn’t necessarily mean you do need to look at it right away, but if it’s someone in your direct reporting path I’d highly recommend following up on the lip service.

Whether you like it or not, sysadmins are on the front line, it doesn’t matter if you are labelled as “3rd line” or “4th line” you are still on the front line. The buck stops with you, sorry. This also means your customer service skills need to be exceptional, again, sorry. I realise that most sysadmins are technology focused, the latest and the best technology geek-o-rama is all important etc etc yadda yadda *Yawn* For reference that is what the Sales / Marketing guy is doing while you’re explaining the problem that he really doesn’t care about.

At this point you’re wondering what this has to do with fire fighting, absolutely nothing, Wrong!

Welcome to the world of expectation management

So you’ve had the angry management bod, You’ve correctly not told them “it’s all okay” and you didn’t chew their ear off about all that wonderful technology stuff that may be causing the potential issue. At this point there’s a few acceptable ways to mitigate another bigger (and redder) management bod turning up.

“Let me look at that right now for you”

“Let me have a quick look and see what the issue is and I’ll give you an update in 5 mins”

“Thanks for letting us know about this issue, we’ll send out a note to everyone with an update shortly”

Here we have our 3 easy answers to all issues:

1, You know what it is and you know it’s an easy fix, You could ask them to log a ticket (this will annoy them for what it’s worth..) or you can just do it quickly. Even if you have a policy where tickets must be logged, just do the work and log the ticket for them. Once you’ve fixed it you can let them know that the easiest way to get this change done is through the ticketing system.

2, You know what the issue is, It may be 5 mins, it may be 25 mins. At least this way you can get rid of the person standing over your shoulder and focus on fixing the issue. Best case is you fix it in 5 mins walk over (yes, walk over and say you fixed it!) and you have a happy person. Worse case scenario is you spend 5 mins working out what the issue is and walk over (yes, again with the walking over) and you update the person with the ETA, at which point you rush back to your desk and try to fix the issue, shortly followed by… you guessed it, A walk ver to their desk / office to inform them it is now fixed, “Would you mind having a quick test?” Why would you risk it not working! or something else going wrong… Customer service, remember you’re on the front line, if it’s not fixed you’ll just have the same manager or a bigger manager standing at your desk, this, in case you’re struggling to make the connection is “pro-active” shock!, horror! it’s sort of like a management buzz word but it just means you do a good job and get confirmation from the affected person or parties that you did do a good job.

3, Quite frankly, if you’re using this response you’ve got bigger issues with environment. Something catastrophic has gone wrong, emphasise if needed. Your aim here is to re-assure the person raising the issue “Oh, that is not good, I can see how this is affecting your ability to do your job” (I wouldn’t use those words, but you get the idea) Make sure that you escalate this to your boss so they can take care of the management with the business freeing you up to do what you do best, Grab coffee. I was half serious this time, You’re going to be at your desk a while, take 2 mins to sort your self out so you can focus on the issue. Make sure that someone (ideally your boss) is sending out regular updates, in the updates tell people when you will next update them and stick to these time frames; and remember when you start talking about the DNS cache being corrupted on one of the slave caches in the backup data centre affecting the transfer to the internet they are ….*yawning* Do 3 things:

Summarise in the subject / heading (yes this will not be a direct contact event…)

Give a one or two sentences at the top of the email with a tiny bit more detail

Knock your self out and write all that pent-up technological mumbo jumbo you really feel the urge to get on paper for the techno-geeks in the company to read, but please, do it at the bottom so as people become less interested in the technology they can stop reading higher up

Why was that so important?

Well, in short people like being informed, they like being given relevent information and they like it even more when you do what you say you are going to do.
If you can stick to doing those things and ensuring that you try to understand the real issue to the people affected you will be a better sysadmin for it. I realise it’s easy to build up the walls and start blaming other people or to become complacent around outages. If you were thinking, even for one moment “well it was their fault” … Bad sysadmin! really if it is any ones fault it is your fault. Why didn’t your monitoring pick it up? Why were you not able to send out an announcement before you had someone standing at your desk all red faced.

If you ever get anyone at your desk for something more than “How do I do…” something is not right. Try and be self critical and remember that you can not change other people (well you can but it’s much harder than you might think.)

The whole point about managing expectations is that people will start to have trust in you, the department and when you say 5 mins, they know it means 5 mins not 20. If you’re unfortunate enough to be in a position where your IT department already is the lowest of the low and people are always fuming about what you do and that you never deliver, well expectation management will help you out, in a time of crises people typically accept the authoritarian figure’s word so you can use this implied authority as a foot hold into fixing the bigger issue which is trust.

So who did burn down that building?

Well the most important thing here is that it wasn’t you, find some other sucker to pin the blame on, Why build up your reputation to knock it back down again! … Seriously?

Okay the most important thing is to actually not be part of any blame culture, it’s so counter-productive it’s not funny, you end up in a position where people will actively not do work because they won’t take the risk / blame if something goes wrong. If you start hearing someone blaming another department or you hear your self doing it. Stop them, interject, put the record straight. There’s a big difference between saying “Because Bob didn’t ask for the backups to be done we lost the data” and “when we were setting up the database our process wasn’t good enough to catch the fact we had missed the backups” “we missed the backups”, “Our process wasn’t..” key terms, after all even in a team of one it is a team effort between you and the rest of the business.

Obviously if you’re in the meeting and saying that the best continuation of that sentence is “But Ted has now checked the other database servers and we have also updated our process to catch this if it was to happen again”. Hang on, The problem was fixed and you didn’t take any glory, even though you spent an hour explaining to Ted how to even check if there was a back up and you re-wrote the entire process! Correct! If you want to be taken seriously and not destroy trust you’re busy building up in the rest of the business, don’t ever take the glory. People are smart enough to know that you were involved and doing a good job, and by thanking Ted publicly you’ve motivated a colleague and more importantly made your self not look self-centered. Which for those of you struggling… this means you gain more trust from the business as they believe you have brought into the bigger picture, what the company is trying to achieve.

In some case you would have used your best political speech to explain what the problem was and how it was fixed, for some reason people are still not happy. Ideally in this position your manager will speak up and take the blame, if you’re unfortunate to not be in this position, or you are the one that made the mistake, or maybe your colleague did but he is not taking the blame; Don’t be afraid to take some of the blame, “Sorry, I can see now that I should have double checked the backups”

Summary

• Be nice, have empathy with the users issues and really try to understand how this affects them and their day to day business.
• Don’t be afraid of talking face to face with people, if you can you should, people like talking to people, and they love talking to people that think they’re special and important.
• Do what you say you are going to do, no excuses, if you’re struggling to do things in the time frame you set out, check back in and expand the update, explain why it hasn’t happened yet
• Don’t blame others, Thank those that helped, Take no glory, And if necessary Take the blame

With thanks to: Oldonliner For the fancy image.

Puppet inheritance, The insane mine field

I love Puppet

I’m what you would call a newbie when it comes to puppet, I’ve be fortunate enough to work with it for the last 2 years but I only started using it 15 months back. I’ve used other configuration management tools in the past, and one day maybe I will bore you with the inner workings os RHN Satellite Server, but puppet was different.

For me it had great power over the RHN Satellite Server I had used, it allowed for me to have dynamic configuration that was structured and sensible, although the Satellite server allows you to replace certain variables it was nowhere near as powerful.

Luckily for me I had my own personal puppet guru in the form of Adrian Bridgett who has been working on puppet since the early days. It enabled for me to progress quite quickly through some of the less interesting features and straight into the world of classes and parameters.

As time progressed I learnt all about the different resources and various structures I could utilise, the various quirks of working with puppet, such as loops, if you ever need to do a loop, good luck. In short I have had the privilege of working with some very interesting puppet code created by people who knew puppet a lot better than I, all the time with a resource on hand to help me learn.

I hate duplication!

One of my pet peeves with puppet at the moment is the way you have to pass variables around to make things work and that proper inheritance of the classes is just so poor.

For example, The following is a simple set of classes that works with no issues.

init.pp

class frank ( $foo ="bar" ) {

include frank::bob

  class {
    "frank::joe":
    foo => $foo
  }
}

bob.pp

class frank::bob {

  file {
    "/var/lib/bob":
    ensure => "directory"
  }
}

joe.pp

class frank::joe ($foo="bob") {

  file {
    "/var/lib/bob/${foo}.txt":
    content => template("module/file.txt"),
    owner => "frank",
  }
}

Okay, so we have some class called frank, which takes a param $foo, includes a class bob and calls another class called joe and passes a param foo to it. This is a pretty standard way to pass variables around and it works flawlessly. The downside? It’s rubbish, on simple classes as Puppetlabs always shows you it’s fine you have a handful of parameters and it’s not too much hassle to maintain or they show you a very simple ntp class that doesn’t give you a feel for the more complicated modules you may be writing yourself. You can of course just reference the variable directly or not pass them into the child classes… Wrong If you are using these in templates you will end up having variables that are not in the scoped path for the template. You could reference the nicely scoped variable i.e. $frank::joe::foo this works as long you want the default or you can spend ages working out how to get the variables around, if you actually want to set the variable you need to call the class directly so you end up with a more complicated nodes manifest which isn’t the end of the world, but in the world of keep it simple less is more.

So imagine the above but where you want to pass in 25 or 40 parameters, believe it or not, we have some complicated configuration files and to help us keep the flexibility in the module as much as possible. You basically end up duplicating a lot of variables around, and your init.pp will be huge as a result.

It’s probably worth saying that we try to keep where possible the most simplest node manifest, so typically the largest node manifest would be maybe 10 lines.

So here’s a better way to achieve the same things with less configuration.

init.pp

class frank ( ) inherits frank::joe {

  include frank::bob
}

bob.pp

class frank::bob {

  file {
    "/var/lib/bob":
    ensure => "directory"
  }
}

joe.pp

class frank::joe ($foo="bob") {

  file {
    "/var/lib/bob/${foo}.txt":
    content => template("module/file.txt"),
    owner => "frank",
  }
}

One simple change, inherit! it makes so much sense, the init.pp has has everything that is in the frank::joe and some more, wouldn’t it be nice if that worked. Well it Doesn’t thanks to this and a number of other bugs, the inheritance within puppet just isn’t powerful enough, and more importantly doesn’t work as expected.

It seems the only way to get around this is the first method, which means that even if you are splitting your classes out you end up with duplication and a more confusing modules.

I encourage everyone to give their own personal view-point on this as it’s an area I’d like to learn more about.

Summary

I want an easy life, I want puppet modules that allow for proper inheritance and dependency management, I don’t want to be working around quirks of an application, I want to work with the application and I want it to be simple.

I am not a puppet expert, but I do think that you should be able to pass parameters through to the inherited class without having the variables re-defined at every level, What do you think?

Welcome to the 21st century

Today is the Day…

Today is the day I decided to enter the 21st century, I finally decided to sign up to twitter. Obviously being a Sysadmin I’ve heard of this “twitter” milarky for a while but I could never bring myself to sign up. I tend to leave signing up to social networks quite late as typically they are fads, MySpace anyone?

Either way, Twitter has proven it is not going anywhere, not now and not in the immediate future, so I am taking this brave new step into the 21st Century; part of my brave new step is also this blog. For those that don’t know I did use to have a blog but that was back in the days when it was cool to have one where as these days it is more of a necessity, especially if you want to be taken seriously as an authority on a subject. With that in mind, this Blog is formed; my aim is write things that are not as personal but hopefully more useful than my previous blogging attempts.

Does Technology help us

So joining the 21st century as I have was to fix a niggling hole I felt I had in my online presence, I felt the urge to communicate the projects I’m working on and the day to day challenges that are faced in rapidly expanding environments.

As I said earlier I was late to the boat on Twitter, I had a blog, I stopped blogging and now have returned to blogging, I was late to Facebook, I was however very quick on MySpace.

This all got me thinking, so often as a sysadmin you are faced with technical decisions and the balance between stability and innovation.

Which leads nicely onto “How important are your clients?” and “What problem are you actually trying to solve?” a lot of the time a technology decision can be made in the heat of battle or with out considering all of the facts, Our jobs as sysadmins is to make that technology decision that is the best fit for the current problem, not one that may happen at some point.

So “Does Technology help us?” The short answer is no. Technology is not the answer to a problem, it never has been it never will be. Oh my, shock horro, A sysadmin who said technology is not the answer! Correct.

“Why is that?” you may be thinking, well most problems in IT are created by the technology that has come before hand. It is rarely caused by the Client or Customer asking for more features, and even when the request comes from Clients or Customers they often are not asking for a largely complex solution, lets examine this a little bit.

So your Client has asked for… “I’d like to have my website available in multiple countries”
For anyone that thought…”Oh host a server in which ever country, bingo” Bad Sysadmin! It is a website, it already is available in multiple countries. Our job is to work out what the client means when they ask these questions, depending on the answers the solution should be derrived. Did they just want a .de domain name pointing to the site and a German site presented? Did they want extra resilience?

Even if the client comes back with “I’d really like more resilience around my web server” we need to think about the why they would ask that, has the site been unstable lately? Are they expanding? Are they getting slow load times in the USA?

Our job is to ask questions and to provide the simplest solution to that problem. It may just be that we need to implement some monitoring as a first step or a better back up strategy.

Granted not every solution will be simple, you may need to set up some geo load balancing multi-site international data centre, or simply copy the site to another box in another country. Either way, start of simple, add complexity as needed.

Summary

I guess the point I’m getting too is don’t create a solution because you want to play with some new technology, don’t come up with a solution with out working out what the real point of the question was.

Adding in technology complicates a solution, so you should only add it in if it fixes a problem. In larger environments where you are supporting hundreds of servers you may need tools like Puppet, Chef, OpsView etc etc but in smaller environments you do not always need them.

The only exception to this is if you think there will be a need for it in the medium term, if you are rolling out a new solution that requires multiple environments and potentially will grow rapidly, then adding in these tools is a good start, but please start them out as simple as possible, you have all the time in the world to improve upon them later, don’t get caught up in technology for technologies sake, think simplicity, agility and rapid deployment.