Bash-off – a way to relieve your self

Bash off!

In our team at work we have this concept which we call a “Bash off” the idea is simple. Take something that you could do very simply in bash in several steps with manual fudging in the middle to end up with a result. The sort of thing that may take you an hour to brute force your way through. Now in your programming language of choice, automate the entire process.

Sounds simple right ? I remember the first task we had which was to grab two unique columns of data and to stream them onto the screen, I think it was originally done with watch, tail and tee, took about 20 mins of playing with bash. As a team we chose our own languages and went for it, an hour later we all had a working prototype, another hour later we all had our programs really efficient.

Obviously these are really pointless, but they do have a couple of benefits that help a DevOps person stay good at what they do. It gets you to do more with your language of choice than you normally would and it also causes you to think about how you’re structuring code to make it more efficient, it also helps bond the team together with a bit of healthy competition.

To clarify just because it is called a bash off that dosen’t mean that a solution can’t be in bash. So if it was a very manual human process that you believe you can fully automate in bash, Go for it :)

The challenge

This weeks challenge came from our friends in finance, they have a spreadsheet with some 14k rows in it, two of the columns have many duplicated fields in them and the have a one to many mapping between them. So the challenge is to get from 14k rows down to just the unique entries in each of these two columns and to then make sure the mapping in the spreadsheet can be looked up from a database (where the spreadsheet results must end up)

Due to some meetings I was late to the game on this task, pesky meetings; we had a bash driven prototype which we excluded because it required the excel spreadsheet to be turned into a csv and a python one that sort of got some fields from the spreadsheet and pushed them into the DB.

This is where I picked it up, sort of working; I’ve not done a lot of python so I forked the code and me and my boss went our separate ways to achieve the task, in the end it took us a couple of hours to get this working and for us to have the same results. All in all it we had achieved the processing of 14k rows of data and the manipulation into a db with the correct data, but was that enough? No We decided that it taking about 1 min was not good enough so we started focusing on making it better, I think my run time was 58 seconds and my bosses 40.

We had both chosen different ways of doing it, I had chosen to use the DB to ensure the fields were unique by checking if that field existed in the DB and if not to create it and returned the id, if it was in the DB it would return the id. My bosses approach was better, he created lists with the data in and then made the lists unique. I decided I had to get mine down to a similar speed so I started hacking it around; I decided I would store the unique entries in a list for each table and then before calling the method that puts the data in the database I would check if that unique value was already there. The first issue with this is I lost the ID number which I needed to add to the lookup table so I had to change the list to a dictionary.

I also found by moving the commit messages for the database out side of any of the methods and just dropping it at the end saved a few seconds, I was also able to remove a couple of additional DB queries all of this helped bring the time down. One of the best changes I did was on the lookup table when doing a query only pulling back one ID, rather than two; I didn’t even need it but I couldn’t see a way of querying sql alchemy with out having a field to bring back that would be quick.

As the night progressed we both made good progress, my Boss got his down to 14.1 seconds I managed to get mine down to 21 seconds so we had both made massive improvements in our codes efficiency. My boss was making use of gevent but when I tried this it slowed my program down so I left it out not understanding it anyway. I kept pluggin away and I made it down to 13.5 seconds.

Summary

I urge you to take an afternoon out in your team and to push your skills forward with a programming challenge and to see what happens, it will make you better, you will learn stuff, it is fun, it does bond the team and you will enjoy it, it is also a waste of time but what else were you going to do ?

Yummy, Jira Jelly Escalations

Who doesn’t like Jelly?

At work we have some SOC2 fun stuff to attend to, one of the areas that I’m looking at is incident management, we’ve been doing incident management and analysis of incidents for a while so this was really about formalising it all. We use a highly customisable ticketing system called Jira which is good at its job although getting to grips with its configuration can be tricky.

One of the areas we have to ensure process is followed is with escalation of incident tickets, our incident tickets are identified with an issue type of “Incident” and we have a special “incident priority” field which is simple P1, P2 or P3

So the challenge for us was how do we make sure that each tier of escalation gets notified and that it is clearly documented the ticket has been escalated?

This is where Jelly comes in, Jira has Jelly which is an XML defined way of doing programatic actions within Jira, a good link to open and read later will be this one all about Jira:Jelly and if like me who only found it afterwards these will be useful docs for Jelly: email http SOAP SQL core

Defining Filters

To get the escalations to work I used filters to show me all tickets that matched a certain pattern, so to do the first tier of escalation the filter looks for all tickets in the project and that are of the issue type “incident” and have an incident priority of “P1” and are older than X mins. To stop this coming up more than once the filter was also given some extra parameters to only show when it wasn’t assigned to the current escalation person or one of the higher up escalations. If it was assigned to anyone else it would then be escalated.

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -30m AND
assignee not in (escalation1, escalation2, escalation3)

I won’t explain this too much, but “PROJECTID” will be the short code for the project in jira, and the escalations at the end are the usernames of those in the escalation tiers.

Here’s tier 2 as well so you can see how it filters for the next tier:

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -45m AND
assignee in (escalation1) AND
assignee not in (escalation2, escalation3)

and the third for completeness:

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -75m AND
assignee in (escalation2) AND assignee not in (escalation3)

As long as “assignee in” has the person of the previous tier in it and the “assignee not in” has those still left to be assigned to it should work okay…

All these filters do is show tickets at various stages of the escalation process, because of that it is possible to associate that with a jira jelly script to actually carry out actions on it.

Jira Jelly

Remember that link earlier? the one to the Jelly page? well that will help with the others above… Now you need to know what to do.

You have a way of filtering out the tickets so you can now write a script to do something at each stage of the escalation as needed. In our case we need to maker sure the user receives a notification and that a comment appears in the ticket so it is clear it has been updated.

The first attempt of this was to simply add a comment with the newish @user mentioning system which then automatically lets the user know they were mentioned. Unfortunately when using the add comment method it simply puts this in as text and when expanding the short hand to the full [~user] it just entered as text which means this method was not viable for us so instead we just simply assigned the ticket which has the affect of also adding the person to the watchers.

To do this sensibly it requires a few steps, a Login, the use of the filter to find the relavent tickets, a comment added and an assignment to happen.

<JiraJelly xmlns:jira="jelly:com.atlassian.jira.jelly.enterprise.JiraTagLib" xmlns:core="jelly:core" xmlns:log="jelly:log" >
  <log:info>Running 'Escalation for Incidents - t1' service</log:info>
  <jira:Login username="escalationuser" password="XXXXXXX">
    <!--  Escalation People -->
    <core:set var="escalate" value="escalation1" />
    <!-- Run the SearchRequestFilter Filter is Open PROJECTID P1's-->
    <jira:RunSearchRequest filterid="FILTERID" var="issues" />
    <core:forEach var="issue" items="${issues}">
      <log:warn>Escalating ${issue.key}</log:warn>
      <jira:AssignIssue key="${issue.key}" assignee="${escalate}"/>
      <jira:AddComment issue-key="${issue.key}" comment="Escalating to ${escalate}"/>
    </core:forEach>
  </jira:Login>
</JiraJelly>

The filterid can be found by simply viewing the filter and copying the number from the end of the URL, hopefully everything else is self explanatory. for each escalation simply copy the file, update the filter ID and the escalation user

Install

1, Copy files to the server and put somewhere sensible that tomcat has access too, ensure the files are owned by tomcat and that tomcat can write to that directory.
2, In Jira, go to Administration -> System -> Advanced -> Services and add a new service, give it a name and select the built in jelly runner. Enter the location of the script and provide a log location.

That’s it. I noticed that having server access was invaluable, mainly because I was using VI and some syntax errors made it in which were hard to spot with out seeing the log, you can test your scripts using the Jelly runner which is also useful.

In short that’s it, hopefully that will be useful for people :)

Time for an idea

Why not

It’s been a while since I’ve thrown myself into an idea and tried to come out the other side, so I’ve spent the last couple of days just thinking about what’s missing. It doesn’t take much to have an idea; but making sure it’s a good idea, making sure it is unique in it’s offering and making sur eit’s better than anything else is not easy.

At work we are working on an idea, a concept of some sort of DevOps tool that takes a lot of what we do already and simplifies it and merges multiple tools into one place, the driving goal is easy of use, take an entire system, data centre what ever you want and within minuets you’ll have the whole thing monitored, feeding metrics back for reporting, performing real time analysis and trending. It’s still very much prototype phase but it’s a very exciting project that wraps up several elements that we as a team are passionate about, ease of use, efficiency, performance, monitoring, measurements and of course, cool technology; but with that said I still have this urge to do something else, I’m not really sure why, I’m busy enough as it is but I feel like the world is missing something that is more than just an amalgamation of parts, or a re-skin of an existing thing, I feel like it’s missing something, the question is as always what?

There’s a saying “All the good ideas are gone” probably true, but that doesn’t stop people striving for new things, look at Glass I’m not convinced it has a long term future in that styling, but wearable tech certainly does, look at this wrist computer from the tv show chuck, just what I’ve always wanted.

a lot of the best ideas today are based on things that have come before and re-envisioned, walkman -> ipod; iPaq -> iPad; alta vista -> google

Just because it’s been done in a similar way before doesn’t mean you can’t do it better, or take from their ideas and make it work and half of the battle is the conviction to want to do it better or different. Which is why everyone should try a new idea and everyone should try something new, to make something better.

What to do

This is the bit I’m struggling with, and it’s the hardest bit of the whole thing, for me it’s not good enough to take an idea and make it better, if someone gives you a product and says make it better it wouldn’t take long for a few ideas to bubble up. I’m thinking more along the lines of taking some wacky out there thinking and making it a reality in a way that works and works well.

I think over the next few weeks I’m just going to just write some things down and see which ones stand out, which ones seem stupid/crazy to do and then probably come up with one that works.

I’m not really sure what that is at all, I could take something like sentinel and munge that into something else, but it just doesn’t feel like the right idea. I had an idea a long time ago, probably 3 years ago, which I talked my self out of because it would take me forever to make and I didn’t have the skills, but things have changed, it isn’t even ground breaking it’s just another internet site.

Either way I’ll keep on plodding along for now and see if I can come up with something but until then much more scribbling on paper and throwing things at a wall.