Yummy, Jira Jelly Escalations

Who doesn’t like Jelly?

At work we have some SOC2 fun stuff to attend to, one of the areas that I’m looking at is incident management, we’ve been doing incident management and analysis of incidents for a while so this was really about formalising it all. We use a highly customisable ticketing system called Jira which is good at its job although getting to grips with its configuration can be tricky.

One of the areas we have to ensure process is followed is with escalation of incident tickets, our incident tickets are identified with an issue type of “Incident” and we have a special “incident priority” field which is simple P1, P2 or P3

So the challenge for us was how do we make sure that each tier of escalation gets notified and that it is clearly documented the ticket has been escalated?

This is where Jelly comes in, Jira has Jelly which is an XML defined way of doing programatic actions within Jira, a good link to open and read later will be this one all about Jira:Jelly and if like me who only found it afterwards these will be useful docs for Jelly: email http SOAP SQL core

Defining Filters

To get the escalations to work I used filters to show me all tickets that matched a certain pattern, so to do the first tier of escalation the filter looks for all tickets in the project and that are of the issue type “incident” and have an incident priority of “P1” and are older than X mins. To stop this coming up more than once the filter was also given some extra parameters to only show when it wasn’t assigned to the current escalation person or one of the higher up escalations. If it was assigned to anyone else it would then be escalated.

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -30m AND
assignee not in (escalation1, escalation2, escalation3)

I won’t explain this too much, but “PROJECTID” will be the short code for the project in jira, and the escalations at the end are the usernames of those in the escalation tiers.

Here’s tier 2 as well so you can see how it filters for the next tier:

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -45m AND
assignee in (escalation1) AND
assignee not in (escalation2, escalation3)

and the third for completeness:

project = PROJECTID AND issuetype = Incident AND
status in (Open, "In Progress", Reopened) AND
"Incident Priority" = P1 AND created <= -75m AND
assignee in (escalation2) AND assignee not in (escalation3)

As long as “assignee in” has the person of the previous tier in it and the “assignee not in” has those still left to be assigned to it should work okay…

All these filters do is show tickets at various stages of the escalation process, because of that it is possible to associate that with a jira jelly script to actually carry out actions on it.

Jira Jelly

Remember that link earlier? the one to the Jelly page? well that will help with the others above… Now you need to know what to do.

You have a way of filtering out the tickets so you can now write a script to do something at each stage of the escalation as needed. In our case we need to maker sure the user receives a notification and that a comment appears in the ticket so it is clear it has been updated.

The first attempt of this was to simply add a comment with the newish @user mentioning system which then automatically lets the user know they were mentioned. Unfortunately when using the add comment method it simply puts this in as text and when expanding the short hand to the full [~user] it just entered as text which means this method was not viable for us so instead we just simply assigned the ticket which has the affect of also adding the person to the watchers.

To do this sensibly it requires a few steps, a Login, the use of the filter to find the relavent tickets, a comment added and an assignment to happen.

<JiraJelly xmlns:jira="jelly:com.atlassian.jira.jelly.enterprise.JiraTagLib" xmlns:core="jelly:core" xmlns:log="jelly:log" >
  <log:info>Running 'Escalation for Incidents - t1' service</log:info>
  <jira:Login username="escalationuser" password="XXXXXXX">
    <!--  Escalation People -->
    <core:set var="escalate" value="escalation1" />
    <!-- Run the SearchRequestFilter Filter is Open PROJECTID P1's-->
    <jira:RunSearchRequest filterid="FILTERID" var="issues" />
    <core:forEach var="issue" items="${issues}">
      <log:warn>Escalating ${issue.key}</log:warn>
      <jira:AssignIssue key="${issue.key}" assignee="${escalate}"/>
      <jira:AddComment issue-key="${issue.key}" comment="Escalating to ${escalate}"/>
    </core:forEach>
  </jira:Login>
</JiraJelly>

The filterid can be found by simply viewing the filter and copying the number from the end of the URL, hopefully everything else is self explanatory. for each escalation simply copy the file, update the filter ID and the escalation user

Install

1, Copy files to the server and put somewhere sensible that tomcat has access too, ensure the files are owned by tomcat and that tomcat can write to that directory.
2, In Jira, go to Administration -> System -> Advanced -> Services and add a new service, give it a name and select the built in jelly runner. Enter the location of the script and provide a log location.

That’s it. I noticed that having server access was invaluable, mainly because I was using VI and some syntax errors made it in which were hard to spot with out seeing the log, you can test your scripts using the Jelly runner which is also useful.

In short that’s it, hopefully that will be useful for people :)

Applying AMPs to Alfresco

A bit of background

Alfresco comes with the ability to be extended in a nice easy way, that is through the use of Alfresco Module Packages. In essence it is the delta of changes you would have made to the raw source code if you wanted to make some sort of customisation or apply one of the ones alfresco supplies like the S3-connector.

Over the last 3 years I’ve seen it done a number of ways, using the mmt tool to apply them manually, a shell script to do it and now I decided that wasn’t good enough.

Using the mmt tool manually is obviously not brilliant, some poor person has to sit there and run it to apply the amps. So you may have guessed this is not a good idea.

What about wrapping the mmt tool in a shell script that can be triggered by say a sysadmin to apply all the amps or just have it executed once per amp using some configuration managent tool like puppet. This is good. You put the amp into the configuration management tool push the right buttons and it magically get’s applied to the war files and all is well. Well sort of, what happens if someone just throws an amp on the server? who puts it in configuration management? who’s made a backup? Well I decided that I’d write a new script for applying amps so that it can be used both with a CM and as a ad-hock script.

What does it do?

I’ve written it so it will trawl through a directory and pull up every amp in the directory and it will apply the amps to alfresco or share as needed. What’s quite handy is that it will take several versions of an amp and work out what the latest version is, it will check the latest version against what is already installed in the war and then if the amp is a newer version it will apply it after making a backup.

For some odd reason I also made it cope with a variety of amp naming schemes, so you could upload alfresco-bob-123.amp or you could upload frank-share-super-1.2.3.4-5.amp it’s your amp, call it what you want. All the script cares about is the correlation of terms between the file name and the amp info when it’s installed. So as long as you use 2 words from the file name that also appear in the amp description it will work it out for you. The higher the correlation the more accurate it would be, it is configurable too but I set it to 2 occurrence of at least 2 words to match, so far… it’s working.

I also forgot to mention that the script will stop alfresco clear the caches and restart it for you in a pretty safe way.

A Script

Firstly I realise this is a bad format to get the script I’ve in the past put them in a git repo and shared it that way, I have put this one in a git repo and I hope to share that repo with some of the things we have done at alfresco that are useful for either running servers in general or for running alfresco either way I hope to shortly get that out on a public repo but for now here it is:

#!/usr/bin/ruby
# Require libs
$:.unshift File.expand_path("../", __FILE__)
require 'lib/logging'
require 'fileutils'
require 'timeout'

# Set up logging provider
Logging::new('alfresco_apply_amps')
Logging.log_level("INFO",false)
$log.info "Starting"

#CONSTANTS
BKUP_LOCATION="/var/lib/alfresco/alf_data/backups"
ALF_MMT="/var/lib/tomcat6/bin/alfresco-mmt.jar"
WEBAPPS_DIR="/var/lib/tomcat6/webapps"
AMP_LOCATIONS=["/var/lib/alfresco/alf_data/amps/"]

#Defaults
@restart=false

#Methods
def available_amps(amp_dir)
  #Get a list of Amps
  amps_list = `ls #{amp_dir}*`
  amps_array = amps_list.split("\n")
end

def backup(war)
  version=`/usr/bin/java -jar #{ALF_MMT} list #{WEBAPPS_DIR}/#{war}.war | grep Version | awk '{print $3}'`

  #Date stamp the war
  $log.info "Backing up #{WEBAPPS_DIR}/#{war}.war to #{BKUP_LOCATION}/#{war}-#{current_date}.war"
  `cp -a #{WEBAPPS_DIR}/#{war}.war #{BKUP_LOCATION}/#{war}-#{current_date}.war`
end

def clear_caches()
  $log.debug "Cleaning caches"  
  delete_dir("#{WEBAPPS_DIR}/alfresco/")
  delete_dir("#{WEBAPPS_DIR}/share/")
  delete_dir('/var/cache/tomcat6/work/',true)
  delete_dir('/var/cache/tomcat6/temp/',true)
  $log.info "Caches cleaned"  
end

def compare_strings(str1,str2,options={})
  matches = options[:matches] || 2
  frequency = options[:frequency] || 2

  #Make one array of words
  words=Array.new
  words << str1.split(' ') << str2.split(' ')
  words.flatten!
  #Hash to store each unique key in and number of occurances
  keys = Hash.new
  words.each do |key|
    if keys.has_key?(key)
      keys[key] +=1
    else
      keys.merge!({key =>1})
    end
  end

  #Now we have a Hash of keys with counts how many matches and what frequency
  #where a match is a unique key >1 and frequency si the count of each key i.e. 
  #matches=7 will mean 7 keys must be >1 frequency=3 means 7 matches must be > 3
  
  act_matches=0
  keys.each_pair do |key,value|
    if value >= frequency
      act_matches +=1
    end
  end
  if act_matches >= matches
    true
  else
    false
  end
end

def compare_versions(ver1,ver2)
  #return largest
  v2_maj=0
  v2_min=0
  v2_tiny=0
  v2_release=0
  v1_maj=0
  v1_min=0
  v1_tiny=0
  v1_release=0
  if ver1 =~ /\./ && ver2 =~ /\./
    #both are dotted notation
    #Compare maj -> release

    #Conver '-' to '.'
    ver1.gsub!(/-/,'.')
    ver2.gsub!(/-/,'.')

    v1_maj = ver1.split('.')[0]
    v1_min = ver1.split('.')[1] || 0
    v1_tiny = ver1.split('.')[2] || 0
    v1_release = ver1.split('.')[3] || 0

    v2_maj = ver2.split('.')[0]
    v2_min = ver2.split('.')[1] || 0
    v2_tiny = ver2.split('.')[2] || 0
    v2_release = ver2.split('.')[3] || 0

    if v1_maj > v2_maj
      return ver1
    elsif v1_min > v2_min
      return ver1
    elsif v1_tiny > v2_tiny
      return ver1
    #Don't compare release for now as some amps don't put the release in the amp when installed so you end up re-installing
    #elsif v1_release > v2_release
    #  return ver1
    else
      return ver2
    end
  else
    #Validate both are not-dotted
    if ver1 =~ /\./ || ver2 =~ /\./
      $log.debug "Eiher both types aren't the same or there's only one amp"
      return ver2
    else
      result = ver1<=>ver2
      if result.to_i > 0 && !result.nil?
        return ver1
      else
        return ver2
      end
    end
  end
end

def current_date()
  year=Time.now.year
  month=Time.now.month
  day=Time.now.day
  if month < 10
    month = "0"+month.to_s
  end
  if day < 10
    day = "0"+day.to_s
  end
  "#{year.to_s+month.to_s+day.to_s}"
end

def current_version(app, amp_name)

#
# THIS needs to cope with multiple amps being installed, produce a array hash [{:amp=>"ampname",:version => ver},etc]
#

  if app == "alfresco" || app == "share"
    amp_info = `/usr/bin/java -jar #{ALF_MMT} list #{WEBAPPS_DIR}/#{app}.war`
    amp_title=""
    amp_ver=0
    #$log.debug "Amp info: #{amp_info}"
    amp_info.each_line do |line|
      if line =~ /Title/
        amp_title=line.split("Title:").last.strip.gsub(%r/(-|_|\.)/,' ')
      elsif line =~ /Version/
        # strip/replace ampname, downcase etc
        if compare_strings(amp_name.gsub(%r/(-|_|\.)/,' ').downcase,amp_title.downcase)
          amp_ver=line.split("Version:").last.strip
          $log.info "Installed Amp found for #{amp_name}"
          $log.debug "Installed version: #{amp_ver}"
        else
          $log.debug "No installed amp for #{amp_name} for #{app}"
        end
      end
    end
  else
    $log.warn "The application #{app} can not be found in #{WEBAPPS_DIR}/"
  end
  return amp_ver
end

def delete_dir (path,contents_only=false)
  begin
    if (contents_only)
      $log.debug "Removing #{path}*"
      FileUtils.rm_rf Dir.glob(path+"*")
    else
      $log.debug "Removing #{path}"
      FileUtils.rm_rf path
    end
  rescue Errno::ENOENT
    $log.warn "#{path} Does not exist"
  rescue Erro::EACCES
    $log.warn "No permissions to delete #{path}"
  rescue
    $log.warn "Something went wrong"
  end
end

def firewall(block=false)
  if block
    `/sbin/iptables -I INPUT -m state --state NEW -m tcp -p tcp  --dport 8080 -j DROP`
  else
    `/sbin/iptables -D INPUT -m state --state NEW -m tcp -p tcp  --dport 8080 -j DROP`
  end
end

def get_amp_details(amps)
  amps_hash = Hash.new
  amps.each do |amp|
    amp_hash = Hash.new
    #Return hash with unique amps with just the latest version
    amp_filename = amp.split("/").last
    amp_path = amp
    amp_name=""
    amp_version=""
    first_name=true
    first_ver=true
    #Remove the ".amp" extension and loop through
    amp_filename[0..-5].split("-").each do |comp|
      pos = comp =~ /\d/
      if pos == 0
        if first_ver
          amp_version << comp
          first_ver=false
        else
          #By commenting this out the release will get ignored which because some amps to put it in their version is probably safest
          #amp_version << "-" << comp
        end
      else
        if first_name
          amp_name << comp.downcase
          first_name=false
        else
          amp_name << "_" << comp.downcase
        end
      end
    end

    #If a key of amp name exists, merge the version down hash else merge the lot
    if amps_hash.has_key?(amp_name)
      amp_hash={amp_version => {:path => amp_path, :filename => amp_filename}}
      amps_hash[amp_name].merge!(amp_hash)
    else 
      amp_hash={amp_name =>{amp_version => {:path => amp_path, :filename => amp_filename}}}
      amps_hash.merge!(amp_hash)
    end
  end
  return amps_hash
end

def install_amp(app, amp)
  $log.info "applying amp to #{app}"
  $log.warn "amp path must be passed!" unless !amp.nil?

  $log.debug "Command to install = /usr/bin/java -jar #{ALF_MMT} install #{amp} #{WEBAPPS_DIR}/#{app}.war -nobackup -force"
  `/usr/bin/java -jar #{ALF_MMT} install #{amp} #{WEBAPPS_DIR}/#{app}.war -nobackup -force`
  restart_tomcat?(true)
  $log.debug "Setting flag to restart tomcat"
end

def latest_amps(amp_hash)
  amp_hash.each_pair do |amp,amp_vers|
    latest_amp_ver=0
    $log.debug "Comparing versions for #{amp}"
    amp_vers.each_key do |version|
      $log.debug "Comparing #{latest_amp_ver} with #{version}"
      latest_amp_ver = compare_versions(latest_amp_ver,version)
      $log.info "Latest version for #{amp}: #{latest_amp_ver}"
      if latest_amp_ver != version
        amp_vers.delete(version)
      end
    end
  end
  return amp_hash
end

def next_version?(ver, current_ver, app)
  #Loop through amp versions to work out which is newer than the installed
  #Turn list into array
  next_amp=false
  $log.debug "if #{ver} > #{current_ver}"
  if ( ver.to_i > current_ver.to_i)
    $log.debug "Next #{app} amp version to be applied:  #{ver}"
    next_amp=true
  end
end

def restart_tomcat()
  #If an amp was applied restart
  if (restart_tomcat?)
    $log.info "Restarting Tomcat.... this may take some time"
    $log.debug"Getting pid"
    if (File.exists?('/var/run/tomcat6.pid') )
      pid=File.read('/var/run/tomcat6.pid').to_i
      $log.debug "Killing Tomcat PID= #{pid}"
      begin
        Process.kill("KILL",pid)
        Timeout::timeout(30) do
          begin
            sleep 5
            $log.debug "Sleeping for 5 seconds..."
          end while !!(`ps -p #{pid}`.match pid.to_s)
        end
      rescue Timeout::Error
        $log.debug "didn't kill process in 30 seconds"
      end
    end
    $log.debug "Killed tomcat"

    #Clear caches
    clear_caches
    $log.info "blocking firewall access"
    firewall(true)
    $log.debug "starting tomcat"
    `/sbin/service tomcat6 start`
    if ($?.exitstatus != 0)
      $log.debug "Tomcat6 service failed to start, exitstatus = #{$?.exitstatus}"
    else
      #Tomcat is starting sleep until it has started
      #For now sleep for 180 seconds
      $log.info "Sleeping for 180 seconds"
      sleep 180
      $log.info "un-blocking firewall access"
      firewall(false)
    end
  else
    $log.info "No new amps to be installed"
  end
end

def restart_tomcat?(bool=nil)
  @restart = bool unless bool.nil?
  #$log.debug "Restart tomcat = #{@restart}"
  return @restart
end

# - Methods End

#
# doGreatWork()
#

#Store an Hash of amps
amps=Hash.new

#For each AMP_LOCATIONS find the latest Amps
AMP_LOCATIONS.each do |amp_loc|
  $log.debug "Looking in #{amp_loc} for amps"
  amps.merge!(get_amp_details(available_amps(amp_loc)))
end

#Sort through the array and return only the latest versions of each amp
latest_amps(amps)

amps.each do |amp, details|
  #The Amps in here are the latest of their kind available so check with what is installed
  details.each_pair do |version,value|
    if amp =~ /share/
      if next_version?(version,current_version("share",amp),"share")
        $log.debug "Backing up share war"
        backup("share")
        $log.info "Installing #{amp} (#{version}): #{value[:path]}"
        install_amp("share",value[:path])
      else
        $log.info "No update needed"
      end
    else
      if next_version?(version,current_version("alfresco",amp),"alfresco")
        $log.debug "Backing up alfresco war"
        backup("alfresco")
        $log.info "Installing #{amp} (#{version}): #{value[:path]}"
        install_amp("alfresco",value[:path])
      else
        $log.info "No update needed"
      end
    end
  end
end

$log.debug "Restart tomcat?: #{restart_tomcat?}"
restart_tomcat

$log.info "All done for now"

Okay 2 things, it’s a long script all in one file to make it easy to transport, I’ve also used a logging class that enables logging to screen / file that is …below :) you could also just remove the require at the top and replace “$log.debug” with “puts” up to you.

#
#   Set up Logging
#

require 'rubygems'
require 'log4r'

class Logging

  def initialize(log_name,log_location="/var/log/")
    # Create a logger named 'log' that logs to stdout
    $log = Log4r::Logger.new log_name

    # Open a new file logger and ask him not to truncate the file before opening.
    # FileOutputter.new(nameofoutputter, Hash containing(filename, trunc))
    file = Log4r::FileOutputter.new('fileOutputter', :filename => "#{log_location}#{log_name}.log",:trunc => false)

    # You can add as many outputters you want. You can add them using reference
    # or by name specified while creating
    $log.add(file)
    # or mylog.add(fileOutputter) : name we have given.

    # As I have set my logging level to ERROR. only messages greater than or 
    # equal to this level will show. Order is
    # DEBUG < INFO < WARN < ERROR < FATAL

    # specify the format for the message.
    format = Log4r::PatternFormatter.new(:pattern => "[%l] %d: %m")

    # Add formatter to outputter not to logger. 
    # So its like this : you add outputter to logger, and add formattters to outputters.
    # As we haven't added this formatter to outputter we created to log messages at 
    # STDOUT. Log messages at stdout will be simple
    # but the log messages in file will be formatted
    file.formatter = format
    
  end

  def self.log_level(lvl,verbose=false)
    # You can use any Outputter here.
    $log.outputters = Log4r::Outputter.stdout if verbose

    # Log level order is DEBUG < INFO < WARN < ERROR < FATAL
    case lvl
        when    "DEBUG"
            $log.level = Log4r::DEBUG
        when    "INFO"
            $log.level = Log4r::INFO
        when    "WARN"
            $log.level = Log4r::WARN
        when    "ERROR"
            $log.level = Log4r::ERROR
        when    "FATAL"
            $log.level = Log4r::FATAL
        else
             print "You provided an invalid option: #{lvl}"
    end
  end

end

I hope this helps people out, if there’s any issues just leave comments and i’ll help :)

Updated Alfresco Solr Checks

As some may know…

A little while back I put up some checks for Alfresco Solr Here and wrote a little blog Here

Well over the last few weeks I have added yet more checks to it and I’ve also added some caching of the results so it will now no longer make a separate request to solr for each check and instead will use a local cached copy of the results and after 5 mins get a new one. The reason for this is that most of the results don’t change that frequently and with nagios it was calling each check so 20 calls to solr over a 5 min period, well each individual check is only verified once every 5 mins so now it will pull the report once and reference that cached copy for 5 mins, after that it will simply pull a new one…

In addition to the caching it now has 13 new checks! including cumulative hit ratios which are typically more relavent than the normal hit ratios as they are based over all time (Since reboot) and no, I don’t know how long the normal hitratios are based over.

There is also some checks for the number of Transactions remaining and the number of change sets remaining, these combined with the Lag can give you an indication of how far behind / how much work is left for Solr to do so quite useful.

If you need any help with these or have a few additional checks that are relavant let me know I’m happy to help.