Secure salted passwords with ruby

Step one, Understand

So I’ve been playing with Sinatra and Redis over the past few months and as part of my more professional side I am creating a blog platform for my other website and as a result I wanted user authentication to ensure I and only I could update it, there’s a chance that at some point I may want to allow others to sign up and login but quite frankly not yet and this is over kill but none the less we learn an develop, so here’s the first step in building it from scratch.

Understanding some key concepts when it comes to authentication is key, so first some history lessons and why it is a bad idea to use those approaches today. Back in the day, way back when, people were trusting and assumed no evil of this world, we call these people naive; predominantly they relied on servers being nice and snug behind firewalls and locked cabinets. As such the passwords were saved in plain text in a database, or a file who cares, it’s human readable, so the attack on this is trivial. Let’s assume you are not silly enough to run the database over a network with no encryption and are instead running it locally well if I have access to your machine I will probably find it in a matter of minuets and if I have physical access within an hour it’s mine, all of them. On a side note, if you’re using plain text passwords anywhere you’re an idiot or in early stages of testing…

After realising this approach was bad, people thought, I can protect this and I will hash the password! wonderful, so you use md5, or sha it doesn’t matter which but for this example let’s say you chose md5. What you have done here is very cunningly created a password that is not human readable. However, be ashamed of your self if you still think this is secure, and here’s why. There are a lot of combinations (2^128 = 340 trillion, trillion, trillion) which to be honest is a lot! the chances of two clashes are so slim why worry about it! Wrong just plain wrong. So here’s why it’s bad, 1, people are idiots and for some strange reason we use typically words to make up passwords so if your password was “fridge” I’d imagine, what this means is if I as mr Hacker get hold of your DB I sit there while I run a dictionary style attack generating thousands of common words into md5 sums and before long I have a word that matches the same md5 sum as your password see this what’s better is because you’re a human it’ll probably work on all sites you use, Nice. The second problem is I don’t have to actually do that grunt work, people have already done it for me and they are called Rainbow tables so trivially I can download it and just do a simple query to find a phrase that matches yours. Don’t think it’s an issue, LinedIn did they fixed it perfectly.

Excellent, So now we’re getting to a point where we need something better, and this is where Salts come in. The basic concept is you type in a password and I as a server concatenate a random string to it and generate a hash. This over comes the rainbow table, because the likely hood of someone having generated a rainbow table from my random salt is highly unlikely, certainly a lot less likely than 2^128. However, lets assume I’m big web provider that got hacked and lost everyones password, “Ha! I used a salt, Good luck cracking that!” they say. Sure, a few points, 1, the salt is stored in the DB, 2, Computers are quicker than they use to be. With the advances in graphic processor cracking and amazon boxes Password cracking is more or less a matter of time & money, but how much is the key.

Using a single salt on all passwords is still bad, the best that I know of today is to use a unique salt for every password. Even if all users have the same password they all have unique hashes, and this is the corner stone. For every user that joins up, generate a secure random salt, add it to the password and generate a hash. So for every user a massive amount of time would be needed to crack just one password, so unique hashes very good.

The code

So history lesson over, now some code; as identified above the best way was to generate a unique salt for every user and then hash their password. This isn’t hard but you do need to ensure the salt is securely random, else you may just be generating something not as secure as you thought.

Have a look at this gist Auth.rb, so useful point, the salt should be large, so if you produce a 32 bytes hash, your salt should be at least 32 bytes as well. It’s a straight forward lib that will generate and allow you to get passwords back out.

Here’s an example of it in use.

require_relative 'auth'
#This is from the web front end so you can see how to use the Lib to check a hash
def user_auth_ok?(user,pass)
  #Get user's Hash from DB
  user_hash = $db.get_user_hash(user)
  #Validate hash
  if Auth.hash_ok?(user_hash,pass)
    #User authenticated
    return true
  else
    #user is not authenticated
    return false
  end
end

#As A side point This bit os from the backend that writes it to the DB 
#If it was the chunk from what the website used it would simply call a method that took username and password
#so probably not useful...
def add_user(username,password)
  #Poke the appropriate keys, create default users, example post etc.
  uid = @redis.incr('users')
  salt = Auth.gen_salt()
  hash = Auth.gen_hash(salt,password)
  $logr.debug("Salt = #{salt}, hash = #{hash}")
  @redis.set("users:#{uid}:#{username}",hash)
end

def edit_user_password(username, password, newPassword)
  user = @redis.keys("users:*:#{username}")
  if user.size < 1
    $logr.debug("No user #{username} found")
  else
    #Validate old password
    hash = @redis.get(user[0])
    if Auth.hash_ok?(hash, password)
      $logr.info("Setting a new password for #{username}")
      newHash = Auth.gen_hash(Auth.get_salt(hash),newPassword)
      @redis.set(user[0],newHash)
    else
      $logr.info("password incorrect")
      #TODO - Need exception classes to raise Auth failure
    end
  end
end

Sinatra – partial templates

Singing a different song

Firstly apologies, it’s been over a month since my last blog post but unfortunately with holidays, illness and change of jobs I’ve been struggling to find any time to write about what I’ve been doing.

A few months back I did a little research into micro web frameworks, did quite a bit of reading around Sinatra, Bottle and Flask. To be honest they all seem good, and I want to play with Flask or bottle at some point too, but Sinatra is the one I’ve gone for so far as he documentation was the best and it seemed the easiest to use and he easiest to extend, not that I’ll ever be doing that!

Either way I thought I’d have a bit of a play with it and see if I could get something up and working and, locally for now due to a lack of documentation from my hosting provider… I have re-created my website Practical DevOps within Sinatra using rackup, bundler and ERB templates.

Now there’s a few reasons I did this, one, I wanted to stop updating every page when ever I needed to update the header or footer of a page, two, I want to implement Split testing (also know as A/B Testing) using something like Split. With all of this in mind and the necessity of having a bit more programmatic control over the website it seemed like a good idea to go with Sinatra.

The Basics of Sinatra

By default Sinatra looks for a static content in a directory called “public” and will look for templates in a folder called “views” which if needed can be configured within the app. So for basic sites this works fine and would have worked fine for me, but I really wanted partial templates to save having to enter the same details on multiple pages, and this can be done with Sinatra partial.

!/usr/bin/ruby

require 'sinatra'
require 'sinatra/partial'
require 'erb'


module Sinatra
  class App < Sinatra::Base

   register Sinatra::Partial
   set :partial_template_engine, :erb

    #Index page
    ['/?', '/index.html'].each do |path|
      get path do
        erb :index, :locals => {:js_plugins => ["assets/plugins/parallax-slider/js/modernizr.js", "assets/plugins/parallax-slider/js/jquery.cslider.js", "assets/js/pages/index.js"], :js_init => '<script type="text/javascript">
        jQuery(document).ready(function() {
            App.init();
            App.initSliders();
            Index.initParallaxSlider();
        });
        </script>', :css_plugins => ['assets/plugins/parallax-slider/css/parallax-slider.css'], :home_active => true}
      end
    end
  end
end

So let’s look at the above which is simply to serve the index of the site.

   register Sinatra::Partial
   set :partial_template_engine, :erb

The register command is how you extend Sinatra, so the sinatra-partial gem when it is installed simply drops it’s code in the sinatra area and when you call register all of the public methods are registered, this allows you to do stuff like this with magic, or you can use it in the ERB template like this. The next line simple tells sinatra to use ERB rather than haml, I chose this because of puppet and chef all using erb and as a result i’m a lot more familiar with that.

 #Index page
    ['/?', '/index.html'].each do |path|
      get path do
        erb :index, :locals => {:js_plugins => ["assets/plugins/parallax-slider/js/modernizr.js", "assets/plugins/parallax-slider/js/jquery.cslider.js", "assets/js/pages/index.js"], :js_init => '<script type="text/javascript">
        jQuery(document).ready(function() {
            App.init();
            App.initSliders();
            Index.initParallaxSlider();
        });
        </script>', :css_plugins => ['assets/plugins/parallax-slider/css/parallax-slider.css'], :home_active => true}
      end
    end

One of the nice things with sinatra is it’s simple to use the same provider for multiple routes, and the easiest way of doing this is to define an array and simply iterate over it for each path. Sinatra uses the http methods to define it’s own functions of what should happen, so a http get requires a route to be defined using the “get [path] block” style syntax and likewise the same for post, delete etc, see the Routes section of the sinatra docs for more info.

The last section is calling the template, so typically the syntax could just be “erb :page_minus_extension” which would load the erb template from the “views” directory created earlier. If you wanted to pass in variables to this you would define a signal ‘:locals’ which takes a hash of variables. All of these variables are only available to the the template that was called at the beginning, so to get the variables to the partial requires some work within the template.

Now within the the views/index.erb file I have the following:

<%= #include header
partial :"partials/header", :locals => {:css_plugins => css_plugins, :home_active => home_active}
%>

Partial calls another template within the views directory, so as I have a partial called header.erb in views/partials/ it loads that, and by defining the locals again from within the template I am able to pass the variables from index into the header or any other partial as needed.

Okay, that’s all folks, Hopefully that’s enough to get people up and running, have a good look at the examples in the git projects they’re useful, and be sure to read the entire Intro to sinatra, very useful!

Doing it the hard way

Lets make it really complicated

Over the last two weeks i’ve been playing with some open source project that has a bit of a kick starter going to fund their idea. I came across it through my boss who probably found it on redit but in essence it’s a forum using nodejs as the backend and some Jquery at the front end but all in all it looks pretty awesome; sure it has a few flaws but it’s less than 6 months old.

In my first venture into using it on my laptop I came across a bug with the title of a new post, when you reply it locks the title to stop you editing it on reply, unfortunately it didn’t unlock it so when you went to post a new topic you were unable to set a title. I decided that I could raise a ticket or I could just have a look at it so I had a lock and submitted a fix to them; to my amazement they accepted my fix! We decided that this is a good platform to use for what we’re trying to do at work but it needs a few core fixes followed by quite a bit of integration and customisations, the core things we need to fix in the product (in order of importance) is

  1. Deployment to bespoke path
  2. Increased logging for debugging
  3. Additional authentication routes

So I decided that not knowing the code I should start at working on point one, it makes it useful and gets me involved in a lot of the code so hopefully I’ll learn something. It seemed a sensible place to start, it was a sensible place to start; unfortunately being a new project there isn’t a lot of documents or sites to google for this stuff so I’ve been learning the hard way.

The main challenge is learning the code, the other is working out why things were done in a certain way. So one of the issues I have is that currently there’s something like 6 config.json files all used for different things, with different config so I need to backwards engineer all of it, a little annoying seeing as with some better technology choices it could just be one config file, but then I also don’t know why things have been done that way.

Challenging me more!

Up until recently my experience to nodeJS had been rather limited but I had used some cool things like express, Winston and Jade but I has always had the luxury of talking to the developer that wrote it to help me understand why it was done that way and how I should use it; this time I’m on hard mode, I have to understand it from reading and I have questions! Hopefully the people running the project will have some time to spend helping me get up to speed and answering my stupid question, I read through the code and I just don’t know what’s going on, I think this is partly down to not really being a programmer and partly to not knowing the language very well so everything is a little odd; at least I hope thats the case, if it is just bat shit crazy then at least I’m confused for a reason :)

I’m definitely going to persevere as I’m, sure it will be useful and it is in the project teams best interest to help me understand what it does so I can start submitting “awesome” changes back to the project even if they don’t want them :)

Either way I’m looking forward to diving into the code a bit more and trying to guess what it’s doing; hopefully I can make it work for our purposes while providing useful (although not needed) features back to the project. I’m also looking forward to increasing what I’m doing in complication so I can start doing the more bespoke work we needed with integrations and maybe adding an achievements framework or some sort of gamification to the forum tool

Cut down, deliver early and often

Deliver all the things

There’s idealistic people in the world and that’s fine (thanks for reading by the way :) ), and there’s pragmatic people, I want to go through how you can provide solutions that give a pragmatic approach to delivering value and doing it in a way that gets it done on time but still helps you get to your idealistic goals.

Often when sitting down and planning with senior management bods a feature list as long as your arm comes out, the reality is even if this is thought to be the bear minimum list, in reality it probably isn’t and is instead a bloated minimum, there’s always room to cut out features, so agree some prioritisation on the features and operate a bucket approach, one in one out.

After the features are agreed you can now start about delivering them, cutting where necessary.

Deadlines for Deadlines sake

When it comes to deadlines people take them a bit like marmite, you either love it or you hate it. Some people feel that having a deadline is a sure fire way of creating a bad product as corners are cut, others think that if you have a deadline you can at least work towards something with an end in sight rather than running off into the wild forever and ever.

To deliver the features needed it’s better to have a deadline, and one that stretches you and forces you to make some cuts, it’s not about not delivering or delivering badly it’s just about delivering what is needed, worse case scenario you have to deliver everything, but this way at least you do it in stages.

Certainly with a deadline you can help focus people on delivering what is important, I think sometimes people get caught up in trying to deliver everything perfectly for the deadline rather than delivering the value they already have. Some of my colleagues and mysef are currently working on a monitoring and metrics platform that integrates fully with nagios style checks but also allows you to write them in the web browser and test them on a server of your choice before distributing. The idea being that you can take the monitoring up to a real time level while reporting back business level reporting and everything in between so you have one place to go to to find out why something isn’t working and how well it has been doing, how many people have signed up; it’s a devops dashboard really.

Anyway, for a couple of months we have been identifying the core technologies and implementing various key functionality to the product but with at no point was any of it “working” some bits sort of worked but not quite, some bits just weren’t there. There was no real end date to this project as it’s something that will keep involving until it works and is useful however we need something to work towards and after a couple of months of sorting out the technology a deadline was set to do a demo and within a week we had the product up and working with the pages we needed with the correct functionality and everything working fine. Writing a nagios check on the fly, pushing it to the server distributing it to all the others and then reporting all in less than 30 seconds, wonderful.

I’m not saying what we did in that week was “production ready” but if our livelihood depended on it, it was good enough, and thats what being agile and lean is about. What is the least amount of work I can do to get me to the minimum product I need in the least amount of effort. The key is to obviously not get stuck delivering bear minimum all the time, with every sprint you need to improve upon what was there as well as add the new stuff; I think it is necessary to always fix something up when adding new features to get the product better and it certainly works for us.

Alternatively, of course, we could have not had a deadline and kept drifting aimlessly into the distance ensuring that the technology was “just right” all the time but the reality is we have to deliver something somewhere.

Iterate

Anyone that is familiar with Agile, Scrum, Extreme programming etc knows it’s better to deliver in small bite size pieces than in large chunks, you can provide value back to the business quicker and you focus on doing the task rather than doing it well. Not all tasks can be done by cutting a few corners but there’s normally a quick way, a good way and the right way fo doing it, so choose one and go for it, if it is a bit of angular that pulls down a list of plugins, go the quick way, if it’s a graphing engine that needs to draw lots of graphs and is used everywhere do it the right way; you’re sensible people, find a balance.

I’ve been talking all about software development which is where most of these methodologies come from, but they can be applied to systems administration as well, I think the same goes for sysadmins as it does programmers, they tend to get stuck in doing the best solution rather than the solution the business needs. Just to dispel any hopes and dreams, maybe save some time by realising that the business cares it works and is stable not how elegant or easy to maintain it is. So when coming up with a load balancing solution, maybe version 1 is haproxy with basic config and version 2 is a bit more in depth, version 3 is F5 & haproxy, version 4 is F5, haproxy and caching…. By all means have the hopes and dreams of the gold solution, but deliver the bronze one okay. If people really use the system and it provides more value iteratively make it better, maybe a bronze + a bit of silver, litle chunks, often.

Summary

Don’t get stuck in the end goal, think about what does the client really need or the business really need, bear minimum; deliver that, measure usage, iterate and improve.

Bash-off – a way to relieve your self

Bash off!

In our team at work we have this concept which we call a “Bash off” the idea is simple. Take something that you could do very simply in bash in several steps with manual fudging in the middle to end up with a result. The sort of thing that may take you an hour to brute force your way through. Now in your programming language of choice, automate the entire process.

Sounds simple right ? I remember the first task we had which was to grab two unique columns of data and to stream them onto the screen, I think it was originally done with watch, tail and tee, took about 20 mins of playing with bash. As a team we chose our own languages and went for it, an hour later we all had a working prototype, another hour later we all had our programs really efficient.

Obviously these are really pointless, but they do have a couple of benefits that help a DevOps person stay good at what they do. It gets you to do more with your language of choice than you normally would and it also causes you to think about how you’re structuring code to make it more efficient, it also helps bond the team together with a bit of healthy competition.

To clarify just because it is called a bash off that dosen’t mean that a solution can’t be in bash. So if it was a very manual human process that you believe you can fully automate in bash, Go for it :)

The challenge

This weeks challenge came from our friends in finance, they have a spreadsheet with some 14k rows in it, two of the columns have many duplicated fields in them and the have a one to many mapping between them. So the challenge is to get from 14k rows down to just the unique entries in each of these two columns and to then make sure the mapping in the spreadsheet can be looked up from a database (where the spreadsheet results must end up)

Due to some meetings I was late to the game on this task, pesky meetings; we had a bash driven prototype which we excluded because it required the excel spreadsheet to be turned into a csv and a python one that sort of got some fields from the spreadsheet and pushed them into the DB.

This is where I picked it up, sort of working; I’ve not done a lot of python so I forked the code and me and my boss went our separate ways to achieve the task, in the end it took us a couple of hours to get this working and for us to have the same results. All in all it we had achieved the processing of 14k rows of data and the manipulation into a db with the correct data, but was that enough? No We decided that it taking about 1 min was not good enough so we started focusing on making it better, I think my run time was 58 seconds and my bosses 40.

We had both chosen different ways of doing it, I had chosen to use the DB to ensure the fields were unique by checking if that field existed in the DB and if not to create it and returned the id, if it was in the DB it would return the id. My bosses approach was better, he created lists with the data in and then made the lists unique. I decided I had to get mine down to a similar speed so I started hacking it around; I decided I would store the unique entries in a list for each table and then before calling the method that puts the data in the database I would check if that unique value was already there. The first issue with this is I lost the ID number which I needed to add to the lookup table so I had to change the list to a dictionary.

I also found by moving the commit messages for the database out side of any of the methods and just dropping it at the end saved a few seconds, I was also able to remove a couple of additional DB queries all of this helped bring the time down. One of the best changes I did was on the lookup table when doing a query only pulling back one ID, rather than two; I didn’t even need it but I couldn’t see a way of querying sql alchemy with out having a field to bring back that would be quick.

As the night progressed we both made good progress, my Boss got his down to 14.1 seconds I managed to get mine down to 21 seconds so we had both made massive improvements in our codes efficiency. My boss was making use of gevent but when I tried this it slowed my program down so I left it out not understanding it anyway. I kept pluggin away and I made it down to 13.5 seconds.

Summary

I urge you to take an afternoon out in your team and to push your skills forward with a programming challenge and to see what happens, it will make you better, you will learn stuff, it is fun, it does bond the team and you will enjoy it, it is also a waste of time but what else were you going to do ?

To IDE or not

What’s in an IDE!

Over the last 15 years or so I have used a few IDE’s for programming “proper” applications in Pascal, Delphie, C/C++ or Java but since moving to the darker side of computing, the linux world I never really needed an IDE and they were just bloated text editors. I don’t think much has changed but there are some good points and I’m at a cross roads as to what to do.

Should I spend some time making Vim, the best text editor the world will ever see better or give up and use an IDE, it’s not as straight forward as I would like to think or as anyone else would probably think so I wonder how it will turn out.

The basics

I’m currently, and probably most likely to be doing web based development and Ruby / python. Up until recently I had just used Vim to get any Ruby programming I needed done, and to be honest it worked okay. I typically just opened a new terminal tab and then changed to the repo directory and started opening files, So I had to cd to a directory before opening the file it wasn’t killing me. I was able to just open and get on with making changes with no real hassle and with syntax high lighting it worked out quite well. Recently I have started to do more web based things and have started using an IDE for that and that’s what’s lead to this blog, there’s a few features of an IDE that I think I’m missing from Vim.

1, Being able to see the entire project directory easily and visually navigating to supporting files to have a quick view / edit is nice.
2, Refactoring, simple things like changing a variable name can be a real bugger in Vim, yes you can find and replace but that is not the same as refactoring, to refactor like an IDE does Vim would need to understand to some degree the codes structure and not to treat it all like text
3, Hiding chunks of code, being able to rol up / down sections of code can make it a lot easier to focus on what’s actually going on
4, Word completion, IDE’s know about the libs being used and the other variabels so can offer hints to complete the words being typed which is very handy
5, Short hand, being able to type short hand to do lots of things, like starting a html tag and having it auto completed speeds up development which must be good

I think if I could get the above features in Vim I probably would struggle to justify not using Vim, lets see what happens.

Vim IDE

1, This is seemingly an easy oen to address with something like NERD tree Initially this looks good so I’m saying point 1 is solved, just need some better aliases (:help 40.2)

2, In short, no, I’ve only seen normal find / replace or basically using grep, There does seem to be language specific ones like This but who wants to find a new refactoring plugin (if someone wrote it) for every new language…

3, This is again easy if you know what to look for Folding is not new and from my quick play seems good.

4, Again language specific it seems possible: this one for ruby, but what about python? javascript? etc etc

5, This one is a simple one that vim does do: Here but it’s almost like you have to hand crank it all!

Summary

So from what I can see I can spend some time making Vim do what I need, and no doubt there’s some stuff there for Ruby and Python that would make it a lot more useful to use as an IDE but it still doesn’t make it easy and will be rather language specific in terms of making it work well out of the box. It does seem that IDE’s are better at those specific tasks above, not to mention debugging and break points etc, maybe Vim can do more of that too and I just don’t know how.

Either way it coped better with the 5 original points than I thought it would and I have found some things out that suggest it may be worth tweaking my current vim to make it better. In the mean time I think I will continue to persevere with learning the IDE and it’s short cuts as it appears that it could dramatically speed up my development by just simply using the tools already there.

I wasn’t expecting to find vi to be as good as it was but in my head even though I prefer Vim at the moment I think the IDE will cope better in the long run, time will tell.

Pain and suffering with AngularJS

What a week!

Over the last month or so I have been steadily learning Javascript, Bootstrap, NodeJS and AngularJS well last week added to that mix was D3JS. I must have got to the end of my tether every day last week. I’m pretty sure I was suffering from information overload and before going any further I will say D3JS is really powerful and looks to be a tool I’d like to learn to use, the only downside is even after a week of drawing circles, bar charts and trying line graphs I am still struggling, but hopefully my struggling has not been in vein I learn some good things and if I’d known them before I’d started a lot less pain and suffering would have ensued.

The problems

So I don’t know angular, I also really struggle to read large chunks of text like the APIs, I did do the tutorial which helps but not really… There were a number of issues, not knowing how angular worked interms of where the right place to put things was a big set back, not knowing how to get data between the various components and updated was another. Another area of challenge was the asynchronous nature of Angular, making a $http.get request and not having data available instantly is odd but easily over come once you know how.

In importance order…

Callbacks

A long while a go I wrote a blog called What University forgot to mention about programming, it turns out this was one of those things that would have been really useful to be learning if the course wasn’t dumbed down to cater for those that didn’t come from an IT background at college.

In short I was doing this:

var data = [];

function getData(){
  $http.get('/api/data').success(function(response) {
    data = response;
  });
}
getData();
alert("data is 0!? " + data.length);

The problem with this is that getData is called and initiates the $http.get which does not return data, it returns the promise of having data; When it actually gets a response it moves on to the function declared in then(). Remember the way Javascript works is we passed in that function as an Object which then gets run when the data returns, it passes in the response for us to then handle. Because $http.get returns a promise that function is done, everything executed, the anonymous function we declared is a parameter that has been forked in the browser or language somewhere and is running isolated from everything else.

The way around this is to structure code in a way that allows for the callback to then do what you needed so….

function getData(cb) {
  $http.get('/api/data').success(function(response){
    cb(response);
  })
}

function drawGraph(newData) {
  //some code to draw a graph omitted...
}

//Draw a graph
getData(drawGraph);

By using the above method you only draw the graph when there is data, take this; rinse and repeat and all will be well, except this method is only really good for a one time graph. Controllers only get called once, they can be used to set up functions to be used in the view etc but they are ultimately only run once.

Data Binding

Firstly, read about Data binding. This first mistake I made was trying to do everything in a controller, it’s the most natural as everything in there gets run like when you include a normal js file, but it just makes life harder than it needs to be.

Originally I was doing something like this:

function getData(cb) {
  $http.get('/api/data').success(function(response){
     cb(resp);
  });
}

function updateData(data){
  drawGraph(data);
}

function drawGraph(data){
  //some code to draw a graph omitted...
  getData(updateData);
}

//Draws initial graph and then kicks off an infinite loop of getting data/drawing graphs 
getData(drawGraph);

I was using this to update and draw a graph. This works sort of but I was not able to get it to work because of my understanding of D3Js at the time, I’m sure smarter people than I could make it work but that’s not how it should be done; theres a better way like explained in the link above about data binding.

If you take your partial (template) and put certain triggers in it, even manual like ng-click you can do this a lot simpler, you can even use $watch to continuously update. so if you were to define a function or two like this…

In your controller:

$scope.updateGraph = function() {
  getData(reDrawGraph);
}

function reDrawGraph() {
  //code to RE-DRAW not draw the graph
  //Enter new, Transition, exit... more later...
}

In your partial:

<button class="btn btn-success" ng-click="updateGraph()">

You can make it update as you go. One step further would be to use $scope.data and just have a loop in the controller that sets that when data comes back, combined with a $watch to always re-draw.

Directives

Okay, so directives are useful, Until I learnt about them I was drawing all of my graphs from the controller and by far the must successful graphs I did were all in the directives area. I still think there was a better way because I was struggling to get it to work with $watch either way by having this layer you achieve two things, less crap in the controllers, and a nice location to do the DOM manipulation, which for graph drawing is kind of handy. Because this links right in with the html it is easier to manipulate.

Summary

These are just some of the major hurdles I overcame while trying to get D3JS working, and there’s more to come when I get on to D3JS which I have to admit I only got basic graphs drawn but I do think it’s an area worthy of diving into with more detail. All in all these experiences have made my understanding of Angular a lot better I am now able to quickly and easily make changes to pages to do basic things like looping through JSON data to display lists of data or forms to submit data.

A day of Javascript

Urghh!

Today I’ve been trying, trying really hard to apply some of that JavaScript knowledge to a simple problem, something that if I was doing it in HTML and PHP would have taken about an hour, and I’ve not done PHP for as long as I haven’t done JavaScript, but that’s not a fair comparison as like most people that used JavaScript in the past I did very limited things like pop up a window or an alert when something went wrong. I only had one thing to do today, add some functionality to a web app to upload files and store them locally on the server.

I’ve only spent 12 hours on it so far, and it sort of works if you ignore the fact that Drag and Drop won’t work an the progress bar doesn’t appear / update or that it doesn’t close the upload popup at some point, but if you’re willing to apply some manual intervention it will upload a file eventually and it will (after you’ve closed the upload dialog and refreshed the page…) display the new item.

I think some of the challenge is the many layers to the app, with html, css, javascript, bootstrap, angular, jquery and nodeJS; there’s just a lot of places doing stuff and it’s not all quite settled into my head yet… combine that with only recently having played with all of those and it’s a little confusing. I spent a lot of time trying to steal peoples code to save time from various open sourced git repos, I basically found that after stealing 5 and none working that I was wasting my time so I just took one as an example and went from there, This one in fact.

With in a couple of hours I had most of it down but not working, having dedicated this sprint to button first development I got the UI looking just right before moving on to any code. I then set about making the javascript bits work and I managed to get everything working apart from drag and drop, so it wasn’t posting data but it was showing files. For some reason the drop event I had setup was not getting triggered, every time I dropped something on it it did nothing ad the browser just opened it as a file.

Annoying, I imagine that something I’ve done in bootstrap or angular has had an impact on it but I don’t know and have just had to abandon it for now, maybe it’s the modal? maybe it’s some weird local issue, either way I’ll have to come back to it.

Some lessons learned

What’s the point in wasting a day on various elements of JavaScript if I’m not going to learn anything. Well I learnt loads! I learnt how to use the debugger in Webstorm which was pretty neat, once I’d clicked that button I found my issue in my node code really quickly, next time that will hopefully save me 2 hours :)
I also learnt that at my level of skill, using someone else’s complicated library for stuff is a bit hit and miss, especially if it requires working with multiple technologies that I’m also not familiar with…

In short at least by writing it my self / re-writing it from other’s examples I had some time to digest it and understand it, of course it doesn’t work but I fell better for having a go :) I am still relatively hopeful that in a few months this will all be second nature, but which time I would have probably forgotten all the ruby I know :)

Anyway, a short one today, more programming tomorrow and hopefully after another full day I would have the upload box working.

JavaScript and me

Where did it all go wrong

It must have been about 10 years ago when I first did some JavaScript and at the time I thought it was pretty good, I only used it on and off between then and university but with no real direction. I enjoyed JavaScript that much that I didn’t think twice about using it for my final year project at university, at the time Web2.0 and AJAX was all the craze but no libs existed and everything had to be done by hand. I was struggling to find a project in my final year and I happened across a geospatial professor at university and with google maps not having been out long I set about creating a geoblog that could be updated via email. it was a PHP backend with a Javascript and XML front end, there was just an index and everything, management, logins, maps etc was dynamic.

Cutting edge stuff! and yet I hated it, it took the magic out of the process and after I finished that project I vowed never to touch it again! well until I was forced into it…

Learning from the ground up

Last week I was sort of forced into it, although the week before I took my self to that pain so I am some what to blame.

Either way, I decided that this week would be a learning week so I’ve been going through W3 Schools and Code Academy to re-familiarise myself with JavaScript. I’ve got to say the Code Academy stuff was good and I’d recommend it to anyone that wants to learn how to program.

I found it useful to recap the stuff I could remember and I learnt a few things too, but then I came across This this which does a good job of explaining why certain ways of doing things are better.

I imagine that all of this reading and learning will help a little but I don’t do that good with reading… practicals are always good, so next week will be all about actually doing something with Node.js

Making something good

One of the big reasons for learning a language is to use it, and by having a practical use for it it helps motivate you to do more, luckily there is a project that we started and I have purpose. My role is the web UI / server side of a larger piece of work, luckily I have some help… Hopefully in a month or so i’ll be able to give out some more details but it will be a little while to get it up and off the ground and sort of doing what it’s meant to in a prototype style. If we can pull it off it’s fair to say it will revolutionise how we currently work and provide a good platform the managing what we do, really looking forward to getting it to the basic prototype phase so we can start iterating on each bit until it’s perfect!

When in doubt, use more JavaScript

Oh, gawd, why

Over the last couple of weeks i’ve been playing with some new technology, not to any depth but just enough to get a feel for it. The week before last it was all Python & Django followed by AngularJS, well this week has been a bit more Javascript in the form of Node.js; granted I’ve not done much but I’ve done some.

As always it started in a random planning session where we have decided that we will make our own tool that will replace our current monitoring tools, the architecture was all drawn up, queue services, agents, plugins, a super fast “worker” process for handling the queue and a management console for pushing commands in and displaying results.

Because of my last two blog posts I got volunteered to do the front end, which is good as I will learn more AngularJS which I would like and I will learn more bootstrap which I would like, I also thought I’d get to write a nice backend in the language of my choice; turns out, not so much.

Would it be Ruby now I’m getting proficient with it? maybe Python, the rest is in Python, I could live with python, it’s not too dissimilar to ruby. It turns out it would be in node.js, oh gawd why. I’m not a fan of JavaScript, and here’s why; back in the day for my university final year project, back when AJAX had just come out and there were no libraries to do it I made a GeoBlog, using google maps API’s, JavaScript and PHP. It had over 1000 lines of javascript, there were no debuggers, the best you had was a console to point out syntax errors. Well that was a nightmare, at least now things have improved with proper debuggers in the browser etc etc so this time would be better…

Sort of…

To be honest node.js isn’t that bad to write simple things in, I’ve been getting more confused by having controllers and routes for both angular and node and trying to keep that separation in my head. I’ve so far been keeping good to my promise to do button first development so other than writing a simple api to return json I’ve not had to do much in node yet and only a little angular but at some point i’m going to get really stuck.

For some reason all these moving parts haven’t yet fallen into place and I’m wondering if I need to go back to some basics and re-vist javascript in it’s pure form and start building on that to get int node and angular. At the moment my coding is purely google, copy, paste, edit; not much in the way of understanding and if I’m going to make this thing work I need more of an understanding.

Maybe I should start with my old friend w3schools followed by some Code Academy followed by all of these

Busy week ahead I think, wonder if it will help or confuse me more, either way more updates next week I think :)