The perils of ranking

Today, I discovered I was listed on a Top 100 of UK Twitter users by The Independent newspaper based on the algorithm from PeerIndex. I was #47, since you ask, same as Armando Iannucci. It’s not that long since I made it on to a similar PeerIndex list published elsewhere. It’s all very flattering…

But, how can anyone boil down the worth of individuals, organisation, or other entities using half a dozen (almost random) measures? Using some secret algorithm is then used to put those entities, whether twitter users, websites in search engine results or schools and colleges into a ranked order that too many people set incredible store by thereafter.

It’s quite timely that one of my intellectual heroes, Malcolm Gladwell, writing in the most recent issue of the New Yorker, takes on this ranking issue and challenges the U.S. News & World Report’s college rankings, an ordered listing of universities. The list has apparently become the cornerstone of a rankings business that has outlasted its initial host. Gladwell suggests that such ranking is redundant in the modern world (and maybe always has been). There is no way a complex, multivariate phenomenon can be distilled down to a position in a single linear list. Even the presenters of the BBC’s Top Gear car show recognise that there are variables, such as body weight and wet roads, when ranking its “star in a reasonably priced car”.

I think this ranking fail applies equally as well to college honour rolls as it does to twitter users. They’re all complex entities after all and recognition of that is perhaps more flattering than being number 47 on a Top 100.

Related articles

  • How PeerIndex calculated the Twitter 100 (
  • Can you rank journalists by authority on Twitter? PeerIndex thinks so (
  • Top 50 fashion insiders on Twitter list topped by Daily Telegraph’s Hilary Alexander (
  • Politicians ‘have less authority’ than comedians on Twitter (
  • How to: Integrate PeerIndex into Twitter Profile Pages (

6 thoughts on “The perils of ranking”

  1. I gained about 300 followers over the days after The Independent listed sciencebase as somehow worthy of twitter attention. The growth rate remained stable until 21st February at which point half a dozen followers dropped me and after that the growth rate returned to the same incline on the graph as it had been for the weeks before. The increased interest came to nothing much other than a few new followers, no more of a spike then when Guy Kawasaki occasionally retweets me, which usually increases the follower rate to the same degree. Not sure what the lesson is here…interesting that growth rate is not sustained, I suppose that suggests that the new followers don’t spread the word too much if at all.

  2. David

    So the ranking method you describe is one of the features we use in our topic-based rankings.

    To be clear – we look at many thousands of data points and *lots* of features to come up with a score (which is converted to a rank).

    On a topic-by-topic basis we generate a PageRank score for you (using edges based on conversation) which in turn gets modified in several ways–and a PageRank is a relatively good answering ‘how do other people evaluate what you say’

  3. I was really just trying to be modest, I am rather pleased to be in the Top 50 of most influential UK Twitter followers, obviously. I do realise that any multivariate analysis, whatever the underlying statistical methods, can generate a meaningful list for many contexts. As a kid, I used to enter a lot of competitions, my favourite sort were the ones where you had to rank a list of items and the winner would be the one whose order matched that of the judges, I won several prizes in competitions like that, such as a stack of CDs, some barbecue equipment, books, and a supply of Dulux paint! I never won the big prizes though, but I am sure that an analytical tool like PeerIndex would have got me closer rather than the ad hoc method of comparing pairs and adding up scores ever did. Thanks for the insights. Still not sure whether I’m a long, thick cylinder or a hefty cube though ;-)

  4. Hi David

    You’re absolutely right in some ways.
    So let me take your argument down: how can it be possible to take something with multiple criteria and reduce it to fewer? Well the answer is relatively clear – in high dimensional spaces you can use a an SVD or principal components analysis to reduce the number of non-orthogonal dimensions to a smaller number. When you are finally sitting on a series of orthogonal dimensions the challenge is harder – but you can do this by stating (for example) the area, volume or hypervolume – which boils it down to a single number. Or you can simply weight the criteria and add them together.
    So of course it can be done.

    Then the question is – what does that tell you? Are you dealing with a cube with a equal sides and a volume of 1 cubic metric or a long-thin cylinder with the same volume? They are clearly both different, although they have one property in common.

    On the other hand – they have the value of being understandable rather than bringing out dozens of underlying features is useful.

    The measure of a single ranking variable is common. It is what a search engine does when you put in a query which then returns a ranked order list of candidate pages.

    If the point is to say the ‘human’s can’t be brought to a number’ then that is obviously fallacious. Look at numbers like QALYs used in public health planning.

    But yes, it’s difficult bordering on difficult to rate a Lamborghini and a Mini on a single scale.

    So it’s true that people are multi-faceted. It’s also true that there exists plenty of maths (and sociology and social network analysis) that helps us understand people, their position and status within networks. And what we do is apply that maths to the structure of your activity on twitter. Some of it is obvious stuff (for example calculating the number of links in to you as a node on the network). Some of it is less obvious (specifically how we deal with interest communities).

    And half the point is to help people understand their position in the network and in these much wider communities we are a part of.

    Robin Dunbar argues/demonstrates that we can have a few meaningful relationships and juggle a large (but still bounded) number of shallower relationships. Yet we are already well past the point – courtesy of digital technologies–of being able to handle and manage them. Our mission is to help people navigate this space, help you figure out your position in this network, and ultimately benefit from it.

    I’m not sure if our measures – derived from widely used techniques such as PageRank or Ronald Burts work on structural holes and brokerage and other disciplines, as well as our own thinking – are random.

    The challenge for us is to articulate the problem set well; and figure out the key things we need to do to improve the product.

    Your input is welcome
    Azeem Azhar

Comments are closed.