The Rainbow Connection: How We Made Our CDO Connections Graphic

On Wednesday, we launched an interactive news application to help readers understand the cross-owned nature of Collateralized Debt Obligations (CDOs) in 2006-2007. This cross-ownership helped inflate the bubble, and ultimately made the financial crisis worse.

We received a list of cross-owned CDOs as a result of a study ProPublica commissioned from Thetica, a consulting company in New York. It consisted of a list of CDOs, the banks that sponsored them, the CDO managers who managed them, and an enumerated list of other CDOs in which it had both sold and bought a stake. Reporters Jake Bernstein and Jesse Eisinger had already used the data in their story, Banks’ Self-Dealing Super-Charged Financial Crisis.

It's an amazingly complex subject -- here’s a video explaining what a CDO is -- which ought to be obvious when you start exploring some of the relationships. Our goal with the news app was to show readers how interconnected this part of the banking industry was, but also to let them explore individual banks and even individual deals.

Here’s a quick overview of how we did it.

Preparing the Data

Originally the data was in a spreadsheet that listed associations separated by commas like this:

Each CDO had a unique key and a list of associations separated by commas. Those associations described a many-to-many relationship between CDOs. Compared to the majority of the data we deal with, there were relatively few records, so we decided to make this a one page JavaScript app without a back-end database.

JavaScript can't easily read CSV files, so we converted it to JSON with a program called Gridworks with the "Export Filtered Rows > Templating..." command:

And this is the Gridworks template we used to split the associations into a JavaScript array:

{
  "cdo_name" : {{jsonize(cells["cdo_name"].value)}},
  "cdo_associations" : [{{forEach(cells["cdo_associations"].value.split(","), v, jsonize(v)).join(", ")}}],
  "cdo" : {{jsonize(cells["cdo"].value)}},
  "bank" : {{jsonize(cells["bank"].value)}},
  "manager" : {{jsonize(cells["manager"].value)}},
  "close_date" : {{jsonize(cells["close_date"].value)}}
}

Which we put in the "Row Template" here:

Gridworks helpfully spits out a JSON data structure we could work with:

{
  "rows": [
    {
        "cdo_name": "5dtwa7z",
        "cdo_associations": [
            "7e22x"
        ],
        "cdo": "Brushfield CDO 2007-1",
        "bank": "Bear Stearns",
        "manager": "Brushfield Capital Management",
        "close_date": "2007-08-02"
    },
    {
        "cdo_name": "1odeh9wv",
        "cdo_associations": [
            "1kewxn3"
        ],
        "cdo": "Nordic Valley 2007-1 CDO",
        "bank": "Bank of America Securities",
        "manager": "250 Capital",
        "close_date": "2007-08-02"
    },
    ...
    ]
}

Making the Chart

At its core, the chart simply does grouping and lookups on the JSON data to render the cross ownership of particular CDOs and banks.

The majority of the interactive functionality relies on a little JavaScript framework that handles UI behaviors on our site, which we call Glass.js (we're open sourcing it someday soon, we promise), and in order to draw the relationship lines we used a cross-browser framework called Raphaël.

Raphaël solves a key cross-platform hassle: Internet Explorer uses VML for vector graphics, and everything else uses SVG. Effectively, Raphaël provides an abstraction layer so you can write graphics routines once and they work everywhere.

There are three content areas in this application:

Our app has two columns of CDOs, each in separate html divs, and a drawing space in the center column. When a user clicks on a particular CDO, the application picks from two types of connections to draw. When the connected CDOs are in the same column, it draws an arc:


// Draw an arc from a CDO to a CDO in the same column.
_topBottom : function(from, to, paper){
  // The paper variable in this function is a Raphaël object 
  // and paper.path creates a new SVG path from the array we pass
  // in.
  paper.path(["M" // SVG Command to move to a point
            , from.x
            , from.y
            , "A" // SVG Command to begin drawing an arc
            , 3
            , 10
            , 0 // The following 3 items describe how to draw the arc
            , 1
            , 1
            , to.x
            , to.y
            ]).attr({ "stroke-width": 2,   // these attributes describe
                 "stroke-linecap": "round", // the overall appearance 
                          opacity: 0.7});   // of the line.
},

To draw a straight line between CDOs in different columns, the function is a lot simpler:

_leftRight : function(from, to, paper){
  paper.path(["M" // SVG Command to move to a point
            , from.x
            , from.y
            , "L" // SVG Command to begin drawing a line
            , to.x
            , to.y
            ]).attr({ "stroke-width": 2, 
                  "stroke-linecap": "round", 
                           opacity: 0.7});
},

More About SVG

If you want to understand more about the structure of SVG calls, the Arc and Line sections of the SVG spec were helpful to me.

In past projects we’ve used the HTML5 canvas element and Google’s excanvas – see our Stimulus Speed Chart -- but for this project, that technique was too inefficient because we needed to redraw multiple associations with every refresh (that is, when a user clicks on a tab).

Incidentally, whether or not you use excanvas or Raphaël in a project, Internet Explorer users will be looking at VML in both cases.

If you have questions or find any bugs send an email to [email protected]. And as always, please let us know if you use these techniques in your projects.