DATA PIPELINES, ANALYSIS, SCIENCE, TRAINING, MGMT

Knowledge Graphs but, like, for Everything

Presented at EARL 2022

Knowledge Graphs provide a graphical simplification of a specific topic. Not only that, they allow us to use network analysis to transverse a topic, and in so doing, new insight becomes ripe for the picking.

The problem I was trying to solve though, was that I wanted to create a knowledge graph out of any corpus. That would allow me to feed in a pdf, and have a knowledge graph pop out. In this talk, I describe my attempts to make this a reality.

Prioritizing Environmental Complaints using NLP

A project of the Data Science for Social Good (DSSGx-UK)

This project was a collaboration with the Superintendency of the Environment (SMA) in Chile. SMA must respond to all citizen complaints, but needed a better way to prioritize where to allocate its efforts.

The project used a mix of structured (e.g., facility information) and unstructured (e.g., description of environmental problems) data to prioritise complaints by identifying those that are more likely to lead to grave sanctions as well as those that are outside of the SMA remit. The two models we developed, now in production, help SMA be more expedient, and more effective.

Introduction to Network Analysis

It's the network statistics that matter, not the picture!

This talk, presented virtually for EARL 2021, goes over the significant importance of network analysis, and goes over basic principles and how to accomplish these using R.

-

Analyzing my girlfriend wife's text messages

Trying to quantify 'how the relationship is going'

As I was beginning to fall in love w/ NLP, I was also falling in love with a beautiful woman, now my wife. For our one year aniverary I created a dashboard displaying several metrics of our relationship as analyzed by our text messages. Note: this is a stupid idea. Note2: but she married me anyway, so...

Assorted tutorials

Step by step instructions

I have written several entries in my blog providing detailed instructions allowing users to replicate functionality. Particularly my old Excel tutorials were well received, and the questions/feedback helped guide my subsequent content.

Youtube tutorials

Showing mistakes and all!

I have created several youtube tutorials guiding people through tools I have created, or through the analytical process. Please check the two links below, the first goes to my Excel tutorials, the second to Power BI / R tutorials (I lost all the views and comments when I moved the R videos to this new channel, please share widely!).

Chord progressions of 5000 songs!

Trying to identify common patterns in popular music

When songs go from one chord to another (chord progressions), people feel certain things. We like to think that our relationship with music is unique and special, but it turns out that there are highly generalizable rules developed by society that inform how music should make us feel. These feelings are highly cultural, with the "auditory tropes" being different in western rock and middle eastern classical music for example. I connected to the API of Hooktheory in order to visualize common progressions.

ShinyTester package

An R package to help debug Shiny apps during the process itself

Shiny is awesome but a bit daunting and easy to make mistakes in. I built this package as more or less a "Shiny linter" that protects against the most common errors, and also provides a nice network chart showing dependencies. Useful when working with reactives which may have the correct inputs, or not. :)

This package is on CRAN as "ShinyTester".

TileMaker package

An R package to single out one number in dashboards, markdown reports, Shiny apps

This package was developed to allow users to single out one number in their reports. Now there are built in options to accomplish the same functionality in several more popular pacakges, although this package still has some relevance in terms of creating highly customized visuals. Take it for a spin!

This package is not on CRAN, but you can get it from my github.

Asset selector (beta)

How do you structure all unstructured assets?

One of the problems of "having too much stuff" is that it becomes difficult to keep track of it, and to know where gaps exist. This was the case with the AQUASTAT assets. We had the main database, yes, but we also had maps, profiles, tables, info sheets, etc. In an attempt to quantify and make available these resources, I created a Shiny app that displayed them all (and identified broken links). The app is not live anymore, but you can see the motivation document by clicking the "more" button below.

Irrigation Treemaps

Breathing new life into old assets

It can take **years** to gather high quality data on global irrigation. And when you have the data and include it into your reports, everyone rejoices... for a while. And after a year or two, everybody forgets and complains about how old the data is. Well, I discovered that by creating attractive and novel visualizations, we could a) bring a new perspective to the experts, b) introduce the material to a new batch of scientists, and c) create new outputs extremely cost-effectively. Thus were born the irrigation treemaps. Unfortunately they are no longer online.

Transboundary river flows

Creating interesting visuals leads to novel insight

In order to calculate the flow of water from country to country, we kept detailed information about river flows. It ocurred to me that there's a "from", there's a "to"... that's network information! I created force network charts which were interesting, but value really came out of the sankey chart representation. The countries self-organized from upstream to downstream countries. We immediately tried to develop a relationship with the legal department to do an analysis of global water policies based on upstream/downstream-ness, but "it didn't work out" :). Still, the visuals themselves were powerful and created quite a stir by demonstrating the scale of relative flows. Also, they are just pretty.

Best animations since 1900

Learning to augment dataset w/ scraped results

The ggplot2movies R package has a list of old movies. I was curious about the old "animation" movies, and once I extracted these, I wrote a function to query Bing w/ each animation and capture the resulting youtube link. From there, I did some probabilistic string matching magic to determine if I had indeed found the youtube of each animation and then created a list of all these.

Combining info about dams from Wikipedia, OpenStreetMaps, AQUASTAT

How much water is evaporated from dam reservoirs?

The last task I performed for AQUASTAT was to try to find the best way to estimate the anthropogenic evaporation from dams. In order to accomplish this, I had to scrape information from Wikipedia/Wikimedia (ask me how I did this and prepare for hilarity), and OpenStreetMap. The analysis could have gone further as it was prepared while I was preparing my handover notes, but it still yielded the most highest quality repository of global dam information available at the time.

-

Chick flicks vs action movies

How are male and female characters portrayed?

I was watching Shrek when my awareness was focused on the emotional valence of the gender-roles the characters were forced to carry out. After a bit of research, I decided to analyze the emotional valence of different "chick flicks" vs "action movies" (I do not endorse these terms nor their equivalence with gender roles) to see if we could extract semantic differences in how the scripts were constructed.

Bulletchart package

A one stop shop visual for KPIs

I'll be the first to say that it takes training and time for people of all levels to understand this visual, but once "over the hump" it's really cool! It provides a concise tool for managers to make effective interventions from week to week, and for employees to know what they have to work on, and it provides an elegant method of bypassing the (long, expensive, potentially inacurate) forecasting step. We liked it so much, we created a package available for anybody to use here.

This package is not on CRAN.

Testing, Testing, 123!

Building out a CI pipeline the right way.

This presentation shows some of the differences between Unit Tests, Integration Tests, and introduces a fun little surprise at the end!

Presented at LondonR 2020.

-

R is for Reconciliation

how machine learning is helping Colombia move forward

This talk documents optimizations to the reconciliation efforts in post-conflict Colombia.

Presented at EARL London 2019

-
BACK

Why should we care about Social Network Analysis?

Presentation to other data scientists in charities and NGOs. Also discussed the ACDI/VOCA toolset

Presented at Social Data Society - 2018

• ACDI/VOCA toolset -
  
• Why should we care about Social Network Analysis? -

Data Tools: Databases and CRM systems

Efficient tools for managing data

Discussion and training on how data is managed at ACDI/VOCA

Presented at Data4Good 2018

BACK

From Raw Data To Real Insights

A Data Management Approach That Works

Discussing how culture change was brought about within ACDI/VOCA

Presented at Humentum Annual Conference 2018

BACK

LEAP, and How to Bring Data to Life in Your Organization

Data is so much more than just numbers for an auditor. In this presentation, we discussed how the LEAP system allowed the company to come to life, and leverage numbers into data, data into knowledge, and knowledge into results.

Presented at MERL Tech London 2018

BACK

Tanzania COP Learning Event

The event focused on how to use our project monitoring system to measure impact.

Presented at MERL (Monitoring, Evaluation, Reporting and Learning) COP Learning Event 2017- Tanzania

• Data Cleaning -
• Learning, Evaluation & Analysis Platform - BACK

Ethiopia Theory of Change Event

Presented at Ethiopia Theory of Change Evaluation TOPS 2017

• Data collection tools -
• Understanding Phase 3 of Data Processing -
• Ethiopia Viz & modelling -
• Spatial Analysis 1 pager -
  BACK

Making International Development Data Not Useless

So much international development data is not comparable, frought with errors, and incomplete, that it can be frustrating to try to interpret the information to infer something about the world. In this presentation, I explain the efforts of the AQUASTAT team to minimize the frustration and to put forward the best information possible.

Presented at EARL London 2017

- BACK

DBBUDDY - Offline data collection for impact

The DBBUDDY tool allows synchronization of robust offline data gathering tech (ODK) with cloud databases in a completely ad hoc way. This allows users maximum flexibility to gather offline data for any database table, with full assurance that the data will be compliant and correct!

Presented at Society for International Development (SID) 2019

-
Innovation competition finalist BACK