Blog on Amit Kohli

Match the feeling, not the question

Thu, 08 Aug 2024 00:00:00 +0000

I had an interesting conversation with my sister that I thought would be of interest here. Full disclosure, this post illustrates a neurodivergent thought correction, so it might read weird. But oh well.

SDG Direction Estimation

Sun, 30 Jul 2023 00:00:00 +0000

an R-user's guide to Python for data science

Wed, 15 Mar 2023 00:00:00 +0000

R is for sure the best data science language period. That being said, Python is more popular… I suspect it’s a priors thing (i.e. there are just a ton of Python users doing all sorts of stuff, which makes it more likely that companies will choose Python as their de facto language). I will also go on the record and say that Python’s NLP game is better than R’s.

10 reasons why Random Forest is the only algorithm you will need

Wed, 19 Oct 2022 00:00:00 +0000

Quick disclaimer, Random Forests, or RF, don’t solve all problems. If you want to identify cats in an image, or do text stuff like figure out what political candidate mentioned “pudding” more often, or reinforcement learning, RFs can’t help you. But for regression or classification tasks with numerical data, RFs are the best tool around.

How are SDG Indicators interconnected?

Wed, 20 Jul 2022 00:00:00 +0000

This post is an interactive R Markdown document. View the full post with interactive network.

Interactive COVID map by LSOA

Sun, 08 Nov 2020 00:00:00 +0000

Across the world, the COVID 19 pandemic has illustrated how important it is for governments to have the ability to gather, action, and disseminate new case data as close to real time as possible. Luckily in the UK, there is a reasonably good flow of information about new cases. The government has released an interactive map of cases, and they do keep it up to date. This map uses the Middle Super Output Area (MSOA) level of granularity (i.e. not SUUUUPER detailed, but a good place to start), and shows the total number of cases to date. This methodology is really excellent as a first step, but has two important drawbacks:

Introducing CarbonFootprintR, the cool way to figure out the carbon footprint of your in-person event

Fri, 31 Jul 2020 00:00:00 +0000

This post is an interactive R Markdown document. View the full post.

How to create an R package, easy mode

Tue, 07 Jan 2020 00:00:00 +0000

Creating an R package can be a daunting task, but there are several tools to facilitate this process immensely. Of course, each approach has a series of advantages and drawbacks, so it is very much a subjective thing. Since I have changed my mind and worked on this task frequently, here’s a conclusive* guide of the steps I follow. (* This guide will never be finished, so watch this space, I’ll be coming back to make edits as they seem appropriate).

Problems with data

Tue, 07 Jan 2020 00:00:00 +0000

Feature leaking - When you have a dataset that gathers data over a period of time, the feature importance might change, making the model a bit weaker.

Plot shapes for 100000 movies using new package thePlotThickens

Fri, 08 Mar 2019 00:00:00 +0000

This post is an interactive R Markdown document. View the full post.

Is there more violent crime in my hood than in the rest of London?

Sun, 23 Sep 2018 00:00:00 +0000

This post is an interactive R Markdown document. View the full post with interactive charts.

Introducing the Monitoring and Evaluation Bullet Chart

Fri, 16 Mar 2018 10:27:20 +0000

Cross-posted from: http://www.acdivoca.org/2018/03/introducing-the-monitoring-and-evaluation-bullet-chart/.

This edition of Data Digest is for the monitoring and evaluation** **(M&E) people out there, or monitoring, evaluation, and learning (MEL), monitoring, evaluation, research, and learning (MERL), or whatever else the cool kids are calling themselves these days! This post will focus on M&E deliverables, which are called indicators. But it will also come in handy for anyone who depends on key performance indicators (KPIs) or needs to track progress against a target.

Setting up twitter streamR Service on an Ubuntu server

Tue, 07 Nov 2017 15:56:06 +0000

I am working on a super-secret project for which I am harvesting a highly confidential source of data: twitter ≡ƒÖé The idea is to gather a small amount of twitter data, but for a long time… maybe a year. I tried to use the package TwitteR, but it can only grab up to a week of tweets… it’s not really good for a set-it-and-forget-it ongoing capture since it requires user-based authentication, which means (I guess) that a machine can’t authenticate for it. Tangibly this means a human needs to start the process every time. So I could run the script weekly, but of course there’s days you miss, or run at different times… plus it’s just plain annoying…

Announcing ShinyTester - a package that helps you build Shiny apps

Wed, 01 Feb 2017 17:16:57 +0000

Shiny is awesome, but can be a bit daunting and easy to make mistakes in. I recently came back to Shiny after a hiatus of a few years and it was much more challenging than I feel comfortable admitting. I was making bonehead mistakes like writing something instead of output$something, confusing where to put Output commands vs Render commands, etc. I would eventually find my mistake, curse myself and move on with a crumpled ego. Then I had the realization that maybe if I was a beginner, I wouldn’t even know what I was doing wrong. Thusly did I conclude that I was in a unique position to help out the R community: Dumb enough to make mistakes, but experienced enough to eventually remember how to resolve them. So naturally, I wrote an R package that tests the code of the Shiny app itself.

Pop quiz! What is this chart saying?

Wed, 09 Nov 2016 01:06:19 +0000

I have been reading more and more about how people can’t interpret charts… which kinda never occurred to me, if I’m gonna be very honest. Anyway, it kind of made me think of actually testing people informally, to see for myself. So I’ve been doing just that: showing colleagues, friends, etc a chart that we created interactively during the first Accra R-Users session with tons of detail, and asking them to analyze it at length. The results have been staggering! I’m still trying to generalize my conclusions, but thought it would be fun to open up this test to the community, so here it goes! If you feel like sharing, post your observations in the comments section.

Heart-shaped wordcloud, celebrating Colombia peace treaty

Tue, 27 Sep 2016 03:53:07 +0000

This is a lightening quick post just providing the script to draw a heart-shaped wordcloud, using the awesome This is a lightening quick post just providing the script to draw a heart-shaped wordcloud, using the awesome package. See the resulting image here:

Track your local R scheduled tasks with CommandCenter2000!!!

Sun, 29 May 2016 17:32:02 +0000

There are many ways to automate your scripts running, for example using RScript, or in-R itself (and now even as an add-in for RStudio) (check all here). But after a while, it can get a bit overwhelming to track tasks and ensure they are firing as planned. In order to address this, I have developed some futuristic advanced tech that lets us do that. I hereby present to you: the CommandCenter2000!!! Yes, it’s Y2K compliant! This is what it looks like:

I'm leading the Accra R-User Meetups!

Tue, 24 May 2016 20:06:27 +0000

I am not sure how, but I found Data Science Africa, whose mission it is to promote data science in Africa. For now, they are limited to organizing and running meetups across Africa, mostly focused on R, although there are higher albeit more ambitious targets. Anyway, since I’m in Accra, I contacted them to see if I could help in any way, and biz, baz, boom… I’m now leading the Accra R-Community! The community consists of 300+ interested people (another guy set started the meetup and it’s been gathering users sans local leadership) with varying levels of ability in R. To learn more about our meetup and our activities, please check: http://www.meetup.com/Accra-R-Users-Group/

Make dashboard Tiles/Notecards in R

Thu, 25 Feb 2016 19:47:39 +0000

Update!!
#

Due to the popularity of this script, I released it as a package. To install it, use:

KPI dashboard in R with animated icons

Fri, 12 Feb 2016 10:08:04 +0000

So Key Performance Indicators (KPIs) are all the rage in the dashboarding community… well So Key Performance Indicators (KPIs) are all the rage in the dashboarding community… well really. The premise is simple… check a list of measurements against targets and show how they compare using some kind of visualization. I haven’t yet seen, however, a version that can utilize animated icons to display indicators that REALLY need attention. So here you go, a tutorial on how to make your very own animated icon KPI, using the googleVis library.

I'm proudly featured on r-bloggers!

Fri, 05 Feb 2016 00:10:08 +0000

I am proud to announce that with a little bit of encouraging from friends, I submitted my blog for inclusion in the prestigious r-bloggers and it was accepted!

R Tagosphere!

Sun, 31 Jan 2016 17:41:10 +0000

This post explores the inter-relationships of StackOverflow Tags for R-related questions. So I grabbed all the questions tagged with “r”, took the other tags in each question and made some network charts that show how often each tag is seen with the other tags. The point is to see the empirical relationships that develop as people organically describe their problems with R. Full analysis on GitHub, as always.

Who is it hard to shop for?

Fri, 04 Dec 2015 19:57:57 +0000

So people look up what to buy for mom most, wife second, husband third, and finally poor dad last…

Notice that the peaks are always around christmas, then at valentines day, then mothers day and finally fathers day. Stop buying stuff yall!

10 things you didn't know about taxi costs in Accra

Mon, 16 Nov 2015 16:46:47 +0000

“… but I paid 10 yesterday!”

Having just arrived in Accra, Amit and I were trying to wrap our heads around taxi prices. We knew that we had to negotiate with the taxi driver but often the agreed-upon price kinda didn’t make sense…at all.

How to move ODBC DSN information from one computer to another

Thu, 29 Oct 2015 19:02:49 +0000

YAAAAAAAAY!!!! I got a new computer!!

Oh crap how do I move the DSN connections that I made on this computer?? This is how (assuming you’re on windows)! But before, we do, we have to make sure we have the right drivers… so lemme first go over the basikses my precious:

Clickable list of the best animations since 1900, gathered the geek way.

Thu, 22 Oct 2015 13:51:20 +0000

Locations for 75000 dams

Tue, 20 Oct 2015 20:30:39 +0000

The last task I performed for AQUASTAT was to try to find the best way to estimate the anthropogenic evaporation from dams. The paper can be found here, but here I provide one of the fun outputs, a map of 75000 dams!

Chick flicks vs action movies!

Tue, 13 Oct 2015 22:41:44 +0000

I co-wrote this analysis with Laure Belotti to identify the difference between movie scripts for chick flicks and action movies using sentiment analysis, among other stuff. Rather than creating a blog post, we made a freestanding webpage because I wanted to play around w/ bootstrap and make it pretty and stuff.

Anyway, the analysis is here: https://amitkohli.com/movieSentiments/index.html

Leave comments below!

Sentiment analysis on my girlfriend's text messages

Fri, 11 Sep 2015 13:22:14 +0000

When I told my friends that I wanted to give my girlfriend an infographic of us (centered around a sentimental analysis of our texts) as a gift for our first anniversary, most of them told me that was a terrible idea. Yeah… well… CHALLENGE ACCEPTED!! Without further ado, this is what love looks like:

Excel is how I do maps...

Wed, 08 Jul 2015 15:15:55 +0000

Following the footsteps of this cool post I saw by JohnNelsonIDV, I wanted to try to and make a map using Excel.

Fast forward 2 days and the opportunity came up. Lo and behold, a Map of Reference Evapotranspiration built in Excel!

Irrigated Crop Calendar Vizualisation

Thu, 02 Jul 2015 09:37:04 +0000

Visualizations made for FAO AQUASTAT:

After individual Irrigated Crop Calendars for all countries were combined, it allowed for some interesting analysis of global irrigation. These included “The Big Picture” (at two levels of resolution), as well as viewing irrigation by region and by crop. All explanations and visualizations on: http://www.fao.org/nr/water/aquastat/irrigationviz/index.stm

Simplified renewable water resources, visualized

Wed, 01 Jul 2015 16:37:00 +0000

A visualization prepared for FAO AQUASTAT:

In an interest to simplify the information about renewable water resources in all countries in the world, we prepared simplified visualizations of water flows through a country.

The Reddit effect...

Fri, 01 May 2015 19:32:56 +0000

So I had a happy little blog… no one really paid attention, but I didn’t much care because I was happy.

And then I posted one of my visualizations to reddit. This is what happened:

Global dam capacity visualization

Fri, 06 Mar 2015 10:04:00 +0000

A visualization prepared for FAO AQUASTAT:

Using WebGL, which produced by Google for their own purposes, and later converted to a “chrome experiment”. Please see the interactive version on the AQUASTAT webpage:

Irrigation area treemaps

Wed, 04 Mar 2015 19:16:41 +0000

These visualizations were performed for FAO AQUASTAT.

These treemap charts allow for hierarchical organization of information of different types. In the image above, for example, the irrigation areas for each country are represented as the big boxes, within which the technology utilization is displayed in different colors. This allows users to see a large amount of information quickly.

Transboundary surface water flow

Wed, 04 Mar 2015 19:05:38 +0000

A visualization generated for AQUASTAT of FAO.

Two types of charts were prepared: Sankey and Network. The Sankey plots allow for an ‘automatic sorting’ of countries based on who is the water tower and who is the water source. This visualization is useful to demonstrate where a country falls on this continuum.

Chord progressions of 5000 songs!

Sun, 01 Mar 2015 00:00:00 +0000

Update: Full analysis and everything you need at my github https://github.com/datastrategist/Musical-chord-progressions

The Hooktheory.com database contains analyses of over 5000 songs*. These analyses are uploaded by users and allow for all these songs to be analyzed in bulk, as well as individually. One of these ‘all song’ analyses enables users to gather chord progressions on ALL songs (see the analysis file to see how i did it, using the hooktheory API and R). This allowed us to create a Sankey visualization of all chord progressions in the Hooktheory database.

Best way to make a custom palette with ggplot2

Fri, 27 Feb 2015 18:38:44 +0000

A better way to pick colours with ggplot2

First, get the excellent package by Marco D Visser:

Best way to unclog drains

Thu, 19 Feb 2015 18:00:47 +0000

Check out my new instructable on how to clear clogs without using chemicals.

Check it out on instructables.com!

Excel basics

Mon, 16 Feb 2015 18:32:22 +0000

Most of the things posted on this page will make reference to Visual Basic for Applications (VBA).

How do you get to the Visual Basic Editor?
#

Excel versions prior to 2007:
#

Under the Tools menu, go to Macros, and the option should be there.

Excel 2007:
#

Available through the Developer tab on the ribbon. If you don’t see a developer tab:

Cover images for facebook, google+, whatever

Wed, 12 Nov 2014 00:37:33 +0000

So I guess everybody let’s you upload a “cover picture”…. something that should be beautiful, emblematic of you, deep, thoughtful etc. Well… I found a bunch of my pics that fit the wide format nicely.

Arduino theremin concert

Tue, 11 Nov 2014 00:52:47 +0000

One of the arduino tutorials was for a theremin. With a few modifications I was able to properly express my sophisticated musical abilities.

Enjoy!

Arduino code follows (prolly doesn’t make TOO much sense without the wiring schematic… if there’s interest I’ll put up a picture).

Human armature from d3Network!

Tue, 11 Nov 2014 00:17:06 +0000

So a little bit of R, a little bit of d3Network and boom! Instant interactive html human!

<style>
.link {
stroke: #666;
opacity: 0.4;
stroke-width: 1.5px;
}
.node circle {
stroke: #fff;
opacity: 0.4;
stroke-width: 1.5px;
}
.node:not(:hover) .nodetext {
display: none;
}
text {
font: 7px serif;
opacity: 0.4;
pointer-events: none;
}
</style>

<script src=http://d3js.org/d3.v3.min.js></script>

<script> 
 var links = [ { "source" : 0, "target" : 1, "value" : 300 }, { "source" : 1, "target" : 2, "value" : 50 }, { "source" : 1, "target" : 3, "value" : 50 }, { "source" : 2, "target" : 4, "value" : 20 }, { "source" : 3, "target" : 5, "value" : 20 }, { "source" : 4, "target" : 6, "value" : 1 }, { "source" : 5, "target" : 7, "value" : 1 }, { "source" : 8, "target" : 9, "value" : 50 }, { "source" : 9, "target" : 10, "value" : 20 }, { "source" : 9, "target" : 11, "value" : 20 }, { "source" : 10, "target" : 12, "value" : 10 }, { "source" : 11, "target" : 13, "value" : 10 }, { "source" : 12, "target" : 14, "value" : 5 }, { "source" : 13, "target" : 15, "value" : 5 }, { "source" : 2, "target" : 8, "value" : 50 }, { "source" : 3, "target" : 16, "value" : 50 }, { "source" : 1, "target" : 8, "value" : 50 }, { "source" : 1, "target" : 16, "value" : 50 }, { "source" : 8, "target" : 16, "value" : 50 }, { "source" : 16, "target" : 9, "value" : 50 } ] ; 
 var nodes = [ { "name" : "Head", "group" : 1 }, { "name" : "Sternum", "group" : 1 }, { "name" : "Shoulder.r", "group" : 1 }, { "name" : "Shoulder.l", "group" : 1 }, { "name" : "Elbow.r", "group" : 1 }, { "name" : "Elbow.l", "group" : 1 }, { "name" : "Wrist.r", "group" : 1 }, { "name" : "Wrist.l", "group" : 1 }, { "name" : "rib.r", "group" : 1 }, { "name" : "hip.c", "group" : 1 }, { "name" : "hip.r", "group" : 1 }, { "name" : "hip.l", "group" : 1 }, { "name" : "knee.r", "group" : 1 }, { "name" : "knee.l", "group" : 1 }, { "name" : "ankle.r", "group" : 1 }, { "name" : "ankle.l", "group" : 1 }, { "name" : "rib.l", "group" : 1 } ] ; 
 var width = 900
height = 600;

var color = d3.scale.category20();

var force = d3.layout.force()
.nodes(d3.values(nodes))
.links(links)
.size([width, height])
.linkDistance(50)
.charge(-120)
.on("tick", tick)
.start();

var svg = d3.select("body").append("svg")
.attr("width", width)
.attr("height", height);

var link = svg.selectAll(".link")
.data(force.links())
.enter().append("line")
.attr("class", "link")
.style("stroke-width", function(d) { return Math.sqrt(d.value); });

var node = svg.selectAll(".node")
.data(force.nodes())
.enter().append("g")
.attr("class", "node")
.style("fill", function(d) { return color(d.group); })
.style("opacity", 0.4)
.on("mouseover", mouseover)
.on("mouseout", mouseout)
.call(force.drag);

node.append("circle")
.attr("r", 6)

node.append("svg:text")
.attr("class", "nodetext")
.attr("dx", 12)
.attr("dy", ".35em")
.text(function(d) { return d.name });

function tick() {
link
.attr("x1", function(d) { return d.source.x; })
.attr("y1", function(d) { return d.source.y; })
.attr("x2", function(d) { return d.target.x; })
.attr("y2", function(d) { return d.target.y; });

node.attr("transform", function(d) { return "translate(" + d.x + "," + d.y + ")"; });
}

function mouseover() {
d3.select(this).select("circle").transition()
.duration(750)
.attr("r", 16);
d3.select(this).select("text").transition()
.duration(750)
.attr("x", 13)
.style("stroke-width", ".5px")
.style("font", "17.5px serif")
.style("opacity", 1);
}

function mouseout() {
d3.select(this).select("circle").transition()
.duration(750)
.attr("r", 8);
}

</script>

Epic lamp instructable posted

Tue, 04 Nov 2014 21:17:31 +0000

What do you do with a box of 100 slides that you’ve had for 10 years?

BOOM. This is what. Check out the instructable showing the steps!

Wordcloud generator

Mon, 18 Aug 2014 21:23:34 +0000

Using Shiny and R, we made a super-customizable wordcloud maker that takes any .txt file you feed it, and creates a wordcloud just how you like it (without needing to give your email to creepy people). Check it out!

kml maker for Excel -> Google Earth

Wed, 02 Jul 2014 12:36:43 +0000

I made a userform that quickly and easily allows you to make a kml file that’s readable by google earth based on some excel dataset that has coordinates.

Features:

Simple and advanced modes
userform for convenience
Icons change when highlighted
Allows html to be passed to popup contents, everything is customizable
Issue a default icon, and change icons for specific points
Change colours and opacity for points via easy dropdowns
Pass data from excel columns to the popup for each point

I’ve taken the liberty to make an excel file with the form, and some examples and showcasing the features so you know how to do it. Check it out and please provide comments!

Change lists into tables and tables into lists

Fri, 29 Mar 2013 15:04:29 +0000

Designed a userform that converts dimension formats, i.e. information from tabular format to list (or flat) format, and viceversa. Some options include preserving formatting or not, as well as including blank cells or not.

Search or filter through controls in a userform

Sat, 09 Feb 2013 18:30:30 +0000

This bit might be useful if you have a lot of controls on a form, and need a quick way to highlight them. This post supports the video:

I uploaded a video tutorial, you can see it here (please view in fullscreen mode, or watch the video in youtube by clicking on the youtube logo):[http://www.youtube.com/watch?v=5RtV9msZ2RE].

Compare lists and identify matches macro

Thu, 07 Feb 2013 16:24:52 +0000

This macro provides an alternative to vlookup (which looks at two lists and provides data from one list to the second) that retains similar functionality while providing certain benefits. It accomplishes this by physically moving one list to the other. It’s impossible to get a false positive, and it doesn’t provide that annoying #N/A that messes up calculations.

Macro userform - a good visual way to organize macros

Thu, 07 Feb 2013 16:22:10 +0000

At a certain point it can become challenging to quickly find and run the macro we need. Below are some options to organize our macros. I will focus on using a pop-up form because it’s the method I prefer, but at the bottom of the article, you can see other options.