Missing data in databases can cause bugs in applications or incorrect calculations. Recently, while working on a RETS application, I needed to ensure that not many missing values were encountered in one of the MySQL tables. Although one could easily write a SQL query to find the percentage of missing values, I many times find it easier to first get a visual representation of the amount of missing data there is in the table, and then drill-down further if required. One library that I found that lets you easily get a visual representation of missing data in your database tables is missingno – a Python library.
(more…)
data
Statistical distribution of column values in MySQL
Many times we need to get a statistical distribution of values in a database table. Say you have a e-commerce shoe store having a product table with the following fields and values. As this is only an example I’ve limited the table to a few items; there will hundreds of rows in a real-life table.
(more…)
Using the TOML configuration format in your applications
As any one who has programmed knows about configuration files. Configuration files are mostly text files used to configure the parameters and initial settings for computer programs – mostly user applications, operating system settings. Below is a small list of frequently used file formats.
– XML
– Windows INI
– YAML
– JSON
– toml
(more…)
Combinatorial row pairing in databases
While working on a database design I came across the problem of finding combinations or ‘cartesian product’ between all the rows between one table to another table, but not repeating the same permutations.
Take the ‘Teams’ table below which lists four cricket teams. We want to create a combination list of matches between each teams.
(more…)
the design of the US Area Codes
Like the QWERTY keyboard design, which was originally designed so as to slow down typing and prevent key jams in the original typewriters; US Area codes also depend on the design of the then rotary telephones. On the rotary telephone dial, the digits are arranged in a circular layout so that a finger wheel may be rotated with one finger from the position of each digit to a fixed stop position.
(more…)
6 data visualization javascript libraries
A recent web visualization project forced me to explore some data visualization libraries for JavaScript. Below are a few I narrowed down during the development phase.
Leaflet
Leaflet is the leading open-source JavaScript library for mobile-friendly interactive maps. Weighing around 38 KB of JS, it has all the mapping features most developers will ever need.
Although Leaflet is lightweight, and focuses on a core set of features, an easy way to extend its functionality is to use third-party plugins. There are hundreds of nice leaflet plugins you can choose from to extend the functionality of Leaflet. This is one of my favorite mapping libraries and which I’ll surely cover in another post.
(more…)
Scraping Uber ride history data
This month I’ll complete 1 year riding with Uber. The process of booking a ride and the response time has been amazing. This post however is on a different matter. As a data aficionado I was curious on the various locations I had travelled over the year and the cost each month. Heading over to the Uber API docs was a disappointment as Uber does not provide any api for getting the ride history data. My next plan was to scrape the data from Uber pages using PHP or Python. Just when I was going to start the project to scrape Uber trip data, a little Google search returned a nice bookmarklet by @ummjackson that scrapes the data and exports it to a CSV file.
(more…)
Data Quality in web applications
There is nothing painful than having to rebuild a database after you find that the records in the database are corrupted. Data corruption in databases can take many forms – from allowing inaccurate data to be entered to data field truncation due to inadequate field length. This post gives an introduction to the data quality issues you will encounter when designing database applications.
(more…)