Many times we need to get a statistical distribution of values in a database table. Say you have a e-commerce shoe store having a product table with the following fields and values. As this is only an example I’ve limited the table to a few items; there will hundreds of rows in a real-life table.
(more…)
data
Using the TOML configuration format in your applications
As any one who has programmed knows about configuration files. Configuration files are mostly text files used to configure the parameters and initial settings for computer programs – mostly user applications, operating system settings. Below is a small list of frequently used file formats.
– XML
– Windows INI
– YAML
– JSON
– toml
(more…)
Combinatorial row pairing in databases
While working on a database design I came across the problem of finding combinations or ‘cartesian product’ between all the rows between one table to another table, but not repeating the same permutations.
Take the ‘Teams’ table below which lists four cricket teams. We want to create a combination list of matches between each teams.
(more…)
the design of the US Area Codes
Like the QWERTY keyboard design, which was originally designed so as to slow down typing and prevent key jams in the original typewriters; US Area codes also depend on the design of the then rotary telephones. On the rotary telephone dial, the digits are arranged in a circular layout so that a finger wheel may be rotated with one finger from the position of each digit to a fixed stop position.
(more…)
6 data visualization javascript libraries
A recent web visualization project forced me to explore some data visualization libraries for JavaScript. Below are a few I narrowed down during the development phase.
Leaflet
Leaflet is the leading open-source JavaScript library for mobile-friendly interactive maps. Weighing around 38 KB of JS, it has all the mapping features most developers will ever need.
Although Leaflet is lightweight, and focuses on a core set of features, an easy way to extend its functionality is to use third-party plugins. There are hundreds of nice leaflet plugins you can choose from to extend the functionality of Leaflet. This is one of my favorite mapping libraries and which I’ll surely cover in another post.
(more…)
Scraping Uber ride history data
This month I’ll complete 1 year riding with Uber. The process of booking a ride and the response time has been amazing. This post however is on a different matter. As a data aficionado I was curious on the various locations I had travelled over the year and the cost each month. Heading over to the Uber API docs was a disappointment as Uber does not provide any api for getting the ride history data. My next plan was to scrape the data from Uber pages using PHP or Python. Just when I was going to start the project to scrape Uber trip data, a little Google search returned a nice bookmarklet by @ummjackson that scrapes the data and exports it to a CSV file.
(more…)
Data Quality in web applications
There is nothing painful than having to rebuild a database after you find that the records in the database are corrupted. Data corruption in databases can take many forms – from allowing inaccurate data to be entered to data field truncation due to inadequate field length. This post gives an introduction to the data quality issues you will encounter when designing database applications.
(more…)
Quick way to display CSV files as HTML tables
During database projects I frequently need to export certain views or intermediate results as CSV files. These are further used for importing into other databases. However, many times it is required to display the CSV files in web pages as HTML tables. As most times the data is generated on the fly it requires some JavaScript to turn the CSV data to HTML. One of my current favorite is the csv-to-html-table library. This allows you to easily integrate CSV files into your web pages.
(more…)
Semantic Normalization of data
Every database programmer knows about database normalization – the process of organizing the columns and tables a relational database to minimize data redundancy. Semantic Normalization is the process of reshaping free form entered data. I first encountered the concept in Nathan Marzs Big Data: Principles and best practices of scalable realtime data systems
(more…)