SQL like query on CSV and other tabular data

CSV files are ubiquitous in the programming world and most data interchange is done using the CSV format. One problem however is that you cannot quickly check for certain rows depending on a particular field criteria. As an example, say you need to find all the rows from a CSV file that has a particular value in a field. The most probable path would be to import the CSV in a spreadsheet and then process it to find the particular information.

In this post we will look into a command-line tool that allows you to query CSV data with SQL like syntax. q – Text as Data is a command line tool that allows direct execution of SQL-like queries on CSVs/TSVs (and any other tabular text files). You can look at q as a meta-tool which provides access to all the data-related tools that SQL provides (e.g. expressions, ordering, grouping, aggregation etc.), but on tabular data files.
(more…)

Read More

Exchanging columns in a CSV file

In one of my recent project requiring a CSV text file import, the data columns had to be in certain predefined positions for the program to work correctly. The import did not depend on field labels, which is obviously a wrong way to process data. It would not have been much of a concern had the CSV files been in the correct format. However, the CSV files I had had the columns in wrong positions, which required me to exchange the columns to their correct positions for the parent program to import them correctly. For a few files I’d have easily used a spreadsheet for the task, but with around 34 files it was going to be tedious. The task entailed me to write a small script which helped in the matter.
(more…)

Read More