My last resort was to split the huge ‘INSERT’ statements into smaller size files. Manually doing the same is obviously time consuming and error prone; the only other solution is to write a small script to split the insert statements. This was to be a quick hack so the parsing code was to be of minimum complexity. Splitting a sql dump containing extended insert statements is somewhat complex so you need to have the dump file in a simple format – each insert statement should be on its own line as shown below.

INSERT INTO `dt_codes` VALUES ...
INSERT INTO `dt_codes` VALUES ...
INSERT INTO `dt_codes` VALUES ...
.
.

You can create the initial dump file in a simple format using the following command or you can do it using phpMyAdmin.

mysqldump -uUSER -pPASS --databases DATABASE_NAME --tables TABLE_NAME \
--extended-insert = FALSE > dump.sql

Once you have the big dump file you can use the following PHP script for splitting the file into smaller chunks. Change the variable ‘$max_lines_per_split’ to whatever value you think will work on your system.

<?php
 
set_time_limit(600);
 
/* Number of 'insert' statements per file */
$max_lines_per_split = 50000;
 
$dump_file = "dump.sql";
$split_file = "dump-split-%d.sql";
$dump_directory = "./sql-dump/";
 
$line_count = 0;
$file_count = 1;
$total_lines = 0;
 
$handle = @fopen($dump_file, "r");
$buffer = "";
 
if ($handle) {
    while(($line = fgets($handle)) !== false) {
        /* Only read 'insert' statements */
        if(!preg_match("/insert/i", $line)) continue;
        $buffer .= $line;
        $line_count++;
 
        /* Copy buffer to the split file */
        if($line_count >= $max_lines_per_split) {
            $file_name = $dump_directory . sprintf($split_file, $file_count);
            $out_write = @fopen($file_name, "w+");
            fputs($out_write, $buffer);
            fclose($out_write);
            $buffer = '';
            $line_count = 0;
            $file_count++;
        }
    }
 
    if($buffer) {
        /* Write out the remaining buffer */
        $file_name = $dump_directory . sprintf($split_file, $file_count);
        $out_write = @fopen($file_name, "w+");
        fputs($out_write, $buffer);
        fclose($out_write);
    }
 
    fclose($handle);
    echo "done.";
}

Once the dump file has been split into smaller ones, you can gzip them to reduce the size further.

gzip dump-split-*

Doctrine DBAL

Doctrine DBAL (available as RC4 as of this writing) provides a wonderful abstraction layer to process all the needed database actions. DBAL is a lightweight layer built around a PDO-like API which offers features like database schema introspection, schema migration, database access and manipulation through an OO API. Note that DBAL API can be used independently of the Doctrine ORM API. So you can use the DBAL library even if you are unfamiliar with the ORM API.

Creating a MySQL schema from DBAL

In this post I’ve created a small example script using the DBAL API to generate a MySQL schema for a couple of tables. Note that the DBAL API uses php namespaces, so you will need to be familiar with them to understand the code, which also means that you must be at least using PHP 5.3.0

<?php
 
use Doctrine\Common\ClassLoader;
 
/* Doctrine uses a class loader to autoload the required classes */
require 'Doctrine/doctrine-dbal/Doctrine/Common/ClassLoader.php';
 
/* Lets first load the Doctrine DBAL lbrary */
$classLoader = new ClassLoader('Doctrine', './Doctrine/doctrine-dbal/');
$classLoader->register();
 
/* Provide DBAL with some initial database infor */
$config = new \Doctrine\DBAL\Configuration();
 
$connectionParams = array(
        'dbname'    => 'test',
        'user'      => 'root',
        'password'  => '',
        'host'      => 'localhost',
        'driver'    => 'pdo_mysql',
);
 
/* Connect to the database */
$conn = \Doctrine\DBAL\DriverManager::getConnection($connectionParams);
 
/* We will now generate a table definition, 
   but first lets get a Schema object .
*/
$schema = new \Doctrine\DBAL\Schema\Schema();
 
/* Now use the Schema object to create a 'users' table */
$usersTable = $schema->createTable("users");
 
/* Add some columns to the table */
$usersTable->addColumn("id", "integer", array("unsigned" => true));
$usersTable->addColumn("first_name", "string", array("length" => 64));
$usersTable->addColumn("last_name", "string", array("length" => 64));
$usersTable->addColumn("email", "string", array("length" => 256));
$usersTable->addColumn("website", "string", array("length" => 256));
 
/* Add a primary key */
$usersTable->setPrimaryKey(array("id"));
 
 
/* Create another table called 'login' and add some columns */
$loginTable = $schema->createTable("login");
$loginTable->addColumn("id", "integer", array("unsigned" => true));
$loginTable->addColumn("username", "string", array("length" => 64));
$loginTable->addColumn("password", "string", array("length" => 64));
$loginTable->addUniqueIndex(array("username"));
$loginTable->setPrimaryKey(array("id"));
 
/* Assign a foreign key constraint to the table */
$loginTable->addForeignKeyConstraint($usersTable, 
                        array("id"), 
                        array("id"), 
                        array("onDelete" => "CASCADE")
                        );
 
/* Set the Schema output platform, as we are using MySQL
   a Mysql schema will be generated. */
$platform = $conn->getDatabasePlatform();
 
/* The 'queries' variable will now hold the 
   an array of sql statements.
*/
$queries = $schema->toSql($platform);
 
?>

This will generate the following MySQL statements.

CREATE TABLE users
  (
     id         INT NOT NULL,
     first_name VARCHAR(64) NOT NULL,
     last_name  VARCHAR(64) NOT NULL,
     email      VARCHAR(255) NOT NULL,
     website    VARCHAR(255) NOT NULL,
     PRIMARY KEY(id)
  )
ENGINE = INNODB 
 
CREATE TABLE login
  (
     id       INT NOT NULL,
     username VARCHAR(64) NOT NULL,
     PASSWORD VARCHAR(64) NOT NULL,
     UNIQUE INDEX login_username_uniq (username),
     PRIMARY KEY(id)
  )
ENGINE = INNODB 
 
ALTER TABLE login ADD CONSTRAINT 
login_id_fk FOREIGN KEY (id) REFERENCES users(
id) ON DELETE CASCADE

Migrating schemas to another platform

Now that you have generated the script to create sql schemas, you can easily migrate the generated schemas to another platform using a couple of statements. For example we can migrate the above MySQL statements to Oracle using the following code.

.
.
$myPlatform = new \Doctrine\DBAL\Platforms\OraclePlatform();
$queries = $schema->toSql($myPlatform);
.

Basic column types

Below are the basic types that you can use in the ‘addColumn’ method when creating tables.

'bigint', 'boolean', 
'datetime', 'date', 'time', 
'decimal', 'integer', 'smallint',
'object', 'string', 'text'.

In the next few posts we will explore some more features of Doctrine DBAL.

Besides these it has a nice query builder, somewhat like the one in Microsoft Access. Toad also can manage your Amazon EC2 instances with its built-in tool, which I’ve yet to try. Other than these it has the other regular features – a database explorer, table builder, database diagram creator etc. Toad is now my favorite tool for managing MySQL data.

Let us take a quick look at some features Adminer provides:

# Connect to a database server with username and password
# Select an existing database or create a new one
# List fields, indexes, foreign keys and triggers of table
# Change name, engine, collation, auto_increment and comment of table
# Alter name, type, collation, comment and default values of columns
# Add and drop tables and columns
# Create, alter, drop and search by indexes including fulltext
# Create, alter, drop and link lists by foreign keys
# Create, alter, drop and select from views
# Create, alter, drop and call stored procedures and functions
# Create, alter and drop triggers
# List data in tables with search, aggregate, sort and limit results
# Insert new records, update and delete the existing ones
# Supports all data types, blobs through file transfer
# Execute any SQL command from a text field or a file
# Export table structure, data, views, routines, databases to SQL or CSV
# Print database schema connected by foreign keys
# Show processes and kill them
# Display users and rights and change them
# Display variables with links to documentation
# Manage events and table partitions (MySQL 5.1)

I don’t know about you but all this features will get me through the day.

Compiled files

But the most interesting idea I liked about Adminer is that if you download the source zip you will see that the software is actually composed of different files (which also are executable). The root folder contains a compile.php script that when executed combines all the required files in a single adminer.php file. During compilation all the php files are compressed (by removing the spaces), the images are base64 encoded and the javascript minified. In the true sense of the word the files are really not compiled but rather compressed and squeezed together in a single file.

Skining

You can also add your own skins to the application by creating a adminer.css file and putting it in the directory where adminer.php is located.

Extensions

Adminer also allows to to override some default functionality by writing your own custom code. Although I’ve not played much with this feature yet.

In conclusion

A fine product, which I find immensely helpful when I need to quickly perform some database task on a clients server.

id      username    password
-----------------------------------
1       admin       *************
2       sameer      *************
3       stewart     *************
 
 
CREATE TABLE IF NOT EXISTS `admin` (
  `id` int(6) unsigned NOT NULL auto_increment,
  `username` varchar(50) NOT NULL default '',
  `password` varchar(100) default NULL,
  PRIMARY KEY  (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 AUTO_INCREMENT=4 ;

Quickly copying tables.

The most common task is to copy the table structures. The following statement copies the structure of the ‘admin’ table to a new table called ‘newadmin’.

CREATE TABLE newadmin LIKE admin

Note that only the structure and not the data is copied to the new table. To also copy the data use the following statement.

CREATE TABLE newadmin AS
(
    SELECT *
    FROM admin
)

In the above examples it is assumed that the source and the destination tables which we want to copy are in the same database, but you can also copy tables from another database by prefixing the database name before the table. The following for example will copy the structure of the ‘admin’ table from the ‘shop’ database to the ‘newadmin’ table in the current database.

CREATE TABLE newadmin LIKE shop.admin

You can also specify the destination database in the query.

CREATE TABLE newshop.newadmin LIKE shop.admin

This will copy the ‘admin’ table from the ‘shop’ database to the new table ‘newadmin’ in the ‘newshop’ database.

Note: In all the above examples the DATA DIRECTORY or INDEX DIRECTORY table options are not copied to the new table. So if you need an exact replica of a table you have to create two separate statements.

CREATE TABLE newadmin LIKE admin;
INSERT INTO newadmin SELECT * FROM admin;

This will copy the table ‘admin’ exactly to the ‘newadmin’ table with all the correct table definitions, but we lose the flexibility we can have with the other method, which we will be seeing next.

To copy some select columns to our new table, specify the column names in a SELECT statement.

CREATE TABLE newadmin AS
(
    SELECT username, password FROM admin
)

This will only copy the ‘username’ and ‘password’ columns to the new table. This can be really useful when the original table contains over fifty columns and you want to copy only a few.

While copying a table you can also change the column names of the new table.

CREATE TABLE newadmin AS
(
    SELECT id, username AS uname, password AS pass FROM admin
)

By now you may have already guessed that you could copy any queried data to the new table using the SELECT statement. The following for example copies all the rows from ‘admin’ to the new table where the username starts with an ‘s’.

CREATE TABLE newadmin AS
(
    SELECT * FROM admin WHERE LEFT(username,1) = 's'
)

Copies the following two rows from the ‘admin’ table.

id      username    password
-----------------------------------
2       sameer      *************
3       stewart     *************

While copying you can also change the table structure of the new table by specifying the same in your query.

CREATE TABLE newadmin
(
    id INTEGER NOT NULL AUTO_INCREMENT PRIMARY KEY
) 
AS
(
    SELECT * FROM admin
)

This will copy the ‘admin’ table and also assign the appropriate primary key to the ‘id’ column in the new table.

In conclusion

So basically there are two methods of copying tables.
a. By using the LIKE and AS keywords in single sql statement.

CREATE TABLE newadmin AS
(
    SELECT *
    FROM admin
)

b. By using two different statements.

CREATE TABLE newadmin LIKE admin;
INSERT INTO newadmin SELECT * FROM admin;

The first one offers flexibility while copying tables but does not preserve any DATA DIRECTORY or INDEX DIRECTORY table options that were specified for the original table, or any foreign key definitions.

The second method truthfully copies the tables but does not offer the flexibility of the first method. By truthfully I mean only the DATA DIRECTORY or INDEX DIRECTORY table options are not copied.

Which method you use will depend on your specific requirement.

Forward engineering
SchemaBank lets you generate SQL statements and other database objects directly from the the ER diagram. Of course many other tools provide it, so no surprises there.

Reverse engineering
SchemaBank also lets you reverse engineer your existing database schema and import it into SchemaBank for editing.

Versioning and Change Management
One of the unique features of the product, not found in most schema designers, is that of versioning. SchemaBank lets you commit ERD data, Text Objects and database Options to a repository. This lets you easily play with you database schemas without fear of losing vital information, or if you are creating various versions of the database like say for testing and production release. You can also generate diff report showing the difference between two versions of the schemas.

Sharing Database Projects
Database projects created can also be shared with other developers, with the appropriate ‘read’ or ‘write’ access granted by the schema owner. The other developers should have an account with SchemaBank to access the project.

Database objects
You can enter the SQL statements directly for database objects like views, functions, triggers, etc.

GUI
The product also supports various color themes, if you do care about such things. The interface could also have been made a little bit more cleaner, specially the top part. Based on ExtJS, it drags a bit on slower connections but not so much that it breaks your flow of thought. As it relies heavily on Javascript, a faster browser like Chrome can make it run more smoother.

Pricing
SchemaBank provides a free version with a limit of a single project. The other options cost upward of dollar forty per month.

In closing
The product is still in beta, so expect some positive changes before the final release. A live demo can be found here.

<?php
 
function get_column_names($conn, $table_name) {
 
    $query = "SHOW COLUMNS FROM {$table_name}";
 
    if(($result=mysql_query($query, $conn))) {
 
        /* Store the column names retrieved in an array */
        $column_names = array();
        while ($row = mysql_fetch_array($result)) {
            $column_names[] = $row['Field'];
        }
 
        return $column_names;
    }
    else
        return 0;
}
?>

Now we create a function that returns a comma separated list of column names which we can then pass to the SELECT statement. The function also takes a array of column names we want to exclude.

<?php
 
function create_statement($connection, $table_name, array $exclude)
{
    /* $exclude contains the columns we do not want */
    $column_names = get_column_names($connection, $table_name);
    $statement = "";
 
    foreach($column_names as $name) {
        if(!in_array($name, $exclude)) {
            if($statement == "")
                $statement = $name;
            else
                $statement .= "," . $name;
        }
    }
 
    return $statement;
}
 
?>

Now we create a list of columns and pass it to the SELECT statement.

/*
We want to exclude columns 'post_author' and
'comment_count' from the table wp_posts
*/
$exclude = array("post_author", "comment_count");
 
$stmt_list = create_statement($connection, "wp_posts", $exclude);
 
$query = "SELECT {$stmt_list} FROM wp_posts";

Some of the features of HeidiSQL:
# generate nice SQL-exports
# synchronize tables between two databases
# manage user-privileges
# import text-files
# export table-data as CSV, HTML and XML
# browse and edit table-data using a comfortable grid
# batch-insert ascii or binary files into tables
# write queries with customizable syntax-highlighting and code-completion
# monitor and kill client-processes

Version 4.0, which will release soon also supports query completion and various grid editors for column data.

HeidiSQL is also available for Apple iPhone in the form of a Java port jHeidi.

<?php
 
$link = mysql_connect('host', 'username', 'password');
 
$db_name = "your database name here";
$tables = array();
 
mysql_select_db($db_name, $link);
$result = mysql_query("SHOW TABLE STATUS");
 
while($row = mysql_fetch_array($result)) {
    /* We return the size in Kilobytes */
    $total_size = ($row[ "Data_length" ] + 
                   $row[ "Index_length" ]) / 1024;
    $tables[$row['Name']] = sprintf("%.2f", $total_size);
}
 
print_r($tables);
 
?>

It will return data like the following for a ‘wordpress’ database:

Array
(
    [wp_comments] => 80.00
    [wp_links] => 48.00
    [wp_options] => 224.00
    [wp_pollsa] => 16.00
    [wp_pollsip] => 16.00
    [wp_pollsq] => 16.00
    [wp_postmeta] => 48.00
    [wp_posts] => 112.00
    [wp_term_relationships] => 32.00
    [wp_term_taxonomy] => 32.00
    [wp_terms] => 48.00
    [wp_usermeta] => 48.00
    [wp_users] => 48.00
)

This tool allows you to draw and create E-R diagrams in your browser using Javascript without the need for any external plugins like Flash, Java or Flex.

Many database features are supported, such as keys, foreign key constraints, comments and indexes. You can either save your design , print it or export as SQL script. It is possible to import schema from existing database. The application supports MySQL, Sqlite, PostgreSQL ,MSSql.

The application allows you to:
1. Draw E-R designs
2. Edit tables and rows
3. Manage keys
4. Create relations (FK constraints)
5. Save & Load designs
6. Import DB schemas

The importing of MySQL databases can take some time, so make sure you set PHP timeout to more than 60 seconds to avoid errors. A sample WordPress 2.7 schema imported is shown below.

Installation:
Installation is simple. Just download the code from here, change your database settings in the ‘sqldesigner\backend\php-mysql\index.php’ (if you are using PHP & MySQL) file and you are ready.

A live demo of the application can be found here.