wkhtmltoimage is a simple shell utility which can be used to convert html to images using the webkit rendering engine, and qt.

Installation

To get started we need to first download and install the shell program wkhtmltoimage. Select the appropriate binaries for your platform. As I’m using Ubuntu 10, I downloaded the complied binary ‘wkhtmltoimage-0.10.0_rc2-static-i386.tar.bz2′. Extract it to a appropriate folder and you are ready to go.

Getting your first snapshot

The simplest way to get a snapshot of a url is through the following:

wkhtmltoimage http://www.bbc.com bbc.jpg

This will fetch the www.bbc.com index page and save it as a jpg image. Below are a few websites rendered using the wkhtmltoimage tool.

Customizing the output

wkhtmltoimage comes with a plethora of options, a few are shown below, more options can be found in the documentation.

The default output quality of the program is set to ’94′ which can make the size of some images a lot bigger, but you can change it to your liking using the below option.

wkhtmltoimage --quality 50 http://www.bbc.com bbc.jpg

Also by default all the images on a page are rendered in the final image, but you can disable images in the final screen-shot using the following:

wkhtmltoimage --no-images http://www.bbc.com bbc.jpg

You can also set the output height and width of the image (in pixels) as below.

wkhtmltoimage --height 600 --width 1800 http://www.bbc.com bbc.jpg

or crop the image to a specified size;

wkhtmltoimage --crop-h 300 --crop-w 300 --crop-x 0 --crop-y 0 
       www.bbc.com bbc.jpg

Sometimes the JavaScript on the webpage you are rendering can cause problems during rendering, preventing the program from saving the screenshot or causing a huge delay. In such cases you can ask wkhtmltoimage to not run JavaScript on the page while rendering.

wkhtmltoimage --disable-javascript http://www.bbc.com bbc.jpg

Using with PHP

Although this is not a pure PHP solution for taking screen shots, you can wrap the final command in a shell_exec function or download the ‘snappy’ PHP5 wrapper from below. The original library is located here, but as the source keeps changing the examples given here do not work, so use the library given below.

shell_exec('./wkhtmltoimage --quality 50 http://www.bbc.com bbc.jpg');

Below is a short code using ‘snappy’ to take a screen-shot of bbc.com.

<?php
 
/* Tested on Ubuntu 10.0.4, requires PHP 5.3  */
 
namespace Knplabs\Snappy;
 
require_once('Knplabs/Snappy/Media.php');
require_once('Knplabs/Snappy/Image.php');
 
/* 'wkhtmltoimage' executable  is located in the current directory */
$snap = new Image('./wkhtmltoimage');
 
/* Displays the bbc.com website index page screen-shot in the browser */
header("Content-Type: image/jpeg");
$snap->output('http://www.bbc.com');
 
?>

and with a few options added…

<?php
 
/* Tested on Ubuntu 10.0.4, requires PHP 5.3  */
 
namespace Knplabs\Snappy;
 
require_once('Knplabs/Snappy/Media.php');
require_once('Knplabs/Snappy/Image.php');
 
$options = array('zoom' => 0.5, 'no-images' => true);
 
/* 'wkhtmltoimage' executable  is located in the current directory */
$snap = new Image('./wkhtmltoimage-i386',$options);
 
/* Displays the bbc.com website index page screen-shot in the browser */
header("Content-Type: image/jpeg");
$snap->output('http://www.bbc.com');
 
?>

In my opinion you should play with the shell program first till you get to know the complete options and only than use the ‘snappy’ PHP wrapper.

In closing here is the snapshot of my site ,rendered at 50% quality (click to zoom).

Happy rendering!!

“Wallaby” is the codename for an experimental technology that converts Flash content (FLA files) into HTML 5. This allows you to reuse existing Flash content on devices that do not support Flash. Once converted to HTML, you can easily edit them using Adobe Dreamweaver or any other editing tools.

Wallaby comes as a desktop Adobe Air application so you will need the AIR runtime installed on your system to run the application.

Wallaby uses SVG instead of Canvas to recreate the output because SVG is rendered natively by the browser while Canvas is drawn with JavaScript, which tends to slow down the rendering. According to experiments carried out by Adobe, Canvas was a lot slower than SVG on portable devices. Another point cited by Adobe for favoring SVG is that of readability. Using Canvas lowers the readability of the HTML since the output becomes a program instead of markup as in SVG.

Note that as of now not all Flash Professional features are supported in the HTML5 format, so Flash files above average complexity will not be rendered properly. I tried converting a few FLA files but met with a few conversion errors, mainly that of incompatible file format. As Adobe is desperate to make its presence felt in the portable devices market, expect a more complete working product by 2011 end.

You can download Wallaby from Adobe Labs.

Installing MozRepl

Before proceeding you need to first install MozRepl which comes as a Firefox extension. Head over to github to download and install. Once you have installed it, go to Firefox Tools menu and select MozLab → Start MozRepl. MozRepl will now be listening for connections on the default port 4242.

Connecting to MozRepl

Now that we have MozRepl up and running we can easily connect to it using Telnet. By default MozRepl listens on port 4242, which however you can change to your liking from the MozRepl menu.

C:\> telnet localhost 4242

You will get the following response if you are correctly connected to MozRepl.

Welcome to MozRepl.
 - If you get stuck at the "...>" prompt, enter a semicolon (;) 
at the beginning of the line to force evaluation.
 - If you get errors after every character you type, see http://github.c
om/bard/mozrepl/wikis/troubleshooting (short version: stop using 
Microsoft telnet, use netcat or putty instead)
Current working context: chrome://browser/content/browser.xul
Current input mode: syntax
 
repl>

If you have trouble connecting using Windows telnet or encounter some inconsistent behavior, you can use Putty instead. A typical putty connection setup is shown below, note the ‘telnet’ radio selection:

Also make sure you check the ‘Implicit CR in every LF’ check box.

Once you are connected to the server you will be presented with the following prompt:

repl>

Lets try a standard greeting.

repl> window.alert("Hello World!")

MozRepl is now ready to accepts your commands. Lets try a simple one given below. The first line is the command we send to MozRepl to get the title of the current open Firefox window, while the second line is the response.

repl> document.title
"Tutorial - mozrepl - GitHub - Mozilla Firefox"

Lets try changing the document title of the current Firefox window:

repl> document.title = "hello"

Use the following to get all the function and variables in Firefox:

repl> repl.look()

This is just a sampler, you can do a whole lot of things using MozRepl, check this video for some more examples.

Connecting to MozRepl with PHP

As MozRepl creates a server on port 4242, we can easily connect to it via PHP and send commands to it. A small example is given below which you will need to run from the command line. The following code sends 3 commands to the MozRepl server and prints the responses. It uses the socketHelper class to do the dirty work of socket processing.

<?php
 
require_once('socketHelper.php');
 
/* Commands to send to  MozRepl */
 
$commands=<<<EOD
repl.whereAmI()
content.location.href = 'http://google.com'
repl.quit()
EOD;
 
$firefox_socket = new SocketHelper;
if(!$firefox_socket->connect()) exit;
 
foreach(explode("\n",$commands) as $command){
    if($command=='')continue; //Skip blank lines
    echo $firefox_socket->send_command($command);
}
 
?>

Code for socketHelper class

<?php
 
/* socketHelper.php */
 
class SocketHelper
{
 
    private $address = "127.0.0.1";
    private $port    = "4242";
    private $socket  = null;
 
    public function connect()
    {
        $this->socket=socket_create(AF_INET,SOCK_STREAM,SOL_TCP);
        if(!$this->socket){
            socket_strerror($this->socket)."\n";
            return false;
        }
 
        $result=socket_connect($this->socket,$this->address,$this->port);
 
        if(!$result){
            socket_strerror($result)."\n";
            socket_close($this->socket);
            return false;
        }
 
        $this->read();
        return true;
    }
 
    /** Send a command to MozRepl */
    public function send_command($command){
        $command.="\n";
        socket_write($this->socket,$command);
        return $this->read();
    }
 
 
    /* 
        Read from the Socket until we get a "repl>" prompt, 
        or loop forever.
     */
    private function read(){
        $response = '';
        while(1){
            $chunk = socket_read($this->socket,65536,PHP_BINARY_READ);
            if($chunk === false){
                echo "Error reading from socket\n";
                break;
            }
            if($chunk === "") break; //No more data
 
            if(preg_match('|^(.*)\s*repl\d*>\s*$|s',$chunk,$match)){
                $response .= $match[1];
                break;
            }
 
            $response .= $chunk;
        }
        return $response;
    }
}
?>

Security Warning

Because MozRepl can directly poke inside FireFox it can practically work with anything – your bookmarks, cookies, cache etc, so security should be your prime concern. So do not ever allow outside connections in MozRepl (Tools menu → MozLab → Allow outside connections), unless you are sure what you are doing.

I’m still investigating the things I can do with MozRepl. If you have done some cool things of your own using it, please post your ideas here.

Besides these it has a nice query builder, somewhat like the one in Microsoft Access. Toad also can manage your Amazon EC2 instances with its built-in tool, which I’ve yet to try. Other than these it has the other regular features – a database explorer, table builder, database diagram creator etc. Toad is now my favorite tool for managing MySQL data.

Haml

Haml (HTML Abstraction Markup Language) is a lightweight markup language (or a meta markup language) which lets you create views for your Rails applications. But it can also be quite helpful for generating xhtml pages when used outside Rails. As I’m not a ROR developer my sole interest in Haml lies in using it to quickly create well formatted HTML files. Below is a short haml markup and its output. Haml comes with a standalone tool that lets you quickly convert haml markup to xhtml.

!!!
%html
   %head
      %title codediesel
   %body

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
  <head>
    <title>codediesel</title>
  </head>
  <body></body>
</html>

A more involved example using a bit of Ruby code.

!!!
%html
   %head
      %title codediesel
   %body
      %h1 HAML
      / Demo of haml
      %div#main Lots of paragraphs
      %div#paras.list
         - (1...5).each do |i|
            %p= i
 
      / Lots of classes
      %div.myclasses
         - (1...4).each do |i|
            %div{:class => i}
 
      / a select element
      %select#color
         - ['red', 'blue', 'yellow', 'green'].each do |i|
            %option{:value => i}= i

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" 
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
  <head>
    <title>codediesel</title>
  </head>
  <body>
    <h1>HAML</h1>
    <!-- Demo of haml -->
    <div id='main'>Lots of paragraphs</div>
    <div class='list' id='paras'>
      <p>1</p>
      <p>2</p>
      <p>3</p>
      <p>4</p>
    </div>
    <!-- Lots of classes -->
    <div class='myclasses'>
      <div class='1'></div>
      <div class='2'></div>
      <div class='3'></div>
    </div>
    <!-- a select element -->
    <select id='color'>
      <option value='red'>red</option>
      <option value='blue'>blue</option>
      <option value='yellow'>yellow</option>
      <option value='green'>green</option>
    </select>
  </body>
</html>

The useful thing about Haml for me is that I can use it to quickly generate xhtml skeleton files using a few haml markup files and a dash of shell scripting. Editing the haml markup is easier and quicker then editing the xhtml file, especially if the html happens to be a big one. Note that indenting is extremely important in the haml markup, so if you are just trying out haml, be prepared to encounter a bit of errors in the beginning. If you are new to Ruby and are interested in trying haml, I suggest you install the rubyinstaller environment. It works right of the box with many other required components pre-installed.

phpHaml

If you get interested in haml there is a php port of of the same called phpHaml and phamlp. Here we will take a look at phpHaml. phpHaml has nearly the same syntax as haml. So if you are tired of verbose templating codes then phphaml can be an interesting solution for php users. So instead of writing something like the following ( I always hated mixing markup and php):

.
.
    <div id="content">
      <table class="config list">
        <tr><th>ID</th><th>Name</th><th>Value</th></tr>
        <?php foreach ($config as $c) { ?>
          <tr class="<?php echo ($class = forClassName($c)); ?>"
                 id="<?php echo "$class_{$c->ID}"; ?>">
            <td><?php echo $c->ID; ?></td>
            <td><?php echo $c->name; ?></td>
            <td><?php echo $c->value; ?></td>
          </tr>
        <?php } ?>
      </table>
    </div>
.
.

You can write it in phphaml as below. Now that’s really some clean code. Not perfect, but better than the above.

#content
      %table.config.list
        %tr
          %th ID
          %th Name
          %th Value
        - foreach ($config as $c)
          %tr[$c]
            %td= $c->ID
            %td= $c->name
            %td= $c->value

An example

Below is a short example on using the phpHaml template. First we define the phpHaml markup. Let us call the file markup.haml and store it in a ‘tpl’ directory. Note: Make sure you keep the indentation at 2 spaces.

!!!
%html
  %head
    %title= $title
  %body
    %h1= $title
    %p= $text
    %select#colors.general
      - foreach ($colors as $c)
        %option{:value => $c}= $c

Update: 27th Jan 2011
If you need to select a particular element in the drop-down you can use the code below.

!!!
%html
  %head
    %title= $title
  %body
    %h1= $title
    %p= $text
    %select#colors.general
      - foreach ($colors as $c)
        - if($c == 'red')
          %option{:value => $c, :selected => 'selected'}= $c
        - else
          %option{:value => $c}= $c

We then call the above template from our php code, passing along some variables.

<?php
 
require_once './includes/haml/HamlParser.class.php';
 
$title = 'phpHaml Demo';
$text =  'A short example of the markup';
$colors = array("blue", "red", "green");
 
display_haml('./tpl/example3.haml', array());
 
?>

The following is what we get after rendering.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN" 
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd">
<html>
  <head>
    <title>phpHaml Demo</title>
  </head>
  <body>
    <h1>phpHaml Demo</h1>
    <p>A short example of the markup</p>
 
    <select class="general" id="colors">
      <option value="blue">blue</option>
      <option value="red">red</option>
      <option value="green">green</option>
    </select>
  </body>
</html>

Although it looks good, there are some major drawbacks. First, the stringent rules for indention, mess that up and it can be quite frustrating to indent everything back again, and second the inability to easily breakup the markup into multiple files.

The PHP script reads two text files – ‘malicious.txt and ‘urls.txt’ : the first containing a list of web pages to be scanned and the other containing malicious script signatures. The script scans the urls for malicious scripts and if any infections are found it saves the result in the ‘infected.txt’ file. The script needs to be run from the command line as you can easily see the progress of the scan if you are scanning a large number of urls.

D:\localhost\test\scan>php url_scan.php

A sample output of a scan is show below:

 
D:\localhost\test\scan>php url_scan.php
 
Checking 3 sites for malicious scripts.
3 malicious signatures in file.
---------------------------------------------------------
 
Now scanning :http://www.amazon.com/
[OK]
---------------------------------------------------------
 
Now scanning :http://www.google.com/
[OK]
---------------------------------------------------------
 
Now scanning :http://www.example.com/admin.php
[OK]
---------------------------------------------------------
 
Total 0 sites infected of 3

Note that the script only scans the url path given and not the complete web site. So if given a url like ‘http://www.example.com’ it will only scan the index file of the site. It may happen that the index file may not be infected but some other file in a sub-directory is, in that case the malicious code will not be found. But a larger percentage of malicious script injections are usually inflicted on the index page.

Setting a cron for automatic scanning

The best way to regularly check for any infections is to setup the script as a cron job. This can help you in checking malicious script on a regular interval, the cron job can then send the ‘infected.txt’ file via a email if any infections are found.

Updating your malicious.txt file

You cannot fight new code injections if your ‘malicious.txt’ file is not updated. So if you find some new malicious Javascript code, then it is essential that you include a new signature in the file. Well I know I’m putting the cart before the horse but you can find various new information about infections at malware.com.br or malwaredomainlist.

Other ways to check malicious code injections

One main problem with the script is that if some new infection occurs and the signature is not in the ‘malicious.txt’ database then that particular infection will be missed. One other solution is to check the filesize of the particular url you are checking. The filesize needs to be added to the ‘urls.txt’ file, so the script can check to see if the filesize of the url scanned is the same as the one given. But for that we will need to use the ftp functions of php but we will leave that to another post.

How Fiddler works

The proxy bascially sits between the Windows WinINET interface and the internet. WinInet is high level API to work with HTTP, FTP, and Gopher protocols.

Installing Fiddler

Fiddler is an HTTP Proxy running on port 8888 on your local PC. You can configure any application which accepts a HTTP Proxy to run through Fiddler so you can debug the particular applications traffic. WinINET-based applications (E.g. Microsoft Office, Internet Explorer, etc) should automatically use Fiddler while it’s running and the “Capture Traffic” box is checked on the Fiddler File menu.

Current versions of Internet Explorer, Google Chrome, Apple Safari, and Opera all automatically use Fiddler when Fiddler is configured to Capture Traffic.

If your browser does not automatically use a proxy then you can configure it to point to address: 127.0.0.1, port: 8888.

Logging with Fiddler

You can easily log HTTP traffic using Fiddler to track down bugs from your browser or website and save it to a file. This can be quite handy if you need to send the logs to the testing team.

Transfer timelines

Like many other browser addons, Fiddler also supports transfer timelines.

Extending Fiddler Using Rules

Fiddler lets you modify its HTTP request or response by using the JScript.NET scripting language. You can also write your own plugins to change Fiddlers behavior in any .NET language. Sadly being from the php camp I do not have much experience there.

Text Encode/Decode Wizard

I specially like the ‘Text Wizard’ which lets you process text in a variety of formats. You can encode/decode data using: Base64, URLEncode, HexEncode, JS string, HTML Encode, UTF-7, Deflated SAML.

HTTP Performance tuning

The author of Fiddler, Eric Lawrence, has an excellent article that goes into the details on HTTP performance tuning.

Fiddler in a Nutshell

Fiddler is an excellent tool for both troubleshooting client/server applications and tweaking the performance of web applications. There are many more things you can do with Fiddler for which this post cannot do justice. Fiddler gives you an inside view into the mechanics of HTTP requests and responses, allowing you to understand what your application is doing. With that knowledge under your belt, you can write better, faster, and lighter applications.

Currently the software includes three tools:
a. Visualizer : Helps to browse components and associations using one simple unified view.
b. Navigator : Interactive visual representation of all code associations.
c. Search : Incremental search for any type of component, including classes and methods.

Visualizer

The visualizer presents the application components and associations in a graphical representation. Each component of the application appears as a node and associations appear as edges, with the association name given on the edges. When there are many associations of a given type, the associations are grouped into a sub-node. The graph can be traversed: expanding nodes to reveal associated components while the layout automatically adjusts.

The following is a visualization of the Amazon API class I had posted about earlier.

Another visualization of WordPress 2.8, centered on index.php.
(Click image to view a larger version)

Navigator

The navigator presents the components and associations of the application which were discovered by the analyzer. It starts from a given selected component and shows all associated components. This includes inbound and outbound associations. The following figure shows the navigator with the WordPress ‘wp_meta()’ function selected.

Search

The search feature lets you search for components by name or a partial name using wild cards.

Price

nWire is not OpenSource, but a 30-day trial is available. Single user license will cost you $59. If you are dealing with plethora of heavy applications, than the price is worth it.

Installation

Papercut is a Windows application requiring .NET Framework 3.5 . Thankfully no installation is required as the application can be run directly. Upon activation it sits in the system tray and pops a message when a email is received from a local web application.

The messages can then be viewed in the Papercut window.

In all a simple and useful tool for any web developer.

Installation

XML_Beautifier being a Pear package we will use the Pear installer as below. I recommend to always use the Pear installer to download packages rather than dowloading it manually as the Pear installer automatically downloads any dependent packages.

pear install XML_Beautifier

Usage

You can beautify a xml file or a on-the-fly generated xml string. The following will nicely format a untidy xml file. The input file must be a valid xml document or the parser will flag an error.

<?php
 
require_once "XML/Beautifier.php";
 
$fmt = new XML_Beautifier();
$result = $fmt->formatFile('unformated.xml', 'beautified.xml');
 
if (PEAR::isError($result)) {
    echo $result->getMessage();
    exit();
}
 
echo "Done";
 
?>

Formatting a xml string is quite as simple.

<?php
 
require_once "XML/Beautifier.php";
 
/* Unformatted xml string */
$xml = '<rootNode><foo   bar = "pear">hello world!</foo></rootNode>';
$fmt = new XML_Beautifier();
echo $fmt->formatString($xml);
 
?>

Formatted xml string.

<rootNode>
    <foo bar="pear">hello world!</foo>
</rootNode>

If you frequently need to format xml documents than it would be better to have a command line access to the beautifier as below.

<?php
 
/** xmlformat.php
 *  usage: php xmlformat.php untidy.xml beautified.xml
 */
 
if ($argc < 3 ) {
    echo "\nUsage: $argv[0] unformatted.xml formatted.xml\n";
} else {
 
    require_once "XML/Beautifier.php";
 
    echo "Formatting {$argv[1]}...";
 
    $fmt = new XML_Beautifier();
    $result = $fmt->formatFile($argv[1], $argv[2]);
 
    if (PEAR::isError($result)) {
        echo $result->getMessage();
        exit();
    }
    echo "Done";
}
 
?>

Options

The XML_Beautifier class also accepts a array of options which you can pass to the constructor during initialization. The details of the options can be found here.

.
.
$options = array(
                    "caseFolding"       => true,
                    "caseFoldingTo"     => "uppercase",
                    "normalizeComments" => true
                );
 
$fmt = new XML_Beautifier($options);
.
.

One thing to note in closing is that the formatting of big xml documents can take some time depending on your machine configuration, so if you plan on formatting documents on a live server you need to take that into account.