Apdex: Measuring user effectiveness

Apdex (Application Performance Index) is an open measurement standard developed by a group of companies. It is a numerical measure of user satisfaction regarding performance of web or desktop applications. It enables management to gain insights about user satisfaction with respect to the developed application.

Apdex converts multiple measurements into a single number on a scale of 0 to 1 (0 = no users satisfied, 1 = all users satisfied). This metric can be applied to any source of end-user performance measurements that gathers timing data.

The index is based on three areas of application responsiveness, we will see this in the context of web applications. The three areas are partitioned by two time limits, T and F. T (satisfied-tolerating
boundary) represents the base time limit (in seconds) below which the application is responsive; and F represents the maximum limit after which the application response drops considerably and the user may close the application or navigate to another site. F is taken as (4xT) or four times T.

■ Satisfied—The user is completely productive. This represents the time value T below which
users are not impeded by application response time. For e.g when a web page loads in less than 3 seconds.

■ Tolerating— With response time greater than T the user starts noticing the performance lag but continues the process. E.g. when the page takes around 10-15 seconds to completely load.

■ Frustrated—Performance with a response time greater than F seconds leads to frustration and the user usually navigates away from the site.

So if we need to measure the Apdex score for a 75 page enterprise web application, we have to initially set the target times T and F around which our three measurements will be based.

Lets set T to 3 seconds and F to 12 seconds. After measuring the load time of the 75 pages we find that 28 pages load in less than 3 seconds, 40 page load between T and F seconds, and the remaining load in more than F seconds. In short:

Load time of 28 pages <= T
T < Load time of 40 pages <= F
Load time of 7 pages > F

We can then calculate the Apdex score using the following formula:

Which in our case will give a score of 0.64:

The score gives a value of ’1′ when all the users are satisfied with the application performance, a ’0′ when no one is satisfied and ’0.5′ when all the users are tolerating the application performance. For other variations the score can range between 0 – 1.

Advantages of Apdex

The main advantage of the metric is that it provides a single score that lets the IT manager easily compare performace between different applications or different versions of the same applications. In the context of web applications a IT manager can easily compare the Apdex score before and after optimizing a web application.

References:
http://en.wikipedia.org/wiki/Apdex
http://www.apdex.org/index.html

With PHP functions you can inspect variables, watch changes in variables over the whole runtime, measure partial runtimes, set checkpoints and write logfiles. debugConsole also replaces the PHP error-handling so that notices, warnings and other errors are also shown in a popup, instead of displaying them in the application to be debugged.

IP-based access restrictions can be defined in the configuration file so that the popup only opens for developers from that IP and not for visitors. After finishing work, the popup can simply be switched off.

Below is a simple example, and its output:

/* load debugConsole functionality */
require_once 'debugConsole/debugConsole.php';
 
/* test variable debugging */
$foo = 'Hello World!';
dc_dump($foo, '$foo wants to say something:');
 
/* test variable debugging again */
$bar = 42.0;
dc_dump($bar, '$bar tells us the meaning of life:');
 
/* test array debugging */
$foobar = array (
	'foo' => $foo,
	'bar' => $bar
);
 
dc_dump($foobar, '$foobar is a neat array!');

Take an example class method addUser() shown below. When the query is executed the result is logged in a file with the function name and a user_id or a serialized error string in case of an error. Whenever the application is in a testing phase, the error logs are an immense help in diagnosing error points. This is better than a splattering of print_r statements all around the code.

define("LOG",  true);
 
/** Add new user to the system
 *
 *  @param array $user_data
 *  @return boolean
 */
public function  addUser($user_data)
{
.
.
    if(!$sql->execute()) {
        if(LOG) {
            $err_string = serialize($sql->errorInfo());
            $this->_log->LogError("addUserd | error | " . $err_string);
        }
        return 0;
    } else {
        if(LOG) {
            $this->_log->LogInfo("addUser| no. | " . $user_id);
        }
        return 1;
    }
}

The contents of a sample log file are shown below.

2010-07-31 4:52:58 - INFO  | addUser | no. | h1280551978 
2010-07-31 4:53:04 - INFO  | user edit | no. | 1 
2010-07-31 4:53:06 - INFO  | user edit | no. | 1 
2010-07-31 4:53:07 - INFO  | addUser| no. | 1 
2010-07-31 4:53:09 - ERROR | addUser| error | 
a:3:{i:0;s:5:"23";i:1;i:148;i:2;s:30:"Column 'fileno' cannot be null";}

Although logs can add some overhead to your code, it is a small price to pay for the benefits it provides. And if you do not want the logging feature you can easily disable it by setting the LOG constant to ‘false’ in the above code.

Which logger to use

Although there a variety of libraries, I prefer KLogger as it has a small footprint and is easy to use. Other more advanced contender is Log4PHP, a PHP port of log4j, a Java logger. Zend_Log also looks nice but I am yet to give it a try.

Analyzing logs

For big, frequently used applications, log files can quickly get quite huge. The easy way to analyze them is to use GUI tools like LogMX and Apache Chainsaw. But if you just need to get some simple statistics from your logs, you could use the trusty old grep and other *nix utilities. For example to count the total error lines having the string ‘mysql’ you could use the following code.

$ grep 'ERROR ' activity.log | grep 'mysql' | wc -l

The advantage of scripts over GUI tools is that you can automatically process them using CRON jobs, e.g – emailing the admin if some particular type of errors in a log reaches some critical limit.

In conclusion, logs are one of the simplest ways of keeping track of your code problems or to check the execution sequence of your statements. This can be immensely helpful during development and for tracking down bugs.

Globs

Globs or the verb Globbing is familiar to most people who have ever used file matching patterns. If you have ever searched for a file in Linux or DOS using a line like *.exe or photos*, then you have used globs. But globbing in Selenium is not as rich as the one in Linux – it supports only three special characters: *, ? and [ ].

* matches any number of characters, by any we mean ‘nothing’, ‘a single character’ or ‘many characters’.

? , matches a single character.

[ ] , called a character class, lets you match any single character found within the brackets. e.g

[0-9] matches any digit
[a-zA-Z] matches any alphabet, regardless of case

To specify a glob in a selenium command, prefix the pattern with the glob: string. For example if you would like to search for the texts color or colour then you can use the colo*r glob as shown below.

However you are free to eliminate the glob: prefix and only specify the text pattern because globbing patterns are the default in Selenium.

Regular Expression patterns

Of the three type of patterns, Regular Expressions are the one that are the most useful. Selenium supports the complete set of RegEx patterns that Javascript supports. So now you are not limited by the *,? And [] globbing patterns. To use RegEx patterns you need to prefix each RegEx with either regexp: or regexpi:, the latter being case-insensitive.

For example the following will test if a input field with the id ‘name’ contains the string ‘javascript’, ‘JavaScript’ or ‘Javascript’.

Below are a few common regular expression patterns:

regexp:(0[1-9]|1[012])[- /.](0[1-9]|[12][0-9]|3[01])[- /.](19|20)\d\d
:match a date in ‘mm/dd/yyyy’ format with any of the ‘-’, ‘/’, ‘.’ as separators.

regexpi:^[A-Z0-9+_.-]+@[A-Z0-9.-]+$
:match a generic email address.

regexp:^[0-9]{5}(?:-[0-9]{4})?$
:match a ZIP code (U.S. postal code), allowing both the five-digit and nine-digit (ZIP + 4) formats.

regexp:^(?!000|666)(?:[0-6][0-9]{2}|7(?:[0-6][0-9]|7[0-2]))-?(?!00)[0-9]{2}-(?!0000)[0-9]{4}$
:match U.S Social Security numbers in the in the AAA-GG-SSSS format.

regexp:^(https?|ftp|file)://.+$
:match almost any url.

Exact Patterns

Patterns with the prefix ‘exact:’ will match the given text as it is. For example if you give the search pattern as below, then it will match a glob pattern ‘*’ or ‘*.java’.

But if you want an exact match with the value string, i.e without the glob operator doing its work, you use the ‘exact’ pattern as below. In this example the ‘*’ (asterisk) will work as a normal character rather then a pattern-matching wildcard character.

In conclusion the glob: and the exact: pattern are the subsets of the Regular Expression pattern matcher. Everything you can do with glob: or exact: you can accomplish with RegExp.

Faker.js is a JavaScript implementation inspired by Benjamin Curtis’s Ruby Gem Faker and Perl’s Data::Faker that lets you generate commonly required data quickly. You can check the demo page to get an idea.

Getting Fake.js

You can download the library form github and include in your pages where you need to generate the data.

..
<script src = "Faker/Faker.js" type = "text/javascript"></script>
..

Generating fake data

The API is divided into various sections as show below:

# Name
    * findName
# Address
    * zipCode
    * city
    * streetName
    * streetAddress
    * secondaryAddress
    * ukCounty
    * ukCountry
# PhoneNumber
    * phoneNumber
# Internet
    * email
    * userName
    * domainName
    * domainWord
...

So if you need to generate a random email address or a zip code you could do it like this:

var randomEmail = Faker.Internet.email();
var randomZip = Faker.Address.zipCode();

Some sample API methods are shown below, but there are more of them which you can find here in the API section.

<script>
var longParagraph = Faker.Lorem.paragraphs();
var singleSentence = Faker.Lorem.sentence();
var companyName= Faker.Company.companyName();
var catchPhrase = Faker.Company.catchPhrase();
</script>

One of the interesting methods is ‘ Faker.Company.catchPhrase()’, which generates catchy technological phrases. A kind of a tag line generator for your new multi-million dollar company . A sample output is shown below.

Stand-alone 5th generation utilisation
Ergonomic explicit focus group
Horizontal human-resource solution
Automated contextually-based knowledge base
Distributed multi-state encoding
Extended zero administration interface
Proactive coherent productivity
Streamlined national approach
Total transitional algorithm
Stand-alone fault-tolerant moderator

The API also includes several helper methods that lets you create bulk fake data using the single API methods. For example you can generate complete user information templates cards by using a single helper method ‘Faker.Helpers.userCard()’. A sample run is shown below:

name: Blair Nikolaus
username: Fernando_Olson
email: Jacklyn_Brekke@aurore.name
address:street: Kozey Meadow
suite: Apt. 664
city: Felipastad
zipcode: 88071
 
phone: 571.540.8605 x136
website: reba.co.uk
company:name: Kuhic and Sons
catchPhrase: Integrated solution-oriented Graphical User Interface
bs: seize scalable web services

The ‘Faker.Helpers.userCard()’ method returns a nested object, so you will need to recursively traverse the object to extract the data. A simple function to do the same is given below:

<script>
// Recursively traverse a nested Javascript object and return it.
function printObj(obj)
{
  var prop, res = '';
 
  for(prop in obj)
  {
    if(typeof obj[prop] == 'object'){
        res += prop + ':' + printObj(obj[prop]) + '<br>';
    }else{
        res += prop + ': ' + obj[prop] + '<br>';
    }
  }
   return res;
}
</script>

So now you can do something like this:

...
var obj = Faker.Helpers.userCard();
alert( printObj(obj) );
...

Another Randomizer library

Club AJAX Mock Data Randomizer library is another library that lets you generate random data quickly. This library lets you generate random dates, colors, boolean values which is lacking in the Faker library. Check a simple demo using this library.

Selenium IDE offers many useful commands (often called Selenese) for testing purposes. You use this sequence of commands to create various tests. But many times these commands are not sufficient and you need to add your own custom commands. For example a reader recently requested on how to input unique email ids for testing. Of course you can easily do this using the Selenium RC server and a language like PHP or Java. But we want to accomplish this in the IDE itself.

Types of Selenium commands

Selenium commands come in three varieties : Actions, Accessors and Assertions. Each Selenium command being given in the format below:

Actions are commands which in general modify the state of the application; click, typeKeys, deleteCookie etc. are all actions. Many actions also come with a sibling with a “AndWait” suffix added; e.g. ‘typeKeysAndWait’. This tells Selenium to wait until the page loads after performing the particular action.

Accessors examine the state of the application and store the results in variables, e.g. “storeLocation”.

Assertions are like Accessors, but they verify that the state of the application conforms to what is expected

In this post we will see how to write Selenium Action command using a simple example.

Writing your first custom Selenium Action Command

In the following example we will add a new action, ‘typeRandomEmail’, which generates a random email address and types it in an input field.

Every Selenium Action command starts with a ‘do’ prefix. The corresponding action comprising ‘AndWait’ is automatically registered. e.g when we create the ‘typeRandomEmail’ action, the ‘typeRandomEmailAndWait’ command is automatically added to the command list. An action command takes up to two parameters, which will be passed the second and third column values in the test table. In our example we will be using only one parameter – the locator id where the random email will be typed.

The following shows the code for the typeRandomEmail action.

 
Selenium.prototype.doTypeRandomEmail = function(locator) {
    /**
    * Sets the value of an input field to a random email id, 
    * as though you typed it in.
    *
    * @param locator an <a href="#locators">element locator</a>
    */
 
    // All locator-strategies are automatically handled by "findElement"
    var element = this.page().findElement(locator);
 
    /* The following block generates a random email string */
    var allowedChars = "abcdefghiklmnopqrstuvwxyz";
    var stringLength = 8;
    var randomstring = '';
 
    for (var i=0; i<stringLength; i++) {
        var rnum = Math.floor(Math.random() * allowedChars.length);
        randomstring += allowedChars.substring(rnum,rnum+1);
    }
 
    // Append a domain name
    randomstring += "@somedomain.com"
 
    // Replace the element text with the new text
    this.browserbot.replaceText(element, randomstring);
};

Importing your custom commands in Selenium

Now that we have completed writing our own action command, it is time to import it to the Selenium IDE. Your custom commands are usually added to the file ‘user-extensions.js’. By default the ‘user-extensions.js’ file does not exist, so you need to first create a blank file into which you will add the above code. Once that is done you then need to tell Selenium to load the particular extension file from the ‘Options’ menu.

Once you have added the file to the options page you need to restart the Selenium IDE for the changes to take effect.

Whenever you add some new code or make changes to the exiting one in the ‘user-extensions.js’, you need to restart the Selenium IDE for the changes to take effect.

Now once you have restarted the IDE, you can see that your command has been added to the drop-down.

Writing other Action commands

You may be wondering how to go about writing other Selenium Action commands; what the this.browserbot object means and where it is defined. Although no clear documentation exists, the Selenium source code can itself be a valuable tool. All the Selenium commands are stored in the ‘selenium-api.js’ file. I have included it here for easy download and reference. The original source file and other support files are located in the ‘selenium-ide.jar’ file in your Selenium installation directory. Unzip that file and you have all the Selenium source code at your disposal.

Dependencies in PHPUnit tests

PHPUnit 3.4 now supports dependencies between different test methods. It allows you to execute a particular test ONLY IF the test that it depends on executes successfully. Take the following example (Listing 1.) where we test a linked-list class I developed earlier.

A simple unit-test with PHPUnit

The test basically creates a link-list with 100 nodes, and tests a few class methods – reverseList, deleteFirstNode, deleteLastNode. As you can see the linklist creation part and method tests are combined in the single test method testLinkList().

The primary focus here is on testing the three class functions. There is nothing wrong with this. But what if the linklist creation part itself fails before the other tests are conducted? Of course we can add another test to check that the links are being created correctly, but that mixes up the two parts.

Listing 1.

<?php
 
require_once 'linklist.class.php';
require_once 'PHPUnit/Framework.php';
 
class LinkListTest extends PHPUnit_Framework_TestCase
{
    /* How many nodes to create */
    private $_nNodes = 100;
 
    public function testLinkList()
    {
        /* Linklist creation block */
        $theList = new LinkList();
 
        for($i=1; $i <= $this->_nNodes; $i++)  
        {
            $theList->insertLast($i);
        }
 
 
        /* 
            Linklist methods testing block :
 
            $theList->totalNodes() is a public function 
            in the 'linklist' class that returns the total
            number of nodes in the link-list.
        */
 
        $theList->reverseList();
        $this->assertEquals($this->_nNodes, $theList->totalNodes());
 
        $theList->deleteFirstNode();
        $this->assertEquals($this->_nNodes - 1, $theList->totalNodes());
 
        $theList->deleteLastNode();
        $this->assertEquals($this->_nNodes - 2, $theList->totalNodes());
    }
}
 
?>

Splitting up tests

A better way is to split the linklist creation part and class method testing part into two separate test functions as below (Listing 2). Here the linklist is created in the ‘testCreateList()’ method and the testing of the class methods is done in the ‘testLinkList()’ method. This allows us to keep the two testing blocks separate.

Listing 2.

<?php
 
require_once 'linklist.class.php';
require_once 'PHPUnit/Framework.php';
 
class LinkListTest extends PHPUnit_Framework_TestCase
{
    /* How many nodes to create */
    private $_nNodes = 100;
 
    public function testCreateList()
    {
        $theList = new LinkList();
 
        for($i=1; $i <= $this->_nNodes; $i++)
        {
            $theList->insertLast($i);
        }
        $this->assertTrue(!empty($theList));
 
        return $theList;
    }
 
    /**
     * @depends testCreateList
     */
    public function testLinkList($theList)
    {
        $theList->reverseList();
        $this->assertEquals($this->_nNodes, $theList->totalNodes());
 
        $theList->deleteFirstNode();
        $this->assertEquals($this->_nNodes - 1, $theList->totalNodes());
 
        $theList->deleteLastNode();
        $this->assertEquals($this->_nNodes - 2, $theList->totalNodes());
    }
}
 
?>

The important point to note here is that we have specified that the ‘testLinkList‘ method depends on the ‘testCreateList‘ method, with a @depends annotation. This tells PHPUnit that if the ‘testCreateList‘ test fails then the test dependent on it, ‘testLinkList‘, will have to be skipped, which is a logical way to go.

/**
 * @depends testCreateList
 */

Producers & Consumers in a Test

First some definitions:

a. A producer is a test method that yields its unit under test as return value.
b. A consumer is a test method that depends on one or more producers and their return values.

What it means is that a ‘Producer’ is a test method that returns some data that will be used by a ‘Consumer’ test method. In our example the ‘testCreateList‘ is a Producer and the ‘testLinkList‘ is the Consumer. The ‘testCreateList‘ method returns a linklist object which is then received by the dependent method ‘testLinkList‘. So if for some reason the ‘testCreateList‘ fails to return some data then the test that depends on it, ‘testLinkList‘, will have to be skipped, as it has nothing to work with.

Bear in mind that it does not mean that every test method that depends on some other method has to accept some arguments from the method it is dependent on. Lets breakup the above example further (Listing 3). Here we have added one more test method, ‘testNodesDefined‘, which makes sure that the ‘$_nNodes’ variable is not null. The ‘testCreateList‘ method is dependent on this method, so that if the ‘$_nNodes’ is null then the ‘testCreateList‘ is skipped, which causes the test ‘testLinkList‘ to be skipped.

Listing 3.

<?php
 
require_once 'linklist.class.php';
require_once 'PHPUnit/Framework.php';
 
class LinkListTest extends PHPUnit_Framework_TestCase
{
    /* How many nodes to create */
    private $_nNodes = 100;
 
    public function testNodesDefined()
    {
        $this->assertNotNull($this->_nNodes);
    }
 
    /**
     * @depends testNodesDefined
     */
    public function testCreateList()
    {
        $theList = new LinkList();
 
        for($i=1; $i <= $this->_nNodes; $i++)
        {
            $theList->insertLast($i);
        }
        $this->assertTrue(!empty($theList));
 
        return $theList;
    }
 
    /**
     * @depends testCreateList
     */
    public function testLinkList($theList)
    {
        $theList->reverseList();
        $this->assertEquals($this->_nNodes, $theList->totalNodes());
 
        $theList->deleteFirstNode();
        $this->assertEquals($this->_nNodes - 1, $theList->totalNodes());
 
        $theList->deleteLastNode();
        $this->assertEquals($this->_nNodes - 2, $theList->totalNodes());
    }
}
 
?>

A output from a failed test, when we set ‘$_nNodes’ to null, is shown below.
Listing 4.

D:\localhost\datastructures>phpunit --verbose LinkListTest
PHPUnit 3.4.2 by Sebastian Bergmann.
 
LinkListTest
FSS
 
Time: 0 seconds
 
There was 1 failure:
 
1) LinkListTest::testNodesDefined
Failed asserting that <null> is not null.
 
D:\localhost\test\datastructures\LinkListTest.php:13
 
There were 2 skipped tests:
 
1) LinkListTest::testCreateList
This test depends on "LinkListTest::testNodesDefined" to pass.
 
 
2) LinkListTest::testLinkList
This test depends on "LinkListTest::testCreateList" to pass.
 
 
FAILURES!
Tests: 1, Assertions: 2, Failures: 1, Skipped: 2.
 
D:\localhost\test\datastructures>

Note: The tests are executed in the sequence in which you wrote them. So take care while writing dependent tests. For example, sequencing test methods as below will not work.

    public function testNodesDefined()
    {
        $this->assertNotNull($this->_nNodes);
    }
 
    /**
     * @depends testCreateList
     */
    public function testLinkList($theList)
    {
        ..
    }
 
    /**
     * @depends testNodesDefined
     */
    public function testCreateList()
    {
        ..
    }

What all of this has brought us

Testing a complex class in a single humongous test method is not wrong, but can create a confusing mess of ‘asserts’. Splitting a single test method into multiple dependent tests can help you segregate your test methods by functionality which in turn will get you to quickly localize defects in your code.

Editing the selenium-testrunner.js file

First you have to find the ‘selenium-testrunner.js’ file. It is located in your ‘Windows\Application Data’ directory. The rough path is given below:

Documents and Settings\Application Data\Mozilla\Firefox\
Profiles\extensions\{a6fd85ed-t567-s34f-a5af-ddf48bda539f}\content-files

Your exact directory path will be different. It is best do do a search for the file in your ‘Mozilla’ directory. Once you have found it, you can look for the following lines of code:

reset: function() {
        this.runInterval = this.speedController.value;
        this._switchContinueButtonToPause();
    },

and replace it with:

reset: function() {
        this.runInterval = 2000;
        this._switchContinueButtonToPause();
    },

This will by default add a 2 second delay after every command execution. Note that as we have made hardcoded changes to the speed variables, the speed slider in the TestRunner window will no longer have any effect on the execution.

Setting the default speed controller delay to a high value

In view of a recent comment, you can also set the default high delay for your slider by replacing the following lines in ‘selenium-testrunner.js’.

this.speedController = new Control.Slider('speedHandle', 'speedTrack', {
            range: $R(0, 1000),
            onSlide: fnBindAsEventListener(this.setRunInterval, this),
            onChange: fnBindAsEventListener(this.setRunInterval, this)
        });

with

this.speedController = new Control.Slider('speedHandle', 'speedTrack', {
            range: $R(100, 5000),
            onSlide: fnBindAsEventListener(this.setRunInterval, this),
            onChange: fnBindAsEventListener(this.setRunInterval, this)
        });

Which will set the default delay to 100 milliseconds and the maximum to 5 seconds. Now you can also use the speed slider.

The most common sense thing you can do is to test your code even for small changes; whether you are testing it yourself or sending it to the testing team. Seems simple, but how many times have you really done that. Its not that we are stupid, rather software was never so simple. Maybe we are careless sometimes. The moral is plain – treat even small changes important, and move it through a test before committing it to a version control.

The following figure from Steve Mcconnell’s Code Complete shows the relation between lines of code changed and errors introduced.

Data Generation

Once you connect ADG to your MySQL server and register a database, you are ready to go. Generating the test data is quite simple. First you create a project for your data generation work. Once done, you select the table and fields you want to generate data for, select the type of test data to generate – dates, urls, names etc. Set the total number of records to generate and then run the project. This will create the test data for your database project. The test data can be saved directly to the connected database or you can save it as a separate SQL script, YAML or CSV file. Below is a sample screenshot of a data generation project.

The Advanced Data Generator comes with a very useful repository of real-life like data, in order to generate meaningful test-data for the database. You can also use custom SQL scripts or Macros to generate custom string data.

In closing

Although not free, its a small price to pay for the excellent automation it provides during database creation.