Measuring user effectiveness with the Apdex metric

Creating a business application is one thing, measuring user-satisfaction another. Most software houses in recent years have woken up to the importance of user-facing metrics, the knowledge that user satisfaction regarding software application performance is an integral part of business success. But the problem in application performance measurement is that there are too many metrics and measurements which in the end adds nothing of real value to the performance reports. What most people need is a metric that is easy to calculate and interpret. Apdex is one such metric.

Apdex: Measuring user effectiveness

Apdex (Application Performance Index) is an open measurement standard developed by a group of companies. It is a numerical measure of user satisfaction regarding performance of web or desktop applications. It enables management to gain insights about user satisfaction with respect to the developed application.

Apdex converts multiple measurements into a single number on a scale of 0 to 1 (0 = no users satisfied, 1 = all users satisfied). This metric can be applied to any source of end-user performance measurements that gathers timing data.

The index is based on three areas of application responsiveness, we will see this in the context of web applications. The three areas are partitioned by two time limits, T and F. T (satisfied-tolerating
boundary) represents the base time limit (in seconds) below which the application is responsive; and F represents the maximum limit after which the application response drops considerably and the user may close the application or navigate to another site. F is taken as (4xT) or four times T.

Satisfied—The user is completely productive. This represents the time value T below which
users are not impeded by application response time. For e.g when a web page loads in less than 3 seconds.

Tolerating— With response time greater than T the user starts noticing the performance lag but continues the process. E.g. when the page takes around 10-15 seconds to completely load.

Frustrated—Performance with a response time greater than F seconds leads to frustration and the user usually navigates away from the site.

So if we need to measure the Apdex score for a 75 page enterprise web application, we have to initially set the target times T and F around which our three measurements will be based.

Lets set T to 3 seconds and F to 12 seconds. After measuring the load time of the 75 pages we find that 28 pages load in less than 3 seconds, 40 page load between T and F seconds, and the remaining load in more than F seconds. In short:

Load time of 28 pages <= T T < Load time of 40 pages <= F Load time of 7 pages > F

We can then calculate the Apdex score using the following formula:

Which in our case will give a score of 0.64:

The score gives a value of ‘1’ when all the users are satisfied with the application performance, a ‘0’ when no one is satisfied and ‘0.5’ when all the users are tolerating the application performance. For other variations the score can range between 0 – 1.

Advantages of Apdex

The main advantage of the metric is that it provides a single score that lets the IT manager easily compare performace between different applications or different versions of the same applications. In the context of web applications a IT manager can easily compare the Apdex score before and after optimizing a web application.

References:
http://en.wikipedia.org/wiki/Apdex
http://www.apdex.org/index.html