Are customers happy? How SaaS providers can use Apdex to find out

Application response time is key to the user experience, but what's a 'good' response time? SaaS providers can use the Apdex formula to find out.

Editor's note: This is the second part of a two-part series on application performance strategies for Software as a Service providers. Don't miss part one, End-user experience monitoring is key to SaaS provider success.

From the customer's perspective, application performance is synonymous with response time -- none of the metrics for load or activity capture the user experience. Therefore, it is in a Software as a Service (SaaS) provider's interest not just to understand the performance they are actually delivering to users, but also to be open with customers about performance issues and other information.

There are different schools of thought about the importance of measuring performance. Some experts say you can do it objectively with methodologies such as Apdex, but others disagree.

Mark Yablonski, chief technology officer of Valogix, a software company based in Saratoga Springs, N.Y., that offers a SaaS version of its flagship inventory optimization and planning applications, maintains that it's impossible to make generalizations about performance.

"Performance is critical to our business, so we listen to our users and their experiences. When customers complain that a page is loading slowly, we find out how slowly," Yablonski says. "But the problem with performance is that it's a subjective evaluation. People try to put heuristics on it, but every end user is different as to what they will tolerate in terms of good and bad performance. In my opinion, anyone who is trying to measure that is wasting their time."

I can't imagine launching a service and not knowing whether or not your customers are unhappy.

John Essex, New Relic

Application performance exert Peter Sevcik of NetForecast disagrees.

"You can't improve what you don't measure," he says. "You need to find out what your users' performance expectations are and identify a response time threshold at which most of them will be satisfied."

Sevcik recommends translating response time into something meaningful for users. To do this, he created the Application Performance Index, also known as Apdex -- an open standard that defines a method for reporting and comparing the performance of software applications. It provides insight into user satisfaction by specifying a uniform way to analyze and report on the degree to which measured performance meets user expectations.

The Apdex methodology converts many measurements into one number on a uniform scale of 0 to 1 (0 = no users satisfied; 1 = all users satisfied). The resulting Apdex score is a numerical measure of user satisfaction with the performance of enterprise applications. The Apdex formula is the number of satisfied users plus half of the number of tolerating users plus none of the frustrated users, divided by all the total number of users:

Apdext= (satisfied count + tolerating count / 2) / total samples

The subscript t is the target response time. The tolerable time is assumed to be four times the target response time. This ratio is always directly related to users' perceptions of satisfactory application responsiveness.

Example: If there are 100 users with a target time of three seconds, and 60 users experience a response time of less than three seconds, 30 are between three and 12 seconds, and the remaining 10 are above 12 seconds, then the Apdex score is calculated with the following equation:

(60 + 30 / 2) / 100 = 0.75

Although some SaaS providers may feel that picking a target response time is too difficult or subjective, as some users may be more patient than others, Sevcik says the solution lies in the factors that providers can control.

"Research shows that people react very consistently to information displayed on a screen. The greater variable is how the application is designed and what it displays, and this variable is most important to consider when choosing a value for t," he says. "The good news is that variable is within the control of a SaaS vendor, and a SaaS vendor should have all of the information needed set t."

New Relic is a SaaS provider that uses Apdex to report on the performance of its own application performance management product as well as its clients' applications. The San Francisco-based company offers 99.99% availability due to its constant vigilance, says John Essex, marketing director at New Relic.

"We use real end-user monitoring and Apdex to ensure our users are happy," Essex says. "It's about quality of service. We know the minute someone is unhappy. Unhappy customers are at-risk customers. I can't imagine launching a service and not knowing whether or not your customers are unhappy."

About the author:
Rebecca Wetzel is a principal with NetForecast, a network technology consulting firm based in Charlottesville, Va. She provides data communications industry insight and helps vendors and service providers develop successful marketing strategies.

Dig Deeper on SaaS implementation