Performance testing is one of the most common tasks in Web Service testing and probably one of the areas where the confusion is most common. Not so much in the tasks involved, but rather in the terminology. SoapUI unfortunately add to this confusion by calling the Performance Test functionality LoadTests and we tend to use one word for the other on occasion. We’ll try to give a short overview to Performance Testing here and what it means for Web Service Testing.
What is called a LoadTest in soapUI is actually a Performance Test. You can use soapUI for several different kinds of Performance related tests.
What is Performance Testing?
If we should explain our definition of Performance Testing in one sentence it would be;
“Performance Testing is artificially creating or simulating load and measuring how your environment handles it.”
This means it doesn’t necessary have to be how a system performs under high load, it can also be how it performs under base load or expected load. It doesn’t even have to be structured, automated or created in TestWare like soapUI; simply refreshing you web browser over and over again very fast is a Load Test.
What kinds of Performance Testing are there?
There are a great number of definitions of performance testing, please realize that we know that this is only one way to define performance testing. Based on the above we use as an umbrella for many types of Performance related testing;
• Baseline Testing
This is purely technically exactly what Performance Testing is, but we feel it’s important to establish as a separate task. Baseline Testing examines how a system performs under expected or normal load and creates a baseline against which the other types of tests can be related.
Goal: Find metrics for system performance under normal load
• Load Testing
Load Testing includes increasing the Load and see how the system behaves under higher load. During Load Tests you can monitor response times, throughput, server condition and much more. The goal of Load Testing is not to break the target environment though.
Goal: Find metrics for system performance under high load
• Stress Testing
The goal of Stress Test is exactly that; to find the volume of load where the system actually breaks or is close to breaking.
Goal: Find the system breaking point
• Soak Testing
In order to find system instabilities that occur over time, we need to conduct tests that run over a long period. That is what Soak Testing is for; run Load Tests or even Baseline Tests over a long period of time and see how the target environment handles system resources and that we see no unwanted behavior as a result of the time as a factor. The most common defect found by Soak Testing is memory leaks. The most common scenario for Soak Testing is turning on a number of tests when you leave work on a Friday and let it run over the weekend.
Goal: Make sure no unwanted behavior emerges over time.
• Scalability Testing
Scalability Testing is very much like Load Testing, but instead of increasing the number of requests, we instead increase the size or the complexity of the requests sent. This involves for example sending large requests, large attachments, or deeply nestled requests.
Goal: Find metrics for system performance under high volumes of data.
How do I Performance Test Web Services?
If we look into the unique characteristics of Web Service performance two aspect stands out; on the server side there’s quite a bit of XML processing going on, both XML parsing and XML Serialization, and the thing that often fails first is the processing of the payloads. The reasons why this fails can be multifold; it can be in the platform in the shape of for example weaknesses in the application server, and in the implementation in the shape of unnecessarily complex WSDL’s.
Another factor unfortunately, is security. Those who do web performance testing will know that secure sites behind https have considerably lower performance and in Web Service testing we can add a layer of WS-Security to the layer of HTTP security, decreasing the performance even more.
What does this mean for us when we do Web Service Testing?
The complexity of parsing the XML payloads means that we have to put an extra focus on Scalability Testing, while the issue of security means we will have to focus some extra on doing testing of requests that are secure. If the entire Web Service is secure this means Load Testing is more important especially if WS-Security and token handling is used.
It also means we have to examine the WSDL’s closely. If the requests and responses are either complex or larger, or if they include large attachments, we have to focus on emphasizing that complexity and see how it behaves under load.