Command Line Execution

NOTE: This page contains information for SoapUI Pro that has been replaced with ReadyAPI. To try enhanced REST testing functionality, feel free to download a ReadyAPI trial from our website.


Download ReadyAPI and Start Performance Testing


Once created, you might want to run your LoadTests from the command-line, maybe as part of a continuous integration build, or for monitoring the day-to-day performance of your services. SoapUI provides both a command-line runner, and plugins for maven to do this, but since the maven-plugin is covered elsewhere, we will focus on the command-line runner instead.

The runner is available in your SoapUI\bin folder and appropriately named loadtestrunner.bat/.sh. It takes a bunch of arguments related to reports, properties, etc. which can make the creation of the initial argument-list rather tedious (the command-line options are all covered in detail in the LoadTestRunner Reference, here we will just focus on the most common ones). Fortunately the UI includes a wizard for launching the LoadTestRunner, which will help you assess the output as well as to create a command-line invocation string.

The "Launch LoadTestRunner" option is available from the LoadTest popup-menu:

launch-loadtestrunner-option

It brings up the following dialog:

launch-loadtestrunner-tab1

By default the dialog will select the corresponding LoadTest, but the TestSuite/TestCase/LoadTest drop-downs allow you expand the scope of the execution, for example setting all 3 to "<all>" will run all LoadTests in the project. By default SoapUI uses the loadtestrunner in the SoapUI\bin folder.

You can change to another version (perhaps for testing purposes) by specifying the path of the desired loadtestrunner in the “TestRunner Path” field.
The "Save Project" option ensures that the current Project containing the LoadTest(s) to run is saved before launching, so that any modification made in the UI apply to the command-line execution.

The "Save After" option should be used when the executed LoadTest modifies some state in the project file that should be persisted, most notably if you are using a DataGen TestStep with a Number property that you don't want restarting for each run.

The Overrides tab allows you to override the endpoint, host and port used by all requests in the LoadTest.

launch-loadtestrunner-tab2

The configured Limit and initial ThreadCount can be changed to alternative values, allowing you to easily rerun the same LoadTest with different limits and loads. The security-related overrides apply at the request level, for example you might be changing host to a server that requires Basic or WSS authentication.

The Reports tab allows you to configure which reports to generated for your LoadTest results and where to save them.

launch-loadtestrunner-tab3

The last two options are disabled in the open-source version since they are specific to the reporting functionality in ReadyAPI. The "Report to Generate" field should contain the name of one of the corresponding reports available in the "Create Report" dialog:

create-report-names

Correspondingly, the "Report Format(s)" option should then contain a comma-separated list of formats to generate. For printable reports the available values are PDF, XLS, HTML, RTF, CSV, and XML, for Data Export they are XML and CSV.

The Properties-tab, allowing you to override any global, system or project properties when executing the LoadTest(s);

launch-loadtestrunner-tab4

The obvious use case is if you have parameterized your TestCase(s) somehow and need to change those parameters accordingly.

Last up is the Custom Args tab which allows you to add any custom arguments to the command-line invocation:

launch-loadtestrunner-tab5

Ok, now let’s actually run the LoadTest by pressing the "Launch" button. A window pops up showing the command-line invocation and output of the LoadTestRunner as it executes, when the LoadTest finishes we see the following:

launch-loadtestrunner-output-2

All seems to have gone well, the log tells us the LoadTest Report has been created in the specified folder, opening it shows us the desired results;

loadtestreport-output

The above shows us one other (rather important) fact to be aware of, running tests from the command-line can be slightly faster (and thus give better results) compared to running them from the UI. If we run the above test in ReadyAPI we get:

loadtestreport-output-in-ui

Which is about 7% below the TPS and AVG values measured from the command-line. The difference is due to the UI overhead imposed by updating the table, etc.

Let's run the exact same test from the console; as you could see in the first output screenshot, the loadtestrunner.bat file gets launched as follows:

launch-loadtestrunner-output-1

Which we can copy-paste into notepad and get the following arguments:

loadtestrunner.bat -ehttp://127.0.0.1:8088/mockServiceSoapBinding 
-s"Simple TestSuite" -c"Simple Login and Logout w. Properties Steps" 
-l"LoadTest 1" -m120 -n10 -r 
-fC:\Work C:\Users\UserName\SoapUI-Tutorials\Sample-SOAP-Project-soapui-project.xml

If this is the desired configuration, we can take this string and use it in other command-line invocations or when creating automated runs, here from the command-prompt (be sure to surround arguments containing spaces with quotes).

loadtestrunner-from-console-1

And off it goes!