Running Functional Tests

PDF Print E-mail
User Rating:  / 98
Rate this article: PoorBest 

1. TestRunner description of parameters

Running functional tests from the command-line is straightforward using the included testrunner.bat/.sh script, which takes a number of arguments to control which tests to run, output, etc:

  • a : Turns on exporting of all test results, not only errors
  • A : Turns on exporting of all results using folders instead of long
  • c : The TestCase to run, used to narrow down the tests to run
  • D : Sets system property with name=value
  • d : The domain to use in any authentications, overrides any domain set for any TestRequests
  • e : The endpoint to use when invoking test-requests, overrides the endpoint set in the project file
  • E : Sets which environment to use (SoapUI Pro only)
  • F : Sets the format of the report specified with the -R option, for Printable reports this is one of PDF, XLS, HTML, RTF, CSV, TXT, and XML. For Data Export this is either XML or CSV (SoapUI Pro only)
  • f : Specifies the root folder to which test results should be exported (see below)
  • G : Sets global property with name=value
  • g : Sets the output to include Coverage HTML reports ( SoapUI Pro only )
  • h : The host:port to use when invoking test-requests, overrides only the host part of the endpoint set in the project file
  • I : Do not stop if error occurs, ignore them
  • i : Enables SoapUI UI-related components, required if you use the UISupport class for prompting or displaying information
  • j : Turns on exporting of JUnit-compatible reports, see below
  • M : Creates a Test Run Log Report in XML format
  • m : Sets the maximum number of TestStep errors to save for each
  • o : Opens the generated report in a browser (SoapUI Pro only)
  • P : Sets project property with name=value, e.g. -Pendpoint=Value1 -PsomeOtherProperty=value2
  • p : The password to use in any authentications, overrides any password set for any TestRequests
  • r : Turns on printing of a small summary report (see below)
  • R : Selects which report to generate for the test objects executed, for example if running the entire project, this could specify the name of a test-suite-level report that would be generated for each TestSuite. The report is saved as specified with the -F option to the folder specified with the -f option. (SoapUI Pro only)
  • S : Sets to save the project file after tests have been run
  • s : The TestSuite to run, used to narrow down the tests to run
  • t : Sets the soapui-settings.xml file to use, required if you have custom proxy, ssl, http, etc setting
  • u : The username to use in any authentications, overrides any username set for any TestRequests
  • v : Sets password for soapui-settings.xml file
  • w : Sets the WSS password type, either 'Text' or 'Digest'
  • x : Sets project password for decryption if project is encrypted

So for example:

testrunner.bat -FPDF -R"Project Report" c:\projects\my-soapui-project.xml

Runs all tests in the specified my-soapui-project.xml file and creates the default project report available with SoapUI Pro.

2. Launching the TestRunner from within SoapUI

For getting started with the command line testrunner, SoapUI includes a "Launch TestRunner" action available from Project, TestSuite or TestCase popup menus, which launches the bundled command-line tools from inside SoapUI. (The IDE plugins do not include these runners, you will need to download/install SoapUI seperately and point the "TestRunner Path" option in this dialog to the install directory). The dialog looks as follows:

launch-test-runner-basic

Here we've specified some initial settings to run the current TestCase, in the Report tab we can specify which reports to generate:

launch-testrunner-reports

Now if we run the testrunner we get:

launch-testrunner-completed

Scrolling back up in the opened window we can see the actual command issued at the command-line:

launch-testrunner-command

Copy and paste this into your favorite automation tool for rerun these tests as configured.

3. Exporting parsable results from SoapUI

When integrating SoapUI's testrunner with Continuous Integration (CI) you can parse the results of an automated test run in one of the following ways:

  1. You can use the "-r" option of testrunner script to output the results to the console and parse the results from there
  2. You can generate a JUnit style report that will create an HTML and XML report files you can parse
  3. You can also export your results in CSV or XML format to parse the results in your CI tool