Executing Tests

Previous Next Contents

5 Executing Tests

The Security TCK uses the JavaTest harness to execute a set of tests in the test suite. For detailed instructions that explain how to run and use JavaTest, see the JavaTest User’s Guide and Reference in the documentation bundle.

This chapter includes the following topics:

Note

The instructions in this chapter assume that you have installed and configured your test environment as described in Chapter 3, "Installation," and Chapter 4, "Setup and Configuration,", respectively.

5.1 Starting All Tests

It is recommended to run the Security TCK from the command line in your shell environment

Note

The mvn command referenced in the following two procedures and elsewhere in this guide is the Apache Maven build tool, which will need to be downloaded separately. The build.xml file in <TS_HOME>/bin contains the various Ant targets for the Security TCK test suite. Ant commands can be run via maven using maven-antrun-plugin.

5.1.1 To Start All tests in Command-Line Mode

  1. Start the tests using the following command:

    mvn clean verify
Example 5-1 Security TCK Signature Tests

To run the Security TCK signature tests, enter the following commands:

cd tck/signaturetest/
mvn clean verify
Example 5-2 Start JavaTests in jakarta-security-tck/security-tck

To run the Javatests:

cd tck/security-tck/
mvn clean verify
Example 5-3 Subset of Test Directories

To run a subset of test directories, enter the following commands:

cd tck/app-custom-identity-store-handler/
mvn clean verify

5.2 Running a Subset of the Tests

Use the following modes to run a subset of the tests:

5.2.1 To Run a Subset of Tests in Command-Line Mode

  1. Change to the directory containing the tests you want to run.

  2. Start the test run by executing the following command:

    mvn verify

    The tests in the directory and its subdirectories are run.

5.3 Running the TCK Against another CI

Some test scenarios are designed to ensure that the configuration and deployment of all the prebuilt Security TCK tests against one Compatible Implementation are successful operating with other compatible implementations, and that the TCK is ready for compatibility testing against the Vendor and Compatible Implementations.

  1. Verify that you have followed the configuration instructions in Section 4.1, "Configuring Your Environment to Run the TCK Against the Compatible Implementation."

  2. If required, verify that you have completed the steps in Section 4.3.2, "Deploying the Prebuilt Archives."

  3. Run the tests, as described in Section 5.1, "Starting JavaTest," and, if desired, Section 5.2, "Running a Subset of the Tests."

5.4 Running the TCK Against a Vendor’s Implementation

This test scenario is one of the compatibility test phases that all Vendors must pass.

5.5 Test Reports

A set of report files is created for every test run. These report files can be found in the report directory you specify. After a test run is completed, the JavaTest harness writes HTML reports for the test run. You can view these files in the JavaTest ReportBrowser when running in GUI mode, or in the web browser of your choice outside the JavaTest interface.

To see all of the HTML report files, enter the URL of the report.html file. This file is the root file that links to all of the other HTML reports.

The JavaTest harness also creates a summary.txt file in the report directory that you can open in any text editor. The summary.txt file contains a list of all tests that were run, their test results, and their status messages.

5.5.1 Creating Test Reports

Use the following modes to create test reports:

5.5.1.1 To Create a Test Report in Command-Line Mode
  1. Specify where you want to create the test report.

    1. To specify the report directory from the command line at runtime, use:

      ant -Dreport.dir="report_dir"

      Reports are written for the last test run to the directory you specify.

  2. To specify the default report directory, set the report.dir property in <TS_HOME>/bin/ts.jte.
    For example:

    report.dir="/home/josephine/reports"
  3. To disable reporting, set the report.dir property to "none", either on the command line or in <TS_HOME>/bin/ts.jte.
    For example:

    ant -Dreport.dir="none"

5.5.2 Viewing an Existing Test Report

Use the following modes to view an existing test report:

5.5.2.2 To View an Existing Report in Command-Line Mode

Use the Web browser of your choice to view the report.html file in the report directory you specified from the command line or in <TS_HOME>/bin/ts.jte.


Previous Next Contents
Eclipse Foundation Logo  Copyright © 2017, 2022 Oracle and/or its affiliates. All rights reserved.