Automated Testing for the ThreadFix CLI

The Task
ThreadFix offers a command line interface jar to create teams, add applications, assign tags, search for vulnerabilities, and much, much more from the shell or command prompt. The number of actions available in the CLI has grown over time, and with the introduction of permissions-restricted API access, it has become less and less viable to manually test the various scenarios that this component of the ThreadFix ecosystem would be subject to.

(The current list of CLI actions can be viewed here.)

The Approach
The ThreadFix QA team has a robust JUnit/Selenium regression suite that Jenkins triggers nightly runs on. The goal is to execute CLI commands against a deployed ThreadFix instance during the regression run, then to capture and analyze the output.

ProcessBuilder to the Rescue
The driving force behind executing our shell commands from Java was the ProcessBuilder class. With this out-of-the-box class, you simply pass a variable number of String parameters or a single String collection to the constructor, set the directory to execute the command from, and then call start(). You will have to catch any IOExceptions when the command is executed.

ProcessBuilder processBuilder = new ProcessBuilder(finalArgs); File(workingDirectory));
try {
    Process process = processBuilder.start();
} catch (IOException e) {
    log.error("Process Builder command could not be executed gracefully.", e);

Setting (and Settling) Arguments
All of ThreadFix’s CLI actions are called with “java -jar <cli-jar>“, followed by an argument for the intended action and any other required parameters. For example, the call to create a team would be:

java -jar tfcli.jar -ct SomeTeam

With this in mind, we decided to represent the shared starting arguments as a static String list variable called startArgs in our class. The DIRECTORY variable is also instantiated here.

private static List<String> startArgs = list();
private static final String DIRECTORY = ".." + File.separator + "threadfix-cli" + File.separator + "target";

It was important that the CLI tests be executable on both our local Windows machines, as well as the Linux QA servers, hence the use of File.separator when specifying the directory path. One difference when executing ProcessBuilder commands in each of these OS environments is that Windows requires the arguments to begin with “CMD” and “/C” in order to indicate that the command should be run in the context of the command prompt. Our Linux servers did not require this set-up with their terminals.

The following static block takes care of this start-up configuration and sets the startArgs variable for all methods in the class to access.

static {
    if (System.getProperty("").startsWith("Windows")) {
        startArgs.addAll(list("CMD", "/C"));
    startArgs.addAll(list("java", "-jar", "threadfix-cli-2.2-SNAPSHOT-jar-with-dependencies.jar"));

Finally, we created an executeJarCommand(String… args) method which allows us to pass in the argument and parameters for specific CLI actions and appends it to our startArgs before setting up our ProcessBuilder command. With this in mind, our implementation of the createTeam() method looks like this:

public JSONObject createTeam(String teamName) {
    return executeJarCommand("-ct", teamName);

Pretty clean and simple.

Static Shock
We’ve already seen how our startArgs and DIRECTORY variables were statically instantiated and configured when accessing our CommandLineUtils class. Now we will see how we prepare our actual command line jar for command execution via a static block.

When using ThreadFix’s CLI jar, we have to first set the API key and ThreadFix URL. Since we only need to do this once before beginning any number of command line tests, we execute the two configuration commands when first accessing our CLI test suite file,


public class CommandLineIT extends BaseDataTest {

    private static final String API_KEY = System.getProperty("API_KEY");
    private static final String CLI_REST_URL = System.getProperty("CLI_REST_URL");
    private static CommandLineUtils cliUtils = new CommandLineUtils();

    static {
    // Rest of
    // ...
// Config Methods from
public void setApiKey(String apiKey) {
    executeJarCommand("-set", "key", apiKey);

public void setUrl(String url) {
    executeJarCommand("-set", "url", url);

The API_KEY and CLI_REST_URL values are passed in as Maven system options when Jenkins runs the regression suite in order to avoid hard-coding configuration options.

Wrapping Up
This has been a brief rundown of executing command line actions in the context of a regression suite. Other topics not covered were capturing the JSON output of the commands and verifying that the tests had had their intended effect on the ThreadFix application. We have also added the ability to selectively disable tests based on our API versioning, which may be covered in a future post.

Leave a Reply

Your email address will not be published. Required fields are marked *