Introduction

The JS7 - Performance Tests for Agents article explains the metrics and purpose of Agent performance tests.

Test cases are available for download from the JS7 - Self-Test - Getting Started article.

Test cases implement a few catch hooks to prevent users from overloading their machines.

By default the overall number of parallel jobs is limited to 16384 for a single Agent or Agent Cluster. The limit is configurable.
By default user confirmation is required for each next iteration. The confirmation is configurable.

Inventory Objects

After import of JS7 - Self-Test test cases, users find the following objects in their JOC Cockpit inventory that are related to performance tests.

Self-Test
- Controller
  - Job Resources
    - Self-Test-AuthenticationSettings
- Performance-Tests
  - Controller
    - Workflows
      - Self-Test-Performance-Cluster-Agent
      - Self-Test-Performance-Standalone-Agent
      - Self-Test-Performance-Starter-Agent
    - Resource Locks
      - Self-Test-Performance
Self-Test-User
- Automation
  - Calendars
    - Self-Test-AllDays
- Performance-Tests
  - Automation
    - Schedules
      - Self-Test-Performance-Cluster-Agent-High
      - Self-Test-Performance-Cluster-Agent-Low
      - Self-Test-Performance-Cluster-Agent-Medium
      - Self-Test-Performance-Standalone-Agent-High
      - Self-Test-Performance-Standalone-Agent-Low
      - Self-Test-Performance-Standalone-Agent-Medium

Workflows

Test Run Starter Workflow: Self-Test-Performance-Starter-Agent

The workflow implements the starter for test runs, this is the workflow to which users add a single order to run a test.

The Test Run Starter Workflow will not execute the test case itself but will add orders to one of the Test Run Workflows that implement the test run.
No changes to the workflow are required with the one exception that users should assign the jobs in the workflow a Standalone Agent or Subagent Cluster.
The Test Run Starter Workflow can be executed with any Agent on Unix or Windows platforms.

Explanations:

Agent assignment: by default all jobs in the Test Run Starter Workflow are assigned a Standalone Agent with the name: self-test-agent.
- Users can use the "Search and Replace" button in the right upper corner of the Workflow Editor. This allows to replace Agent assignments to jobs in the workflow.
- Users can add the self-test-agent alias name for an existing Standalone Agent in the Manage Controllers/Agents page.
Job Resources
- The workflow is assigned the Self-Test-AthenticationSettings job resource, for adjustments see JS7 - Self-Test - Getting Started.

Workflow Variables:

Variable	Default Value	Purpose
`workflow`	`Self-Test-Performance-Standalone-Agent`	The name of the Test Run Workflow to which orders will be added by this workflow. The following Test Run Workflows are available: `Self-Test-Performance-Standalone-Agent` `Self-Test-Performance-Cluster-Agent`
`run_count`	`1`	The job implementation of the Test Run Workflow specified by the `workflow` variable makes use of `run_count` and `sleep_count` variables: `run_count` specifies the number of times that the job script will write a line of log output and will sleep. `sleep_count` specifies the number of seconds that the job scirpt will sleeep for each `run_count`.
`sleep_count`	`30`
`confirm`	`false`	Specifies that for each next iteration that doubles the number of orders the workflow will prompt using the JS7 - Prompt Instruction and will ask the user to confirm the iteration. This is intended to prevent users from overloading their machines. The `false` value prompts users for confirmation, the `true` value will suppress prompting.
`base`	`2`	Iterations by default use `base` 2 and the power of the iteration: 2^<iteration> A `start_iteration` of 4 means 2⁴and will start 16 parallel jobs for the initial iteration. A `stop_iteration` of 7 means 2⁷and will start 128 parallel jobs for the final iteration.
`start_iteration`	`4`
`stop_iteration`	`7`
`run_id`	`Performance-Test`	Specifies an identifier for the test run, see JS7 - Performance Tests for Agents, chapter: Test Result Evaluation.
`is_unix`	`true`	Specifies if Unix or Windows Agents will be used by the job implemented of the Test Run Workflow that is specified from the `workflows` variable. Typically Test Run Workflows implement jobs for both platforms and decide based on the variable wihich one to use. The `true` value specifies that a Unix Agent will be used, the `false` value specifies use of a Windows Agent.
`max_orders`	`16384`	Specifies the overall max. number of parallel orders in a single iteration.

Test Run Workflow: Self-Test-Performance-Standalone-Agent

The workflow is one of two implementations of a Test Run Workflow that is intended for use with Standalone Agents.

The only adjustment required by users is to assign the jobs in the workflow an Agent.
The workflow offers two jobs for Unix and Windows with related job script implementations. Which job is used will be decided by the is_unix order variable that is passed by the Test Run Starter Workflow.
Variables in the workflow need no adjustment as they will be passed by the Test Run Starter Workflow.

Explanations:

Agent assignment: by default all jobs in the workflow are assigned the Standalone Agent with the name: self-test-agent.
- Users can assign the Agent that should run the test case for the related Unix or Windows job implementation.
- Users can add the self-test-agent alias name for an existing Standalone Agent in the Manage Controllers/Agents page.

Test Run Workflow: Self-Test-Performance-Cluster-Agent

The workflow is one of two implementations of a Test Run Workflow that is intended for use with Cluster Agents.

The workfow makes use of the JS7 - StickySubagent Instruction for Agent Clusters instruction to force use of the same Subagent Cluster throughout the workflow.
The only adjustment required by users is to assign the jobs in the workflow an Agent.
The workflow offers two jobs for Unix and Windows with related job script implementations. Which job is used will be decided by the is_unix order variable that is passed by the Test Run Starter Workflow.
Variables in the workflow need no adjustment as they will be passed by the Test Run Starter Workflow.

Resource Locks

Resource Lock: Self-Test-Performance

The resource lock is used to prevent parallel execution of test runs.

Calendars

Calendar: Self-Test-AllDays

The JS7 - Self-Test includes a global calendar that is used by schedules related to performance tests:

Users who run performance tests manually should consider that the calendar must exist, but is not used. The schedules provided are not configured for automated execution with the Daily Plan.
Users who wish to run performance tests on a regular basis, can copy and adjust the calendar to specify applicable days.

Schedules

A number of schedules are provided that hold reasonable parameterization per test scenario (low, medium, high).

Schedules are not configured for consideration by the JS7 - Daily Plan Service and by default will not make it for automated execution with the Daily Plan.

Users who wish to automate execution of performance tests can copy the schedules and can configure use with the Daily Plan, see JS7 - Schedules.

Schedules for use with Standalone Agents

Available schedules reference the Self-Test-Performance-Starter Test Run Starter Workflow and hold the parameterization for use with Standalone Agents.

Schedule: Self-Test-Performance-Standalone-Agent-Low

The schedule holds the parameterization for low performamce requirements which includes a maximum of 128 parallel jobs.

Schedule: Self-Test-Performance-Standalone-Agent-Medium

The schedule holds the parameterization for medium performamce requirements which includes a maximum of 512 parallel jobs.

Schedule: Self-Test-Performance-Standalone-Agent-High

The schedule holds the parameterization for high performamce requirements which includes a maximum of 2048 parallel jobs.

Schedules for use with Cluster Agents

Available schedules reference the Self-Test-Performance-Starter Test Run Starter Workflow and hold the parameterization for use with Cluster Agents.

Schedule: Self-Test-Performance-Cluster-Agent-Low

The schedule holds the parameterization for low performamce requirements which includes a maximum of 128 parallel jobs.

Schedule: Self-Test-Performance-Cluster-Agent-Medium

The schedule holds the parameterization for medium performamce requirements which includes a maximum of 512 parallel jobs.

Schedule: Self-Test-Performance-Cluster-Agent-High

The schedule holds the parameterization for high performamce requirements which includes a maximum of 2048 parallel jobs.

Starting a Test Run

To start a test run, users add an order to the Self-Test-Performance-Starter Test Run Starter Workflow like this:

This will bring forward a popup window for order parameterization like this:

In the order parameterization popup window users can click the Assign parameterization from Schedule link.
This will open a dropdown list from which users can choose a schedule. Parameterization will be copied from the selected schedule.

Having selected a schedule, related variables are displayed with the popup window for parameterization:

It is preferable to use one of the schedules provided with JS7 - Self-Test to find a matching parameterization for the intended test scenario (low, medium, high).
Users can modify variables at their will.

In most situations parallel execution of test runs should be avoided. To this purpose the Self-Test-Performance resource lock is used that will make parallel orders for the Test Run Starter Workflow wait until the lock becomes free. Users who which to run performance tests in parallel, for example to check if a Controller can cope with a number of Agents each causing a high load of jobs, can increase the resource lock's capacity to the number of parallel test runs like this:

Stopping the Test Run

You were wrongly assuming that iteration 14 from a base of 2 means 28 orders and you are surprised to find 16384 orders being added to your test run?

A few things users should not do in this situation:

Do not shutdown the JS7 products. Not the Agent, not the Controller, not JOC Cockpit.

Users can apply the following actions:

Navigate to JOC Cockpit's Workflows view and to the Self-Test-Performance-Starter-Agent Test Run Starter Workflow. Suspend the workflow from its action menu.
Similarly suspend the Test Run Workflow from its action menu.

Should JOC Cockpit become slow due to high order load, then it might be hard to use the action menu specifically. In this situation, use the Suspend All butoon on top of the page which acts as an emergency stop. The operation will freeze any orders and will prevent them from executing jobs.

When orders are frozen, they can be cancelled from the Orders Overview view that is available by clicking the number related to the order state in the Dashboard view. When cancellation of frozen orders is performed then orders will not leave the Test Run Workflow immediately. Instead the workflow has to be resumed from a similar Resume All operation. This will make orders wake up and leave the Test Run Workflow.