Introduction

The purpose of performance tests is to verify for a given hardware/software system the maximum parallelism of job executions that will not cause delays and slowness.

• The JS7 - Self-Test article offers test cases for download.
• The JS7 - Performance Tests article summarizes performance tests per product.
• The JS7 - Running Performance Tests for Agents article explains test case execution.

Test Metrics

Scenarios are based on increase of parallelism by iterations.

• Each iteration doubles the number of parallel jobs.
• White background numbers represent test parameters, blue background numbers represent typical test results that can be expected on average hardware.
  • This is not about a Unix/Windows comparison, but about differences in performance of the underlying hardware/software system.
  • The Unix machine in use was decently equipped with a number of cores and did never eat up its memory.
  • The Windows machine in use was modestly equipped and starting from a certain load exhausted its CPU, memory and I/O. 

Explanations:

• Iteration:
  • Iterations by default use base 2 and the power of the iteration: 2^<iteration>
    • An iteration of 4 means 2⁴ and will start 16 parallel jobs.
    • An  iteration of 11 means 2¹¹ and will start 2048 parallel jobs.
  • Each of the low, medium and high scenarios performs the indicated iterations.
• Job Executions: 
  • The Parallelism indicates the number of parallel jobs per iteration.
  • The Total gives the overall number of jobs executed with the iterations of the given scenario.
  • The Duration per job is longer for higher parallelism to guarantee that jobs run in parallel for some time in case of delayed job starts.
• Max. Start Time Delay
  • Numbers represent the delay between start times of parallely launched jobs per iteration and platform.
  • Larger delays indicate exhaustion of the underlying hardware/software system to start processes in parallel.
• Test Duration:
  • The duration is calculated from the earliest start time to the latest end time indicated by the JS7 - Order History.
  • The Windows OS typically requires more time to start processes as compared to Unix.

Test Runs

The JS7 - Running Performance Tests for Agents article explains test case execution.

The following chapters explain expectations and possible results per test scenario.

Low Performance Scenario

The scenario should be supported by most hardware/software systems for which the JS7 Agent is operated.

Explanations:

• For iterations 4 to 6 there is hardly any measurable difference in performance between Unix machine and Windows machine.
• For iteration 7 there is no relevant difference in job start delays. However, overall test duration is more than 50% longer on the Windows machine. This indicates that the machine is close to its limits with parallelism of 128 jobs.

During execution of performance tests, users can switch to the JS7 - Dashboard and can monitor the number of orders per state available from the Orders widget.

• The number of parallely running orders will not exceed 128.
• There can be a few waiting orders in the range of 1 digit.

Medium Performance Scenario

The scenario requires decently equipped hardware/software systems for which the JS7 Agent is operated.

Explanations:

• Iteration 8 gives approx. comparable results for start time delays of Unix and Windows machines. However, test duration is by far exceeded by the more modestly equipped Windows machine.
• Iteration 9 proves the collapse in start time delays for the Windows machine. Test duration is similarly longer by a factor of more than 10 compared to the Unix machine.
• The scenario suggests that overloading a given machine is not a good idea as it will create delays in the range of multiples of the normal duration

During execution of performance tests, users can switch to the JS7 - Dashboard and can monitor the number of orders per state available from the Orders widget.

• The number of parallely running orders will not exceed 512.
• There can be a few waiting orders in the lower range of 2 digits.
• There can be a few blocked orders in the lower range of 2 digits. Such orders are waiting for an OS process to become available when starting a job.

High Performance Scenario

The scenario requires a few CPU cores and some GB of free memory for the hardware/software system for which the JS7 Agent is operated.

Explanations:

• Iteration 10 brings acceptable numbers for the Unix machine, though the delay in start times becomes noticable. At the same time, test duration is stable.
• Iteration 11 brings noticeable delays to start times and test duration. The test proves that the Unix machine works fine for approx. 1024 parallel jobs, but not more.
• The Windows machine is not used in this scenario as test results of the previous medium scenario proved exhaustion.

During execution of performance tests, users can switch to the JS7 - Dashboard and can monitor the number of orders per state available from the Orders widget. JOC Cockpit's display of order states is a snapshot taken every 3s.

• The number of parallely running orders will not exceed 2048.
• There can be in progress orders in the range of 2 digits.
• There can be waiting orders in the range of 3 digits.
• There can be blocked orders in the range of 3 digits. Such orders are waiting for an OS process to become available when starting a job.

For this scenario users should identify delays in job execution reported by the JS7 - Order History.

Test Result Evaluation

Checking Test Results

After test execution, users should check the JS7 - Order History for failed orders as any job errors invalidate test results.

When adding orders to run performance checks, then users can specify the run_id argument. Assigning a unique identifier will simplify looking up test case results in the Order History view. For each iteration the related number will be added to the Run ID (separated by a hyphen) and both will be added to the Oder ID, for example, Performance-Test-Low1-4, Performance-Test-Low1-5, etc.

In the Order History view users can specify the Run ID for the "Filter Results" input field:

• Specifying the Run ID Performance-Test-Low1 returns results for all iterations in the test run.
• Specifying the Run ID followed by a hyphen and the iteration, for example Performance-Test-Low1-4, will filter results to the 4th iteration in the test run.
  
  

Acting in response to Test Results

Results of performance checks are intended to assist users in limiting parallelism for a given hardware/software environment at a number of levels:

• The maximum number of parallel jobs can be configured individually when registering Standalone Agents and Agent Clusters, see JS7 - Management of Standalone Agents
• The JS7 - Agent Cluster - Metrics-based Subagent Cluster can be used to limit parallelism in a Subagent Cluster.
• JS7 - Resource Locks can be used to specify limits for parallel execution of one or more jobs across workflows.
• The JS7 - Job Instruction allows to specify parallelism on a per job basis.

Resources

• JS7 - Self-Test
• JS7 - Performance Tests
• JS7 - Performance Tests for Controller
• JS7 - Performance Tests for JOC Cockpit