JS7 JobScheduler

Space shortcuts

Page tree

Introduction

The JS7 JOC Cockpit, Controller and Agents offer JS7 - Metrics for ongoing monitoring.

Metrics can be integrated with Prometheus and can be visualized from JS7 - Dashboards with Grafana.

  • Integration includes to allow Prometheus scraping telemetry data from the /metrics endpoint offered by Controller and Agents or from the /joc/metrics endpoint offered by JOC Cockpit.
  • While this suggests a straighforward proceeding, users should consider the fact that connections from a less secure network zone in which Prometheus is operated are allowed access to JS7 products. Users who regard the approach a security risk should consider using JS7 - Metrics with OpenTelemetry which allows sending telemetry data from JS7 products to an Open Telemetry server that can be configured to feed Prometheus with related data.

Users should switch to one of the offered approaches:

  • Access the single JOC Cockpit endpoint to receive metrics data for JOC Cockpit, Controller and connected Agents.
  • Access the Controller or Agent endpoint individually to receive metrics data for the related JS7 product.

Single Endpoint Configuration

FEATURE AVAILABILITY STARTING FROM RELEASE 2.9.0

JS-2250 - Getting issue details... STATUS   JOC-2103 - Getting issue details... STATUS

The approach allows reading metrics from a single endpoint for JOC Cockpit, Controller and connected Agents (preferred).

JOC Cockpit Configuration

The JOC Cockpit offers the /joc/metrics endpoint that can be used by Prometheus.

Access to the /joc/metrics endpoint is ruled by the ReadMetrics permission that can be added to the

  • JOC Cockpit <jetty-base>/resources/joc/metrics-auth.properties file.

The configuration offers HTTP Basic Authentication:

Example for ReadMetrics permission in metrics-auth.properties file
#############################################################
# Access to the /joc/metrics endpoint is ruled by the ReadMetrics role
# <username>: <password> [,<rolename> ...]
# <password> in plain text without a prefix otherwise MD5:... or OBF:...

# Prometheus: secret, ReadMetrics
Prometheus: MD5:5ebe2294ecd0e0f08eab7690d2a6ee69, ReadMetrics


Explanations:

  • Line 6,7: both make use of the Prometheus account and the ReadMetrics role
  • Line 6: is commented out. It specifies a plain text password
  • Line 7: makes use of the MD5 hashed secret password.
    • To create an MD5 hash
      • Unix users can use:
        echo -n "secret" | md5sum
      • Windows users can use PowerShell:
        [BitConverter]::ToString([System.Security.Cryptography.MD5]::Create().ComputeHash([System.Text.Encoding]::UTF8.GetBytes("secret"))).Replace("-","").ToLower()
    • There can be other commands for the purpose.

Controller Configuration

For single endpoint configuration the JOC Cockpit's Proxy Service will connect to the Controller to fetch metrics data.

Access to the Controller's /metrics endpoint is ruled by the ReadMetrics permission that can be added to the

  • Controller's <controller-data>/config/private.conf file.
  • In a Controller Cluster changes have to be applied to each Controller instance.

Example for ReadMetrics permission in Controller's private.conf configuration file
js7 {
    auth {
        users {
            # JOC Cockpit account (requires UpdateRepo permission for deployment, ReadMetrics for metrics data)
            JOC {
                distinguished-names=[
                    "DNQ=SOS CA, CN=joc-2-0-primary, OU=IT, O=SOS, L=Berlin, ST=Berlin, C=DE",
                    "DNQ=SOS CA, CN=joc-2-0-secondary, OU=IT, O=SOS, L=Berlin, ST=Berlin, C=DE"
                ]
                password="sha512:3662FD6BF84C6B8385FC15F66A137AB75C755147A81CC7AE64092BFE8A18723A7C049D459AB35C059B78FD6028BB61DCFC55801AE3894D2B52401643F17A07FE"
                permissions=[
                    UpdateItem,
                    ReadMetrics
                ]
            }
        }
    }
}


Explanations:

  • Line 5: Specifies the account name used in this example: JOC. The name is used when JOC Cockpit's Proxy Service will access the Controller's /metrics endpoint.
  • Line 13: Assigns the account the ReadMetrics permission.

Prometheus Configuration

The following entries should be added to the prometheus.yml configuration file:

Example of prometheus.yml configuration file
scrape_configs:
  - job_name: 'JS7-TST'
    scheme: http
    metrics_path: '/joc/metrics'
    params:
      deep: ['true']
     basic_auth:
      username: 'Prometheus'
      password: 'secret'
    # tls_config:
    #   ca_file: "/etc/pki/tls/certs/CA.crt"
    #   cert_file: "/etc/pki/tls/certs/prometheus.crt"
    #   key_file: "/etc/pki/tls/private/prometheus.key"
    static_configs:
      - targets:
        - js7-primary-joc:4446
        - js7-secondary-joc:4456

Explanations:

  • Line 2: Creates a job name for the JS7 scheduling environment.
  • Line 3: The example makes use of HTTP.
  • Line 4: Specifies the available endpoint for scraping
  • Line 5-6: The deep parameter is used, if JOC Cockpit's Proxy Service should report metrics for the connected Controller and Agents.
  • Line 7-9: For HTTP the account and password must be specified as configured with the JOC Cockpit's metrics-auth.properties file, see above.
  • Line 10-13: If HTTPS is used, then mutual TLS authentication applies. Prometheus holds the CA Certificate file and the Client Authentication key & certificate files.
  • Line 14-17: The targets are specified from the host and port by which JOC Cockpit is accessible to Prometheus.

Multi Endpoint Configuration

FEATURE AVAILABILITY STARTING FROM RELEASE 2.8.1

The approach allows reading metrics from individual endpoints for Controller and Agent intances.

Controller and Agent Configuration

For multi endpoint configuration the Controller and Agent offer the /metrics endpoint that can be used by Prometheus.

Access to the /metrics endpoint is ruled by the ReadMetrics permission that can be added to the

  • Controller's <controller-data>/config/private.conf file.
  • Agent's <agent-data>/config/private.conf file.

Example for ReadMetrics permission in private.conf configuration file
js7 {
    auth {
        users {
            Prometheus {
                permissions = [ ReadMetrics ]
                # distinguished-names = [
                #     "DNQ=SOS CA, CN=prometheus.sos, OU=IT, O=SOS, L=Berlin, ST=Berlin, C=DE"
                # ]
                password="plain:secret"            }
        }
    }
}


Explanations:

  • Line 4: Specifies the account name used in this example: Prometheus. The name has to be specified when Prometheus will access the /metrics endpoint.
  • Line 5: Assigns the account the ReadMetrics permission.
  • Line 9: Specifies password authentication from one of the following ways:

    • Plain text password:

      password="plain:secret"

    • Hashed password:

      password="sha512:bd2b1aaf7ef4f09be9f52ce2d8d599674d81aa9d6a4421696dc4d93dd0619d682ce56b4d64a9ef097761ced99e0f67265b5f76085e5b0ee7ca4696b2ad6fe2b2"

      The hash can be created from the command like like this:

       echo -n "secret" | openssl dgst -sha512
  • Line 6 to 8: Alternatively to password authentication users can apply mutual HTTP authentication. The Distinguished Name of one or more Client Authentication Certificates used by Prometheus instances is specified:
    • Except for whitespace between attributes the precise sequence and values as available from the certificate's subject have to match the property value.
    • Note that the common name (CN) element in the distinguished name has to match the fully qualified domain name (FQDN) of the Prometheus instance's host.

    • The following command can be used to read the distinguished name from a certificate file:

      Example for OpenSSL command to read a certificate's distinguished name
      # read distinguished name from the a Prometheus instance's certificate file (.crt)
openssl x509 -in prometheus.crt -noout -nameopt RFC2253 -subject

# output is returned with a prefix "subject= " or similar that is not part of the distinguished name
# subject= DNQ=SOS CA,CN=prometheus.sos,OU=IT,O=SOS,L=Berlin,ST=Berlin,C=DE

Prometheus Configuration

The following entries should be added to the prometheus.yml configuration file:

Example of prometheus.yml configuration file
scrape_configs:
  - job_name: 'JS7-TST'
    scheme: http
    basic_auth:
      username: 'Prometheus'
      password: 'secret'
    # tls_config:
    #   ca_file: "/var/opt/prometheus/private/CA.crt"
    #   cert_file: "/var/opt/prometheus/private/prometheus.crt"
    #   key_file: "/var/opt/prometheus/private/prometheus.key"
    static_configs:
      - targets:
        - js7-tst-primary-controller:5444
        - js7-tst-secondary-controller:5544
        - js7-tst-primary-agent:6445
        - js7-tst-secondary-agent:6545
    relabel_configs:
      - source_labels: [__address__]
        # Provide $host with the hostname only (strip "js7-" and port number)
        regex: '(js7-)?(.*):.*'
        target_label: host
        replacement: '$2'


Explanations:

  • Line 2: Creates a job name for the JS7 scheduling environment.
  • Line 3: The example makes use of HTTP.
  • Line 4-6: For HTTP the account and password must be specified as configured with the Controller's/Agent's private.conf file, see above.
  • Line 7-10: If HTTPS is used, then mutual TLS authentication applies. Prometheus holds the CA Certificate file and the Client Authentication key & certificate files.
  • Line 11-16: The targets are specified from the host and port by which the related Controller and Agent are accessible to Prometheus.
  • Line 17-22: Optionally the host target label is reconfigured to hold the hostname from the target without js7- prefix and without port specification. This simplifies use of the host attribute with Grafana dashboards.

Resources


  • No labels