COPT Compute Cluster Service

The Cardinal Optimizer provides COPT compute cluster service on all supported platforms, which allows you to offload optimization computations from COPT client applications over local network.

Once COPT compute cluster server runs at local network, any COPT client application with matching COPT version can connect to server and offload optimization computations. That is, COPT compute cluster clients are allowed to do modeling locally, execute optimization jobs remotely, and then obtain results interactively.

Note that the more computing power server has, the more optimization jobs can run simultaneously. Furthermore, COPT compute cluster service has functionality to cluster multiple servers together and therefore serve more COPT compute cluster clients over local network.

Server Setup

The COPT compute cluster service includes copt_cluster executable and a configuration file cls.ini. The very first thing to do when cluster server starts is to verify cluster license locally, whose path is specified in cls.ini. If local validation passes, cluster server might connect remotely to COPT licensing server for further validation, such as verifying machine IP, which is supposed to match IP range that user provided during registration. This means the server running COPT compute cluster service should have internet access in specified area, such as campus network. For details, please see descriptions below or refer to How to obtain and setup license.

Installation

The Cardinal Optimizer provides a separate package for remote services, which include COPT compute cluster. Users may apply for remote package from customer service. Afterwards, unzip the remote package and move to any folder on your computer. The software is portable and does not change anything in the system it runs on. Below are details of installation.

Windows

Please unzip the remote package and move to any folder. Though, it is common to move to folder under C:\Program Files.

Linux

To unzip the remote package, enter the following command in terminal:

tar -xzf CardinalOptimizer-Remote-8.0.5-lnx64.tar.gz

Then, the following command moves folder copt_remote80 in current directory to other path. For an example, admin user may move it to folder under /opt and standard user may move it to $HOME.

sudo mv copt_remote80 /opt

Note that it requires root privilege to execute this command.

MacOS

To unzip the remote package, enter the following command in terminal:

tar -xzf CardinalOptimizer-Remote-8.0.5-universal_mac.tar.gz

Then, the following command moves folder copt_remote80 in current directory to other path. For an example, admin user may move it to folder under /Applications and standard user may move it to $HOME.

mv copt_remote80 /Applications

If you see errors below or similar signature problem of COPT lib during installation,

"libcopt.dylib" cannot be opened because the developer cannot be verified.
macOS cannot verify that this app is free from malware.

run the following command as root user, to bypass check of loading dynamic lib on MacOS.

xattr -d com.apple.quarantine CardinalOptimizer-Remote-8.0.5-universal_mac.tar.gz

or

xattr -dr com.apple.quarantine /Applications/copt_remote80

Cluster License

After installing COPT remote package, it requires cluster license to run. It is preferred to save cluster license files, license.dat and license.key, to cluster folder in path of remote package.

The following explains how to obtain the license file via the copt_licgen tool and the license credential information key under different systems.

Note

If the user has already obtained the two license files license.dat and license.key, there is no need to obtain them again. You can skip the following steps to obtain the license file and refer to Configuration directly.

Windows

If the COPT remote package is installed under "C:\Program Files", execute the following command to enter cluster folder in path of remote package.

cd "C:\Program Files\copt_remote80\cluster"

Note that the tool copt_licgen creating license files exists under tools folder in path of remote package. The following command creates cluster license files in current directory, given a cluster license key, such as 7483dff0863ffdae9fff697d3573e8bc .

..\tools\copt_licgen -key 7483dff0863ffdae9fff697d3573e8bc

Linux and MacOS

If the COPT remote package is installed under "/Applications", execute the following command to enter cluster folder in path of remote package on MacOS system.

cd /Applications/copt_remote80/cluster

The following command creates cluster license files in current directory, given a cluster license key, such as 7483dff0863ffdae9fff697d3573e8bc .

../tools/copt_licgen -key 7483dff0863ffdae9fff697d3573e8bc

In addition, if users run the above command when current directory is different than cluster folder in path of remote package, it is preferred to move them to cluster. The following command does so.

mv license.* /Application/copt_remote80/cluster

Configuration

Below is a typical configuration file, cls.ini, of COPT compute cluster.

[Main]
Port = 7878
# number of total tokens, which copt jobs can run simultaneously up to
NumToken = 3
# password is case-sensitive and default is empty
# it applies to both copt clients and cluster nodes
PassWd =
# data folder of cluster relative to its binary folder,
# where various of copt libraries and temporary job files reside.
DataFolder = ./data

[Memory]
# Example memory management configuration
# memory watermark defense
MemGuard = true
MemSoft = 0.80
MemHard = 0.92
MemFatal = 0.97
MemEvictPolicy = largest
# automatic per-task memory cap
MemLimitAuto = true
MemUsableRatio = 0.90
MemInflation = 1.35
MemOvercommit = 1.50
# 0 means detecting physical memory from the system
MemPhysicalMB = 0

[SSL]
# needed if connecting using SSL
CaFile =
CertFile =
CertkeyFile =

[Licensing]
# if empty or default license name, it is from binary folder
# to get license files from cwd, add prefix "./"
# full path is supported as well
LicenseFile = license.dat
PubkeyFile = license.key

[WLS]
# WebServer have a default host and no need to edit in most scenarios
# Must specify WebLicenseId and WebAccesskey to trigger web licensing
# If specified, ignore settings in Licensing section
WebServer =
WebLicenseId =
WebAccessKey =
WebTokenDuration = 300

[FLS]
# Lease token from floating server
# If specified, ignore settings in Licensing section and WLS section
TokenServer =
TPort = 7979

[Cluster]
# host name and port of parent node in cluster
# If specified or changed on fly, connect to parent node
Parent =
PPort = 7878

[Filter]
# default policy 0 indicates accepting all connections, except for ones in blacklist
# otherwise, denying all connections except for ones in whitelist
DefaultPolicy = 0
UseBlackList = true
UseWhiteList = true
FilterListFile = clsfilters.ini

[Logs]
LogsFolder = ./logs

The Main section specifies port number, through which COPT compute cluster clients connect to server; token number, the number of optimization jobs that server can run simultaneously up to; password string, if specified, cluster clients should send the same password when requesting for service.

Main:

In the [Main] section of the configuration file, users can set the following connection information:

Port : Specifies the connection port for the cluster server, through which COPT compute cluster clients connect to server;
NumToken : Defines the number of optimization jobs that the COPT cluster server can run simultaneously up to. This number can be estimated based on the server’s hardware resource capacity and is user-configurable. The default value is 3, with no concurrency limit.
PassWd : Sets the password for client connections to the cluster server. The default is empty, meaning no password is required. If specified, clients should send the same password when requesting for cluster service. For a more secure connection, users can enable encrypted communication using RSA certificates in the [SSL] section.
DataFolder : Specifies the path where the COPT solver is installed on the server.

The COPT compute cluster may install multiple versions of COPT to subfolder of DataFolder. Only clients with matching version (major and minor) will get approved and then offload optimization jobs at server side. Note that the COPT compute cluster pre-installs a COPT solver of the same version as server itself, which illustrate how to install other versions of COPT.

For instance, the COPT compute cluster has default COPT v8.0.5 installed and users plan to install COPT of other version v6.0.7. Users may create a folder ./data/copt/6.0.7/ and copy a COPT C lib of the same version to it. Specifically, on Linux platform, copy C dynamic library libcopt.so from the binary folder $COPT_HOME/lib/ of COPT v6.0.7 to subfolder ./data/copt/6.0.7/ of the COPT compute cluster.

Furthermore, users are allowed to install newer version of COPT than cluster server version, such as COPT v9.0.1. To do so, follow the same step of copying a C lib of COPT v9.0.1 to ./data/copt/9.0.1/. In addition, users need a personal license of v9.0.1 to load C lib of COPT v9.0.1 at server side. That is, copy valid personal license files to folder ./data/copt/9.0.1/ as well. However, this simple procedure may break if the newer COPT solver has significant updates. In this case, it is necessary to upgrade the COPT compute cluster to newer version, that is, v9.0.1.

Below is an example of directory structure of cluster server on Linux platform. It includes pre-installed COPT v8.0.5, COPT of previous version v6.0.7, and COPT of newer version v9.0.1.

~/copt_remote80/cluster
│  cls.ini
│  copt_cluster
│  license.dat -> cluster license v8.0.5
│  license.key
│
└─data
    └─copt
        └─6.0.7
        │   libcopt.so
        └─8.0.5
        │   libcopt.so
        └─9.0.1
            libcopt.so
            license.dat -> license v9.0.1
            license.key

Memory:

In the [Memory] section of the configuration file, users can configure the per-task memory cap and the memory watermark defense of the COPT compute cluster server.

Note

The values shown in the [Memory] section above are examples only, and do not represent the default values of the program.

Per-task memory cap:

MemLimitAuto: Whether to enable the automatic per-task memory cap. The default is true. This parameter is the master switch of the automatic per-task memory cap: when set to true, the server computes and applies a per-task memory cap at the start of every solve; when set to false, the server neither computes nor applies that cap, and solving tasks are subject only to the MemLimit set by the client itself and to the memory watermark defense. In addition, when the number of tokens is 0, the server does not apply a per-task memory cap;
MemUsableRatio: The proportion of physical memory available to solving tasks;
MemInflation: The conversion factor between the memory usage reported by the solver and the actual resident memory usage;
MemOvercommit: How far the memory available to a single task may exceed its average memory share;
MemPhysicalMB: The physical memory size of the server, in MB. When set to 0, the server detects it automatically; in a memory-limited container, the cgroup memory limit is used.

Memory watermark defense:

MemGuard: Whether to enable the memory watermark defense;
MemSoft: The soft watermark. On reaching it, the server stops accepting new tasks;
MemHard: The hard watermark. On reaching it, the server selects one solving task and interrupts it;
MemFatal: The fatal watermark. On reaching it, the server interrupts all solving tasks;
MemEvictPolicy: The policy for selecting the task to interrupt when the hard watermark is reached. The following values are supported:
- largest: Select the task with the largest current memory usage. This is the default policy;
- longest: Select the task that has been solving the longest;
- youngest: Select the task whose solve started most recently;
- random: Select a solving task at random.

MemSoft, MemHard and MemFatal all denote the proportion of used physical memory to total physical memory, ranging from 0.0 to 1.0.

For the computation of the per-task memory cap, the watermark triggering mechanism, memory cap exemptions, compatibility with older COPT versions and deployment suggestions, please refer to Server Memory Management.

Licensing:

The Licensing section specifies location of cluster license files. As described in comments above, if empty string or default license file name ( license.dat or license.key ) is specified, cluster license files are read from the binary folder where the cluster executive reside.

It is possible to run COPT compute cluster service, even if cluster license files do not exist in the same folder as the cluster executive. One option is to set LicenseFile = ./license.dat and PubkeyFile = ./license.key. By doing so, the COPT compute cluster reads cluster license files from the current working directory. That is, user could execute command at the path where cluster license files exist to run service.

The other option is to set full path of license files in configuration. As mentioned before, Cardinal Optimizer allows user to set environment variable COPT_LICENSE_DIR for license files. For details, please refer to How to install Cardinal Optimizer. If user prefers the way of environment variable, cls.ini should have the full path to cluster license files.

Cluster:

In the [Cluster] section of the configuration file, users can set the parent node (IP and port) for the current cluster server connection. By default, this is empty, indicating no connection to other nodes.

Parent: The IP address of the parent node within the local network.
PPort: The port number of the parent node.

If there is only one cluster server or if the current server is the root node, this section does not need to be configured.

For multiple cluster servers, specifying a parent node allows users to form a tree-structured network topology for the COPT computing cluster. The current server can join the parent node’s cluster through this configuration.

COPT supports cluster load balancing in multi-task computing scenarios, enabling computing tasks to be reasonably distributed across servers and improving the utilization of computing resources.

Filter:

In the [Filter] section of the configuration file, users can configure the filtering policy for the cluster server.

DefaultPolicy: Default is set to 0, meaning all clients are allowed to connect to the cluster server, except those on the blacklist. If it is set to non-zero value, then all connection are blocked except for those in white lists.
UseBlackList: If set to True, the blacklist will be enabled.
UseWhiteList: If set to True, the whitelist will be enabled.
FilterListFile: Specifies the name of the filtering configuration file for the cluster server, default is clsfilters.ini. Below is an example of the filter configuration file:

[BlackList]
# 127.0.*.* + user@machine*

[WhiteList]
# 127.0.1.2/16 - user@machine*

[ToolList]
# only tool client at server side can access by default
127.0.0.1/32

[MemWhiteList]
# clients exempt from the automatic per-task memory cap
# 192.168.1.*
# 10.0.0.0/16+batch@prod-*

It has four sections and each section has its own rules. In section of BlackList, one may add rules to block others from connection. In section of WhiteList, one may add rules to grant others for connection, even if the default policy is to block all connections. Only users listed in section of ToolList are able to connect to cluster server by Cluster Managing Tool (see below for details). Section MemWhiteList holds the per-task memory cap exemption rules. A client matching any rule in it is not subject to the memory cap computed automatically by the server, but is still subject to the MemLimit it sets itself and to the memory watermark defense. This section never affects whether a client may connect to the cluster server, and it is always active regardless of the connection filtering policy.

Specifically, rules in filter configuration have format of starting with IP address. To specify IP range, users may include wildcard (*) in IP address, or use CIDR notation, that is, a IPv4 address and its associated network prefix. In addition, a rule may include (+) or exclude (-) given user at given machine, such as 127.0.1.2/16 - user@machine. Here, user refers to username, which can be queried by whoami on Linux/MacOS platform; machine refers to computer name, which can be queried by hostname on Linux/MacOS platform.

Note that after modifying the configuration file clsfilters.ini , users can use the ResetFilters command to reset the current rules to those in the filter configuration document. Users can also use the WriteFilters command to output the current rules to the filter configuration document.

Logs:

In the [Logs] section of the configuration file, users can set the log file path for the cluster server, which is by default stored in the ./logs path under the cluster server installation directory.

Web License for Compute Cluster

Besides local cluster license above, users may use web license for compute cluster to run compute cluster service. This requires that machines running compute cluster server must have internet access. However, hardware info are not required any more. That is, users are free to deploy compute cluster servers to any cloud machine or container, as long as they have internet access. Please refer to COPT web license page for details.

Below are brief steps:

Follow steps to register an account and apply for trial of web license for compute cluster.

Once approved, Web License ID is generated for users

On page of API Keys , create Web Access Key using given Web License ID

Afterwards, users edit configuration file cls.ini and add values of both Web License ID and Web Access Key to related keywords in section of WLS . For instance,

[WLS]
# WebServer have a default host and no need to edit in most scenarios
# Must specify WebLicenseId and WebAccesskey to trigger web licensing
WebServer =
WebLicenseId =
WebAccessKey =
WebTokenDuration = 300

As of now, compute cluster server talks to COPT web license page for licensing. Users are able to monitor its token usage and other information online.

Example Usage

Suppose that cluster license files exist in the same folder where the cluster executable reside. To start the COPT compute cluster, just execute the following command at any directory in Windows console, or Linux/Mac terminal.

./copt_cluster

If you see log information as follows, the COPT compute cluster has been successfully started. Server monitors any connection from COPT compute cluster clients, manages client requests in queue as well as approved clients. User may stop cluster server anytime when entering q or Q.

> ./copt_cluster
  [ Info] start COPT Compute Cluster, COPT v8.0.5 20240304
  [ Info] [NODE] node has been initialized
  [ Info] server started at port 7878

If failed to verify local cluster license, or something is wrong on remote COPT licensing server, you might see error logs as follows.

> ./copt_cluster
  [ Info] start COPT Compute Cluster, COPT v8.0.5 20240304
  [Error] Invalid signature in public key file
  [Error] Fail to verify local license

and

> ./copt_cluster
  [ Info] start COPT Compute Cluster, COPT v8.0.5 20240304
  [Error] Error to connect license server
  [Error] Fail to verify cluster license by server

Server Memory Management

The COPT compute cluster server manages memory through two mechanisms: the per-task memory cap and the memory watermark defense. The per-task memory cap controls the memory usage of an individual solving task; the memory watermark defense monitors the overall physical memory usage of the server and takes action when a configured watermark is reached.

Per-task memory cap

Once MemLimitAuto is enabled, the server computes the per-task memory cap at the start of every solve, based on the physical memory of the server, the number of sessions currently holding a token, and the related settings. The cap stays unchanged for the duration of that solve, and is recomputed at the start of the next one.

The server computes the per-task memory cap according to the following formulas:

Budget = PhysicalMemory * MemUsableRatio / MemInflation
Share  = Budget / SessionsHoldingToken
Cap    = min(MemOvercommit * Share, Budget / 2)

Where:

PhysicalMemory denotes the physical memory of the server;
SessionsHoldingToken denotes the number of sessions currently holding a token;
Budget denotes the memory budget available to solving tasks;
Share denotes the average memory share per session holding a token;
Cap denotes the per-task memory cap computed automatically by the server.

When only one session holds a token, a newly started solving task may receive at most half of the memory budget as its cap. As the number of sessions holding a token increases, newly started solving tasks receive smaller caps.

Clients still set MemLimit through the normal COPT parameter interface; no dedicated cluster client interface is involved. If the client has also set MemLimit, the limit actually applied during a solve is the smaller of the client-set value and the value computed automatically by the server. A client-set MemLimit = 0 denotes a strict zero memory limit.

After the solve finishes, the server restores the MemLimit originally set by the client. At the start of the next solve, the server recomputes the per-task memory cap.

MemInflation converts between the memory usage reported by the solver itself and the actual resident memory usage observed by the operating system. For example, setting it to 1.35 means that for every 1 byte of memory the solver accounts for, about 1.35 bytes of actual resident memory are expected. This value only illustrates the meaning of the parameter, and does not represent the default value of the program.

Memory watermark defense

Once MemGuard is enabled, the server checks the system physical memory usage roughly every 500 milliseconds, and takes the following actions according to the current memory usage:

On reaching the soft watermark MemSoft, the server stops accepting new tasks, without affecting the tasks already running. When memory usage falls below MemSoft - 0.03, the server resumes accepting new tasks;
On reaching the hard watermark MemHard, the server selects one solving task according to MemEvictPolicy and interrupts it. After interrupting a task, the server waits at least 5 seconds and waits for that task to actually stop. If memory usage is still falling, the server does not immediately interrupt the next task;
On reaching the fatal watermark MemFatal, the server interrupts all solving tasks, and keeps checking until memory usage falls below that watermark.

The server interrupts a task by calling COPT_Interrupt, and the solve stops at its next interruptible point. If the solver has already obtained a current best feasible solution, the client can still retrieve that solution together with the corresponding interrupted status. The session the task belongs to stays connected, and may start another solve afterwards.

Only tasks that are actually solving are eligible for selection. An idle session is not selected even though it still holds its model, because calling COPT_Interrupt on an idle session does not release the memory of its model.

Memory cap exemptions

Users can set per-task memory cap exemption rules in the [MemWhiteList] section of the filter configuration file clsfilters.ini. A client matching any rule is not subject to the per-task memory cap computed automatically by the server, but is still subject to the MemLimit it sets itself and to the memory watermark defense.

[MemWhiteList] uses the same IP + user@machine rule format as the blacklist and the whitelist. This section never affects whether a client may connect to the cluster server, and it is always active regardless of the connection filtering policy. Restricting exemption rules to specific IP addresses or subnets wherever possible is recommended.

For example, the following rules may be added to clsfilters.ini:

[MemWhiteList]
192.168.1.*
10.0.0.0/16+batch@prod-*
*.*.*.*+alice@*

In the example above:

192.168.1.*: exempts clients from that IP address range;
10.0.0.0/16+batch@prod-*: exempts clients from that subnet whose user name is batch and whose machine name matches prod-*;
*.*.*.*+alice@*: exempts clients whose user name is alice, with no restriction on IP address or machine name.

Multiple rules are independent of each other. A client is exempt from the per-task memory cap as soon as it matches any one of them. For a rule that contains an IP address, a user name and a machine name at the same time, the client has to match every part of that rule.

Compatibility with older COPT versions

The cluster server checks whether the relevant memory management features are available, according to the COPT version each task actually loads:

The soft and fatal watermarks do not depend on a specific COPT version;
The longest, youngest and random policies do not depend on a specific COPT version;
The largest policy requires the COPT version loaded by the task to support COPT_GetMemUsage. If it is not supported, no task can be selected by that policy when the hard watermark is reached, but the fatal watermark remains effective;
The automatic per-task memory cap requires the COPT version loaded by the task to support the MemLimit parameter. If it is not supported, that task is not subject to the memory cap computed automatically by the server;
If the COPT version loaded by the task does not support retrieving the current memory usage, the peak memory usage or the memory limit, GetJobs displays - in the corresponding column.

When the cluster server has to load an older COPT version that does not support COPT_GetMemUsage, MemEvictPolicy can be set to longest, youngest or random.

Deployment suggestions

The following configuration can be used to further reduce the risk of exhausting the memory of the server. The values in it are for reference only, and do not represent the default values of the program:

[Memory]
MemGuard = true
MemSoft = 0.70
MemHard = 0.80
MemFatal = 0.90
MemLimitAuto = true
MemUsableRatio = 0.75
MemInflation = 1.50
MemOvercommit = 1.00

Adjusting MemUsableRatio, MemInflation and MemOvercommit controls the per-task memory cap computed automatically, and reserves room for the other memory overheads of the server. The specific values should be determined according to the resources of the server and the memory usage of the actual tasks.

With MemOvercommit = 1.00, the memory caps of all tasks together do not exceed the proportion of physical memory given by MemUsableRatio / MemInflation, even at full concurrency; 50% of physical memory in the example above.

If the maximum memory usage of the cluster service needs to be further restricted by the operating system, cgroup can be used, or a memory limit below the total physical memory can be set for the service through the MemoryMax= option of systemd.

Client Setup

The COPT compute cluster client can be COPT command-line, or any application which solves problems by COPT API, such as COPT cpp/java/csharp/python interface. The COPT compute cluster service is a better approach in terms of flexibility and efficiency. Any COPT compute cluster client can legally run Cardinal Optimizer without local license.

Configuration

Via the configuration file

Before running COPT as cluster client, please make sure that you have installed COPT locally. For details, please refer to How to install Cardinal Optimizer. Users can skip obtaining local licenses by adding a cluster configuration file client.ini .

Below is a typical configuration file, client.ini, of COPT compute cluster client.

Cluster = 192.168.1.11
Port = 7878
WaitTime = 600
Passwd =

As configured above, COPT compute cluster client tries to connect to 192.168.1.11 at port 7878 with waiting time in queue up to 600 seconds. Here, the default value of Cluster is localhost. WaitTime (or QueueTime ) is set to 0 if empty or not specified. Specifically, empty WaitTime means client does not wait and should quit immediately, if the COPT compute cluster have no more token available. Port default number is 7878. It must be great than zero if specified and should be the same as that specified in cluster configuration file cls.ini. Note that keywords in the configuration file are case insensitive.

In addition, users can set the password for connecting the remote server through Passwd. The Priority can be used to set the priority of cluster optimization jobs. Possible values range from 0 to 99, with higher values indicating higher priority.If jobs are queued, the priority setting will ensure that the next task is processed first, but it will not affect tasks that are already running.

To run as a COPT compute cluster client, an application must have configuration file, client.ini, in one of the following three locations, that is, current working directory, environment directory by COPT_LICENSE_DIR and binary directory where COPT executable resides.

By design, COPT application reads local license files instead of client.ini, if they both exist in the same location. However, if local license files are under the environment directory, to connect to cluster server, user could simply add a configuration file, client.ini, under the current working directory (different from the environment directory).

If a COPT application calls COPT API to solve problems, such as COPT python interface, license is checked as soon as COPT environment object is created. If there only exists proper configuration file, client.ini, the application works as a COPT compute cluster client and obtains token to offload optimization jobs. As soon as COPT environment object is destroyed, the COPT compute cluster server is notified to release token and thus to approve more requests waiting in queue.

Via API Functions

In addition to the method using the client.ini file mentioned above, users can also configure the client in their code through API functions. Taking the COPT Python interface as an example, the corresponding class is EnvrConfig Class, and similar approaches apply to other programming languages. The configuration is as follows:

# Set client configuration parameters
envconfig.set(COPT.CLIENT_CLUSTER, "192.168.1.11")
envconfig.set(COPT.CLIENT_PORT, "7878")
envconfig.set(COPT.CLIENT_WAITTIME, "600")

High Availability

When there are multiple cluster servers, the client can achieve high availability by configuring multiple Cluster server IP addresses in the Cluster field of client.ini, as shown in the following table:

Cluster = 192.168.1.11; 192.168.1.22; 192.168.1.33
Port = 7878
WaitTime = 600
Passwd =

According to the above configuration, the first IP in the Cluster field is the root node IP of the cluster. If the port of other child nodes of the cluster is not 7878, it could be added after the IP address. The above configuration file indicates that the client will first try to connect to the root node 192.168.1.11:7878 . If the root node is not available, it will try to connect to the child node of 192.168.1.22:7878 , and so on, until the connection with the cluster server is successful.

If a cluster node fails when performing a computing task, the client will reallocate the available cluster nodes according to the backup nodes in the configuration file. This can enable the cluster service to have a certain disaster recovery capability, thereby improving the high availability of the COPT computing cluster service.

Example Usage

Suppose that we’ve set configuration file client.ini properly and have no local license, below is an example of connecting to cluster server by COPT command-line tool copt_cmd. Execute the following command in Windows console, or Linux/Mac terminal.

copt_cmd

If you see log information as follows, the COPT compute cluster client, copt_cmd, has connected to cluster server successfully. COPT command-line tool is ready to do modeling locally and then offload optimization jobs to server.

> copt_cmd
  Cardinal Optimizer v8.0.5. Build date Mar 04 2024
  Copyright Cardinal Operations 2025. All Rights Reserved

  [ Info] initialize cluster client with ./client.ini

  [ Info] wait for server in 0 / 39 secs
  [ Info] connecting to cluster server 192.168.1.11:7878
COPT>

If you see log information as follows, the COPT compute cluster client, copt_cmd, has connected to cluster server. However, due to limited number of tokens, it waits in queue of size 5, until timeout.

> copt_cmd
  Cardinal Optimizer v8.0.5. Build date Mar 04 2024
  Copyright Cardinal Operations 2025. All Rights Reserved

  [ Info] initialize cluster client with ./client.ini

  [ Info] wait for server in 0 / 39 secs
  [ Info] connecting to cluster server 192.168.1.11:7878

  [ Warn] wait in queue of size 5
  [ Info] wait for license in  2 / 39 secs
  [ Info] wait for license in  4 / 39 secs
  [ Info] wait for license in  6 / 39 secs
  [ Info] wait for license in  8 / 39 secs
  [ Info] wait for license in 10 / 39 secs
  [ Info] wait for license in 20 / 39 secs
  [ Info] wait for license in 30 / 39 secs
  [Error] timeout at waiting for server approval
  [Error] Fail to initialize copt command-line tool

If you see log information as follows, the COPT compute cluster client, copt_cmd, has connected to cluster server. But it refused to wait in queue, as Queuetime was set to 0. Therefore, client quits with error immediately.

> copt_cmd
  Cardinal Optimizer v8.0.5. Build date Mar 04 2024
  Copyright Cardinal Operations 2025. All Rights Reserved

  [ Info] initialize cluster client with ./client.ini

  [ Info] wait for server in 0 / 9 secs
  [ Info] connecting to cluster server 192.168.1.11:7878
  [ Warn] server error: "no more token available", code = 129
  [Error] Fail to initialize copt command-line tool

If you see log information as follows, the COPT compute cluster client, copt_cmd, fails to connect to cluster server. Finally, client quits after timeout.

> copt_cmd
  Cardinal Optimizer v8.0.5. Build date Mar 04 2024
  Copyright Cardinal Operations 2025. All Rights Reserved

  [ Info] initialize cluster client with ./client.ini

  [ Info] wait for server in 0 / 39 secs
  [ Info] connecting to cluster server 192.168.1.11:7878
  [ Info] wait for license in  2 / 39 secs
  [ Info] wait for license in  4 / 39 secs
  [ Info] wait for license in  6 / 39 secs
  [ Info] wait for license in  8 / 39 secs
  [ Info] wait for license in 10 / 39 secs
  [ Info] wait for license in 20 / 39 secs
  [ Info] wait for license in 30 / 39 secs
  [Error] timeout at waiting for server approval
  [Error] Fail to initialize copt command-line tool

In addition, users can start the COPT command-line tool on the client side to connect to the specified cluster server IP address as follows:

> copt_cmd -cluster <ip>

COPT Cluster Managing Tool

COPT cluster service ships with a tool copt_clstool, for retrieving information and tune parameters of cluster servers on fly.

Tool Usage

Execute the following command in Windows console, Linux or MacOS terminal:

> ./copt_clstool

Below displays help messages of this tool:

> ./copt_clstool
  COPT Cluster Managing Tool

  copt_clstool [-s server ip] [-p port] [-x passwd] command <param>

  commands are:   addblackrule <127.0.0.1/20[-user@machine]>
                  addwhiterule <127.0.*.*[+user@machine]>
                  addmemexempt <127.0.*.*[+user@machine]>
                  delmemexempt <127.0.*.*[+user@machine]>
                  getfilters
                  getinfo
                  getnodes
                  getjobs
                  interrupt
                  reload
                  resetfilters
                  setparent <xxx:7878>
                  setpasswd <xxx>
                  settoken  <num>
                  setmemguard <on|off>
                  setmemlimitauto <on|off>
                  setmemevictpolicy <largest|longest|youngest|random>
                  toggleblackrule <n-th>
                  togglewhiterule <n-th>
                  writefilters

If the -s and -p option are present, tool connects to cluster server with given server IP and port. Otherwise, tool connections to localhost and default port 7878. If cluster server sets a password, tool must provide password string after the -x option.

This tool provides the following commands:

AddBlackRule: Add a new rule in black filters. each rule has format starting with non-empty IP address, which may have wildcard to match IPs in the scope. In addition, it is optional to be followed by including (+) or excluding (-) user name at machine name.
AddWhiteRule: Add a new rule in white filters. Note that a white rule has the same format as a black rule.
AddMemExempt: Add a per-task memory cap exemption rule. A client matching the rule is not subject to the per-task memory cap computed automatically by the server, but is still subject to the MemLimit it sets itself and to the memory watermark defense. This command modifies the rules in memory and takes effect immediately.
DelMemExempt: Delete the given per-task memory cap exemption rule. This command modifies the rules in memory and takes effect immediately.
GetFilters: Get the current black filters, white filters, tool filters and per-task memory cap exemption rules. Each rule in the black and white filters carries a sequence number, which is the parameter for command ToggleBlackRule and ToggleWhiteRule respectively.
GetInfo: Get general information of cluster server, including token usage, connected clients, all COPT versions in support, along with the state of the memory watermark defense, the memory cap a next solving task is expected to receive, and the clients currently exempt from the memory cap.
GetNodes: Get information of nodes in cluster, including parent address and status, all children nodes.
Reload: Reload available token information of all child nodes, in case it is not consistent for various reasons.
GetJobs: Retrieves the information of the tasks on the current server, including the task ID (TID), runtime (in seconds), client ID, current memory usage, peak memory usage and memory limit. The memory related information is given in MB. If the COPT version loaded by the task does not support retrieving the corresponding memory information, - is displayed in that column.
Interrupt: Terminates the specified task (TID) on the current server (cluster server). After executing this command, the task running on the client will be stopped, and the optimization status will return stopped (user interrupt).
ResetFilters: Reset filter lists in memory to those on filter config file.
SetParent: Change parent node address on fly and then connecting to new parent. In this way, it avoids draining operation when stopping a node for maintenance purpose.
SetPasswd: Update password of target cluster server on fly.
SetToken: Change token number of target cluster server on fly.
SetMemGuard: Enable or disable the memory watermark defense. The available values are on and off.
SetMemLimitAuto: Enable or disable the automatic per-task memory cap. The available values are on and off. The modified setting applies to the solving tasks started afterwards.
SetMemEvictPolicy: Set the policy for selecting the task to interrupt when the hard watermark is reached. The available values are largest, longest, youngest and random.
ToggleBlackRule: Toggle between enabling and disabling a black rule, given its sequence number by GetFilters.
ToggleWhiteRule: Toggle between enabling and disabling a white rule, given its sequence number by GetFilters.
WriteFilters: Write all filter rules in memory to the filter config file, including the black filters, white filters, tool filters and per-task memory cap exemption rules.

Changes made to the memory management settings through SetMemGuard, SetMemLimitAuto and SetMemEvictPolicy only apply while the current server is running, and are not written back to cls.ini. After the server restarts, the settings in cls.ini are used again.

Example Usage

The following command lists general information on local machine.

> ./copt_clstool GetInfo

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to localhost:7878
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] general info
  # of available tokens is 3 / 3, queue size is 0
  # of active clients is 0
  # of installed COPT versions is 1
    COPT v8.0.5  Physical Memory: 71.8% used, Defense Status: [OK] (soft 80% / hard 92% / fatal 97%)
  Per-task Memory Cap: 5461 MB (by demand with N=3)
  Memory exempt clients: 127.0.0.1, 127.0.0.*

The following command lists the tasks on the server together with their memory information:

./copt_clstool -s <ip> -p <port> getjobs

The output contains the following fields:

ThreadID  Duration(s)  MemUsage(MB)  MemPeak(MB)  MemLimit(MB)  ClientID

Where:

ThreadID: the task ID;
Duration(s): the runtime of the task, in seconds;
MemUsage(MB): the current memory usage, in MB;
MemPeak(MB): the peak memory usage, in MB;
MemLimit(MB): the memory limit currently set on the problem, in MB. During a solve this is the limit actually enforced; otherwise it is the value set by the client itself;
ClientID: the client ID.

MemUsage and MemPeak both come from the memory accounting of the solver itself, and may therefore differ from the actual resident memory usage observed by the operating system. For the conversion between the two, refer to MemInflation.

If the COPT version loaded by the task does not support retrieving one of these pieces of memory information, - is displayed in the corresponding column.

The following commands show how to enable the memory watermark defense and the automatic per-task memory cap while the server is running, and set the policy for selecting the task to interrupt when the hard watermark is reached to largest:

./copt_clstool -s <ip> -p <port> setmemguard on
./copt_clstool -s <ip> -p <port> setmemlimitauto on
./copt_clstool -s <ip> -p <port> setmemevictpolicy largest

To disable the corresponding features, execute respectively:

./copt_clstool -s <ip> -p <port> setmemguard off
./copt_clstool -s <ip> -p <port> setmemlimitauto off

setmemevictpolicy supports the four policies largest, longest, youngest and random.

Changes made by these commands are not written back to cls.ini. After the server restarts, the settings in cls.ini are used again.

To run managing tool on other machine, its IP should be added to a rule in ToolList section in filter configuration file clsfilters.ini. The following command from other machine lists cluster information of server 192.168.1.11.

> ./copt_clstool -s 192.168.1.11 GetNodes

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7878
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] node info
  [Parent] (null):7878 (Lost)
  [Child] Node_192.168.1.12:7878_N0001, v2.0=3
  Total num of child nodes is 1

The following command changes token number of server 192.168.1.11 from 3 to 0.

> ./copt_clstool -s 192.168.1.11 SetToken 0

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7878
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] total token was 3 and now set to 0

The following command shows all filter lists of server 192.168.1.11, including those in BlackList section, WhiteList section, ToolList section and MemWhiteList section.

> ./copt_clstool -s 192.168.1.11 GetFilters

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7979
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] filters info
[BlackList]

[WhiteList]

[ToolList]
  [1]  127.0.0.1

[MemWhiteList]

The following command added user of IP 192.168.3.13 to black list.

> ./copt_clstool -s 192.168.1.11 AddBlackRule 192.168.3.133

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7979
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] server added new black rule (succeeded)

The following command shows that a new rule in BlackList section is added.

> ./copt_clstool -s 192.168.1.11 GetFilters

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7979
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] filters info
[BlackList]
  [1] 192.168.3.133

[WhiteList]

[ToolList]
  [1]  127.0.0.1

[MemWhiteList]

The following command disable a rule in BlackList section.

> ./copt_clstool -s 192.168.1.11 ToggleBlackRule 1

[ Info] COPT Cluster Managing Tool, COPT v8.0.5 20240304
[ Info] connecting to 192.168.1.11:7979
[ Info] [command] wait for connecting to cluster
[ Info] [cluster] server toggle black rule [1] (succeeded)

The following command adds a per-task memory cap exemption rule for the given client. The server IP, the port and the rule in the command should be replaced according to the actual deployment.

./copt_clstool -s <ip> -p <port> addmemexempt '127.0.*.*+user@machine*'

After adding the rule, the current filter rules can be listed with:

./copt_clstool -s <ip> -p <port> getfilters

To write the filter rules currently in memory back to the filter configuration file, execute:

./copt_clstool -s <ip> -p <port> writefilters

The following command deletes the per-task memory cap exemption rule added above:

./copt_clstool -s <ip> -p <port> delmemexempt '127.0.*.*+user@machine*'

Changes made to the rules in memory by addmemexempt and delmemexempt take effect immediately; writefilters writes all filter rules, including the memory cap exemption rules, back to the configuration file.

In the commands above, <ip> and <port> denote the IP address and the port of the cluster server. They should be replaced with the actual values, and the angle brackets should not be kept.

Running as service

To run COPT compute cluster server as a system service, follow steps described in readme.txt under cluster folder, and set config file copt_cluster.service properly.

Below is readme.txt, which lists installing steps in both Linux and MacOS platforms.

[Linux] To run copt_cluster as a service with systemd

Add a systemd file
    cp copt_cluster.service to /lib/systemd/system/
    sudo systemctl daemon-reload

Enable new service
    sudo systemctl start copt_cluster.service
    or
    sudo systemctl enable copt_cluster.service

Restart service
    sudo systemctl restart copt_cluster.service

Stop service
    sudo systemctl stop copt_cluster.service
    or
    sudo systemctl disable copt_cluster.service

Verify service is running
    sudo systemctl status copt_cluster.service

[MacOS] To run copt_cluster as a service with launchctrl

Add a plist file
    cp copt_cluster.plist to /Library/LaunchAgents as current user
    or
    cp copt_cluster.plist to /Library/LaunchDaemons with the key 'UserName'

Enable new service
    sudo launchctl load -w /Library/LaunchAgents/copt_cluster.plist
    or
    sudo launchctl load -w /Library/LaunchDaemons/copt_cluster.plist

Stop service
    sudo launchctl unload -w /Library/LaunchAgents/copt_cluster.plist
    or
    sudo launchctl unload -w /Library/LaunchDaemons/copt_cluster.plist

Verify service is running
    sudo launchctl list solver.copt.cluster

Linux

Below are steps in details of how to run COPT compute cluster server as a system service in Linux platform.

For instance, assume that COPT remote service is installed under '/home/eleven'. In your terminal, type the following command to enter the root directory of cluster service.

cd /home/eleven/copt_remote80/cluster

modify template of the service config file copt_cluster.service in text format:

[Unit]
Description=COPT Compute Cluster Server

[Service]
WorkingDirectory=/path/to/service
ExecStart=/path/to/service/copt_cluster
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

That is, update template path in keyword WorkingDirectory and ExecStart to actual path where the cluster service exits.

[Unit]
Description=COPT Compute Cluster Server

[Service]
WorkingDirectory=/home/eleven/copt_remote80/cluster
ExecStart=/home/eleven/copt_remote80/cluster/copt_cluster
Restart=always
RestartSec=1

[Install]
WantedBy=multi-user.target

Afterwards, copy copt_cluster.service to system service folder /lib/systemd/system/ (see below).

sudo cp copt_cluster.service /lib/systemd/system/

The following command may be needed if you add or update service config file. It is not needed if service unit has been loaded before.

sudo systemctl daemon-reload

The following command starts the new cluster service.

sudo systemctl start copt_cluster.service

To verify the cluster service is actually running, type the following command

sudo systemctl status copt_cluster.service

If you see logs similar to below, COPT compute cluster server is running successfully as a system service.

copt_cluster.service - COPT Cluster Server
Loaded: loaded (/lib/systemd/system/copt_cluster.service; enabled; vendor preset: enabled)
Active: active (running) since Sat 2021-08-28 11:46:10 CST; 3s ago
Main PID: 3054 (copt_cluster)
    Tasks: 6 (limit: 4915)
CGroup: /system.slice/copt_cluster.service
          └─3054 /home/eleven/copt_remote80/cluster/copt_cluster

eleven-ubuntu systemd[1]: Started COPT Cluster Server.
eleven-ubuntu COPTCLS[3054]: LWS: 4.1.4-b2011a00, loglevel 1039
eleven-ubuntu COPTCLS[3054]: NET CLI SRV H1 H2 WS IPv6-absent
eleven-ubuntu COPTCLS[3054]: server started at port 7878
eleven-ubuntu COPTCLS[3054]: LWS: 4.1.4-b2011a00, loglevel 1039
eleven-ubuntu COPTCLS[3054]: NET CLI SRV H1 H2 WS IPv6-absent
eleven-ubuntu COPTCLS[3054]: [NODE] node has been initialized

To stop the cluster service, type the following command

sudo systemctl stop copt_cluster.service

MacOS

Below are steps in details of how to run COPT Compute Cluster server as a system service in MacOS platform.

For instance, assume that COPT remote service is installed under "/Applications". In your terminal, type the following command to enter the root directory of cluster service.

cd /Applications/copt_remote80/cluster

modify template of the service config file copt_cluster.plist in xml format:

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>solver.copt.cluster</string>
        <key>Program</key>
        <string>/path/to/service/copt_cluster</string>
        <key>RunAtLoad</key>
        <true/>
        <key>KeepAlive</key>
        <true/>
    </dict>
</plist>

That is, update template path in Program tag to actual path where the cluster service exits.

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>solver.copt.cluster</string>
        <key>Program</key>
        <string>/Applications/copt_remote80/cluster/copt_cluster</string>
        <key>RunAtLoad</key>
        <true/>
        <key>KeepAlive</key>
        <true/>
    </dict>
</plist>

Afterwards, copy copt_cluster.plist to system service folder /Library/LaunchAgents (see below).

sudo cp copt_cluster.plist /Library/LaunchAgents

The following command starts the new cluster service.

sudo launchctl load -w /Library/LaunchAgents/copt_cluster.plist

To verify the cluster service is actually running, type the following command

sudo launchctl list solver.copt.cluster

If you see logs similar to below, COPT compute cluster server is running successfully as a system service.

{
    "LimitLoadToSessionType" = "System";
    "Label" = "solver.copt.cluster";
    "OnDemand" = false;
    "LastExitStatus" = 0;
    "PID" = 16406;
    "Program" = "/Applications/copt_remote80/cluster/copt_cluster";
};

To stop the cluster service, type the following command

sudo launchctl unload -w /Library/LaunchAgents/copt_cluster.plist

If the cluster service should be run by a specific user, add UserName tag to config file. Below adds a user eleven, who has privilege to run the cluster service.

<?xml version="1.0" encoding="UTF-8"?>
<plist version="1.0">
    <dict>
        <key>Label</key>
        <string>solver.copt.cluster</string>
        <key>Program</key>
        <string>/Applications/copt_remote80/cluster/copt_cluster</string>
        <key>UserName</key>
        <string>eleven</string>
        <key>RunAtLoad</key>
        <true/>
        <key>KeepAlive</key>
        <true/>
    </dict>
</plist>

Then copy new copt_cluster.plist to system service folder /Library/LaunchDaemons (see below).

sudo cp copt_cluster.plist /Library/LaunchDaemons

The following command starts the new cluster service.

sudo launchctl load -w /Library/LaunchDaemons/copt_cluster.plist

To stop the cluster service, type the following command

sudo launchctl unload -w /Library/LaunchDaemons/copt_cluster.plist