This document discusses the use of multiple threads in client programs and object implementations, and will help you understand the VisiBroker thread and connection model.
A thread, or a single sequential flow of control within a process, is also called a lightweight process that reduces overhead by sharing fundamental parts with other threads. Threads are lightweight so that there can be many of them present within a process.
Using multiple threads provides concurrency within an application and improves performance. Applications can be structured efficiently with threads servicing several independent computations simultaneously. For example, a database system may have many user interactions in progress while at the same time performing several file and network operations.
Although it is possible to write the software as one thread of control moving asynchronously from request to request, the code may be simplified by writing each request as a separate sequence, and letting the underlying system handle the synchronous interleaving of the different operations.
Multiple threads are useful when:
Thread and connection management occurs within the scope of an entity known as a server engine. Several default server engines are created automatically by VisiBroker, which include thread pool engines for IIOP, for LIOP, and so forth. Additional server engines can be used and created in a VisiBroker server by applications. See the example in <install_dir>/examples/poa/server_engine_policy/Server.C. Server engines are created, configured, and used independently. The creation and configuration of one server engine does not affect other server engines in the same server. Usually, each server engine has one transport end point, called the listen point/socket.
The relationship between server engines and POAs is many-to-many. Each server engine can be used by multiple POAs, and each POA may also use multiple server engines.
Server engines can consist of multiple Server Connection Managers (SCMs). An SCM is composed of managers, listeners, and dispatchers. The properties of managers, listeners and dispatchers can be configured to determine how the SCM functions. These properties are discussed in "Setting Connection Management Properties".
Each server engine has a listener and a dispatcher thread. The listener thread is responsible for:
The dispatcher determines which threads to send requests to.
Each server engine uses a certain number of worker threads to receive and process requests. Different requests may handled by different worker threads. For a given request, the request reading, processing (include server side interceptor intercepting), and replying are all handled by the same thread. The number of worker threads used by a server engine depends on:
The two major thread models supported by VisiBroker are the thread pool (also known as thread-per-request, or TPool) and thread-per-session (also known as thread-per-connection, or TSession). Single-thread and main-thread models are not discussed in this document. The thread pool and thread-per-session models differ in these fundamental ways:
The default thread policy is the thread pool. For information about setting thread-per-session or changing properties for the thread pool model, see "Setting dispatch policies and properties."
When your server uses the thread pool policy, it defines the maximum number of threads that can be allocated to handle client requests. A worker thread is assigned for each client request, but only for the duration of that particular request. When a request is completed, the worker thread that was assigned to that request is placed into a pool of available threads so that it may be reassigned to process future requests from any of the clients.
Using this model, threads are allocated based on the amount of request traffic to the server object. This means that a highly active client that makes many requests to the server at the same time will be serviced by multiple threads, ensuring that the requests are quickly executed, while less active clients can share a single thread, and still have their requests immediately serviced. Additionally, the overhead associated with the creation and destruction of worker threads is reduced, because threads are reused rather than destroyed, and can be assigned to multiple new connections.
VisiBroker conserves system resources by dynamically allocating the number of threads in the thread pool based on the number of concurrent client requests by default. If the client becomes very active, new threads are allocated to meet its needs. If threads remain inactive, VisiBroker releases them, only keeping enough threads to meet current client demand. This enables the optimal number of threads to be active in the server at all times.
The size of the thread pool grows based upon server activity and is fully configurable, either before or during execution, to meet the needs of specific distributed systems. With the thread pool model, you can configure the following:
Each time a client request is received, an attempt is made to assign a thread from the thread pool to process the request. If this is the first client request and the pool is empty, a thread will be created. Likewise, if all threads are busy, a new thread will be created to service the request.
A server can define a maximum number of threads that can be allocated to handle client requests. If there are no threads available in the pool and the maximum number of threads have already been created, the request will block until a thread currently in use has been released back into the pool.
Thread pool is the default thread policy. You do not have to set up anything to define this environment. If you want to set properties for the thread pool, see "Setting dispatch policies and properties."
Pool of threads is available
The figure above shows the object implementation using the thread pool policy. As the name implies, there is an available pool of worker threads in this policy.
Client application #1 sends a request
In the above figure, Client application #1 establishes a connection to the Object Implementation and a thread is created to handle requests. In the thread pool, there is one connection per client and one thread per connection. When a request comes in, a worker thread receives the request; that worker thread is no longer in the pool.
A worker thread is removed from the thread pool and is always listening for requests. When a request comes in, that worker thread reads in the request and dispatches the request to the appropriate object implementation. Prior to dispatching the request, the worker thread wakes up one other worker thread which then listens for the next request.
Client application #2 sends a request
As the above figure shows, when Client application #2 establishes its own connection and sends a request, a second worker thread is created. Worker thread #3 is now listening for incoming requests.
Client application #1 sends a second request
The above figure shows that when a second request comes in from Client application #1, it uses worker thread #4. Worker thread #5 is spawned to listen for new requests. If more requests came in from Client application #1, more threads would be assigned to handle them, each spawned after the listening thread receives a request. As worker threads complete their tasks, they are returned to the pool and become available to handle requests from any client.
With the thread-per-session (TSession) policy, threading is driven by connections between the client and server processes. When your server selects the thread-per-session policy, a new thread is allocated each time a new client connects to a server. A thread is assigned to handle all the requests received from a particular client. Because of this, thread-per-session is also referred to as thread-per-connection. When the client disconnects from the server, the thread is destroyed. You may limit the maximum number of threads that can be allocated for client connections by setting the vbroker.se.iiop_ts.scm.iiop_ts.manager.connectionMax property.
Object implementation using the thread-per-session policy
The above figure shows the use of the thread-per-session policy. The Client application #1 establishes a connection with the object implementation. A separate connection exists between Client application #2 and the object implementation. When a request comes in to the object implementation from Client application #1, a worker thread handles the request. When a request from Client application #2 comes in, a different worker thread is assigned to handle this request.
Second request comes in from the same client
In the above figure, a second request has come in to the object implementation from Client application #1. The same thread that handles request 1 will handle request 2. The thread blocks request 2 until it completes request 1. (With thread-per-session, requests from the same Client are not handled in parallel.) When request 1 has completed, the thread can handle request 2 from Client application #1. Multiple requests may come in from Client application #1—they are handled in the order that they come in, no additional threads are assigned to Client application #1.
Overall, VisiBroker's connection management minimizes the number of client connections to the server. In other words there is only one connection per server process which is shared. All requests from a single client application are multiplexed over the same connection, even if they originate from different threads. Additionally, released client connections are recycled for subsequent reconnects to the same server, eliminating the need for clients to incur the overhead of new connections to the server.
In the following scenario, a client application is bound to two objects in the server process. Each bind() shares a common connection to the server process, even though the bind() is for a different object in the server process.
Binding to two objects in the same server process
The following figure shows the connections for a client using multiple threads that has several threads bound to an object on the server.
Binding to an object in a server process
As the above figure shows, all invocations from all threads are serviced by the same connection. For that scenario, the most efficient multi threading model to use is the thread pool model. If the thread-per-session model is used in this scenario, only one thread on the server will be allocated to service all requests from all threads in the client application, which could easily result in poor performance.
The maximum number of connections to a server, or from a client, can be configured. Inactive connections will be recycled when the maximum is reached, ensuring resource conservation.
Thread and connection management on the server side is performed by ServerEngines, which can consist of one or more Server Connection Managers (SCMs). An SCM is a collection of properties of the manager, listener, and dispatcher.
Defining a ServerEngine consists of specifying a set of properties in a properties file. For example, if on UNIX the property file called myprops.properties is in home directory, the command line is
prompt> vbj -DORBpropStorage=~/myprops.properties myServer
vbroker.se.<srvr_eng_name>.scms=<srvr_connection_mngr_name1>,<srvr_connection_mngr_name2>
The set of Server Connection Managers associated with a ServerEngine is defined by this property. The name specified in the above property as the <svr_eng_name> is the name of the ServerEngine. The SCMs listed here will be the list of SCMs for the associated server engine. SCMs cannot be shared between ServerEngines. However, ServerEngines can be shared by multiple POAs.
The other properties are
vbroker.se.<se>.host
The host property is the IP address for the server engine to listen for messages.
vbroker.se.<se>.proxyHost
The proxyHost property specifies the proxy IP address to send to the client in the case where the server does not want to publish its real hostname.
Each POA in a multi-threaded object server can choose between two dispatch models: thread-per-session or thread pool. You choose a dispatch policy by setting the dispatcher.type property of the ServerEngine.
vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=ThreadPool vbroker.se.<srvr_eng_name>.scm.<srvr_connection_mngr_name>.dispatcher.type=ThreadSession
For more information about these properties see the VisiBroker Programmer's Reference.
ThreadPool (thread pooling) is the default dispatch policy when you create a POA without specifying the ServerEnginePolicy.
For ThreadPool, you can set the following properties:
vbroker.se.default.dispatcher.tp.threadMax
TPool server engine's maximum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
vbroker.se.default.dispatcher.tp.threadMax=32or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=32sets the initial maximum worker thread limitation to 32 for the default
TPool server engine. The default value of this property is unlimited (0). If there are no threads available in the pool and the maximum number of threads have already been created, the request is blocked until a thread currently in use has been released back into the pool.vbroker.se.default.dispatcher.tp.threadMin
TPool server engine's minimum number of worker threads in the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the start up property
vbroker.se.default.dispatcher.tp.threadMin=8or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=8sets the initial worker thread minimum number to 8 for the default
TPool server engine. The default value of this property is 0 (no worker threads).vbroker.se.default.dispatcher.tp.threadMaxIdle
vbroker.se.default.dispatcher.tp.threadMaxIdle=120or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=120sets the initial idle worker thread check interval to 120 seconds for the default
TPool server engine. The default value of this property is 300 seconds. With this setting, the server engine will check the idle state of each worker thread every 120 seconds. If a worker thread has been idle across two consecutive checks, it will be recycled (terminated) at the second check. Therefore, the actual idle thread garbage collection time is between 120 to 240 seconds under the above setting, instead of exactly 120 seconds.vbroker.se.default.dispatcher.tp.coolingTime
ThreadPool dispatcher allows a "cooling time" to be set. A thread is said to be "hot" when the GIOP connection being served is potentially readable, either upon creation of the connection or upon the arrival of a request. After the cooling time (in seconds), the thread can be returned to the thread pool. The property can be set statically on server startup or dynamically reconfigured using the property API. For instance, the startup property
vbroker.se.default.dispatcher.tp.coolingTime=6or
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=6sets the initial cooling time to 6 seconds for the default engine (or the IIOP
TPool server engine). The default value of this property is 3 seconds. The maximum value is 10 seconds.Note: The vbroker.se.default.xxx.tp.xxx property is recommended when vbroker.se.default=iiop_tp. When using with ThreadSession, it is recommended that you use the vbroker.se.iiop_ts.scm.iiop_ts.xxx property.
When using the ThreadSession as the dispatcher type, you must set the se.default property to iiop_ts.
vbroker.se.default=iiop_ts
Note: In thread-per-session, there are no threadMin, threadMax, threadMaxIdle, and coolingTime dispatcher properties. Only the Connection and Manager properties are valid properties for ThreadSession.
All code within a server that implements the VisiBroker ORB object must be thread-safe. You must take special care when accessing a system-wide resource within an object implementation. For example, many database access methods are not thread-safe. Before your object implementation attempts to access such a resource, it must first lock access to the resource using a synchronized block.
If serialized access to an object is required, you need to create the POA on which this object is activated with the SINGLE_THREAD_MODEL value for the ThreadPolicy.
The following properties are used to configure connection management. Properties whose names start with vbroker.se are server-side properties. The client side properties have their names starting with vbroker.ce.
The command line options for VisiBroker 3.x backward-compatibility are less obvious in terms of whether they are client-side or server-side. However, the connection and thread management options that start with the -ORB prefix set the client-side options whereas the options with the -OA prefix are used for the server-side options. There are no common properties which are used for both client-side and server-side thread and connection management.
The distinction between client and server vanishes if callback or bidirectional GIOP is used.
vbroker.se.default.socket.manager.connectionMax
-Dvbroker.se.default.socket.manager.connectionMax=128or
-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax=128sets the initial maximum connection limitation on this server engine to 128. The default value of this property is unlimited (0). When the server engine reaches this limitation, before accepting a new client connection, the server engine needs to reuse an idle connection. This is called connection swapping. When a new connection arrives at the server, it will try to detach the oldest unused connection. If all the connections are busy, the new connection will be dropped. The client may retry again until some timeout expires.
vbroker.se.default.socket.manager.connectionMaxIdle
-Dvbroker.se.default.socket.manager.connectionMaxIdle=300or
-Dvbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=300sets the initial idle connection maximum lifetime to 300 seconds. The default value of this property is 0 (unlimited). When a client connection has been idle longer than this value, it becomes a candidate for garbage collection.
vbroker.ce.iiop.ccm.connectionMax
CORBA::NO_RESOURCE exception will result.The following properties have a fixed set or range of valid values:
vbroker.ce.iiop.ccm.type=Pool
Pool is the only supported type.In the following properties, xxx is the server engine name and yyy is the server connection manager name:
vbroker.se.xxx.scm.yyy.manager.type=Socket
Local for LIOP (only in VisiBroker for C++) and BIDIR for bidir (bidirectional) SCMs.vbroker.se.xxx.scm.yyy.listener.type=IIO
LIOP for local IPC (only in VisiBroker for C++) and SSL for security.vbroker.se.xxx.scm.yyy.disptacher.type=ThreadPool
ThreadSession and MainThread.vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=3
The effect of a change in a property value depends on the actions associated with the properties. Most of the actions are directly or indirectly related to the utilization of system resources. The availability and restrictions of the system resources to the CORBA application vary depending on the system and the nature of the application.
For instance, increasing the garbage collector timer may increase the system activities, as the garbage collector will run more frequently. On the other hand, increasing its value means the idle threads will remain in system unclaimed for longer periods of time.
The following properties can be changed dynamically and the effect will be immediate unless stated otherwise:
vbroker.ce.iiop.ccm.connectionCacheMax=5 vbroker.ce.iiop.ccm.connectionMax=0 vbroker.ce.iiop.ccm.connectionMaxIdle=360 vbroker.ce.iiop.connection.rcvBufSize=0 vbroker.ce.iiop.connection.sendBufSize=0 vbroker.ce.iiop.connection.tcpNoDelay=false vbroker.ce.iiop.connection.socketLinger=0 vbroker.ce.iiop.connection.keepAlive=true vbroker.ce.liop.ccm.connectionMax=0 vbroker.ce.liop.ccm.connectionMaxIdle=360 vbroker.ce.liop.connection.rcvBufSize=0 vbroker.ce.liop.connection.sendBufSize=0 vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMax=0 vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=0 vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMin=0 vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMax=100 (The new dispatcher threadMax properties will be reflected after the next garbage collector run.) vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=300 vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.coolingTime=3 vbroker.se.iiop_tp.scm.iiop_tp.manager.garbageCollectTimer=30 vbroker.se.liop_tp.scm.liop_tp.listener.userConstrained=false
For this purpose, the server manager needs to be enabled, using the property vbroker.orb.enableServerManager=true, and the properties can be obtained through the server manager query either through the Console or through a command-line utility.
It is very difficult to determine the impact of changing the value of a property to something other than the default. For thread and connection limits, the available system resources vary depending on the machine configuration and the number of other processes running. The setting of properties allows performance tuning for a given system.
The VisiBroker for Java ORB performs automatic garbage collection of various resources other than the memory. The garbage collection of the memory is performed by the Java Virtual machine. Various properties are provided to control the garbage collection period. In addition, resources like threads and connections define timeout properties that control the collection of these resources.
The ORB garbage collector thread is a normal priority thread. After the expiration of timeout period (specified by the property vbroker.orb.gcTimeout), it wakes up and collects all the resources that are no longer in use (i.e. idle). Classes interested in getting collected register themselves with the garbage collector. Such classes are called collectables. Prominent examples of collectables are threads and connections. Other examples include timeout on various caches (e.g. GateKeeper cache). Most of the collectables null out or properly release the resources (e.g. closing the connection or terminating a thread's "run" method) held by them when they are collected. These resources are later reclaimed by the Java garbage collector. Note that ORB garbage collector is an internal service and is not exposed to the user.
The main property that controls the garbage collection period is vbroker.orb.gcTimeout. The timeout value is in seconds and the default value is 30 seconds.
Threads and connections define properties for idle timeout. For example, the thread pool dispatcher defines the following property:
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle
The value is in seconds and default value is 300 seconds after which the thread is removed from the thread pool. Similarly, the default Server Connection Manager (iiop_tp) defines the following idle timeout property for connections.
vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle
The value is in seconds and default value is 0 which means a connection never gets closed no matter how long it remains idle. However, if the connection gets dropped, the ORB removes all the references to it and its resources are later collected by Java garbage collector. The ORB garbage collector will only collect connections whose connectionMaxIdle property is set to a non-zero value.
The various timeout properties and the vbroker.orb.gcTimeout property have a subtle relationship. For example, suppose following properties are specified:
vbroker.orb.gcTimeout=10 vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=5 vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=5
Here the garbage collection timeout period is set to 10 seconds whereas thread and connection timeouts are 5 seconds. The figure below illustrates how these properties interact. Here we have shown a thread, T1, and a connection, C1, that have gone idle and are then collected.
Note: Although the ORB garbage collector is shown here as running exactly after 10 seconds, in practice this may not be true depending on when the JVM schedules the garbage collector (GC) thread.
Collection of resources by ORB GC
Note that even though T1 and C1 are eligible for collection, they are collected only when the ORB garbage collector runs. Until then they remain in the timed out state.
A dispatcher's thread pool in VisiBroker for C++ has an idle timeout vbroker.se.xxx.scm.xxx.dispatcher.threadMaxIdle. The default value is 300 seconds, and after the idle timeout expires the dispatcher will remove any idle worker threads in the thread pool.
A Server Connection Manager (SCM) has its own garbage collection timeout vbroker.se.xxx.scm.xxx.manager.garbageCollectTimer. The default value is 30 seconds, and after the timeout expires any idle connections are garbage collected.
Since the SCM only garbage collects idle connections, the property vbroker.se.xxx.scm.xxx.manager.connectionMaxIdle needs to be set greater than zero in order for connections to go to an idle state. The default value is 0, which means that a connection is never considered idle and nothing is collected, even if the SCM's garbage collection timeout expires.
Unlike VisiBroker for Java, there is no notion of an ORB garbage collector. The dispatcher and the SCM perform garbage collection independently. Hence given the values below.
vbroker.se.iiop_tp.scm.iiop_tp.dispatcher.threadMaxIdle=5 vbroker.se.iiop_tp.scm.iiop_tp.manager.connectionMaxIdle=5 vbroker.se.iiop_tp.scm.iiop_tp.manager.garbageCollectTimer=10
When the thread pool worker thread, T1, has been idle for 5 seconds it is immediately removed from the dispatcher's thread pool. The connection, C1, which has been idle for 5 seconds is only garbage collected by the SCM after 10 seconds.
Collection of resources by the SCM GC
On the Client side the Client Connection Manager (CCM) cached connections can be given an idle timeout by setting the property vbroker.ce.xxx.ccm.connectionMaxIdle. The default value is 0 (i.e. cached connections do not have an idle timeout). Given an idle timeout, the idle cached connections in the connection pool/cache are marked eligible for garbage collection. Unlike the SCM, the CCM has no garbage collection timer, however whenever any connection is being cached it will attempt to garbage collect any cached connections that are marked eligible for collection.