|
Overview
|
|
|
Pipeline is a distributed network based application composed of several
server daemon processes which may run on one or more hosts. These
server processes communicate with each other, all client programs used by
artists and potentially a large number of job server daemons responsible
for executing and monitoring jobs. Configuration of Pipeline is
therefore necessarily more complex than most user level applications.
The network diagram below (figure 1) shows a
typical Pipeline site configuration.
|
|
figure 1
|
|
|
The Pipeline daemons provide the following services:
|
plpluginmgr
|
The Plugin Manager daemon is responsible for validating, installing and
loading Pipeline plugin classes as well as providing synchronized
versions of loaded plugin classes to all other Pipeline programs.
|
|
|
plfilemgr
|
Th File Manager daemon performs all production related file system
operations on behalf of the other Pipeline programs. This includes
comparing, copying, linking and deleting files in both the repository
and user working areas required by the Pipeline revision control system.
|
|
|
plmaster
|
The Master daemon maintains the Pipeline node database and
coordinates the activity of all Pipeline server programs. Client
programs used by artists communicate with this daemon to initiate all
node, execution queue and system administration operations provided
by Pipeline.
|
|
|
plqueuemgr
|
The Queue Manager daemon manages the job execution queue. This
includes the collection of the dynamic resources available at the
facility, the allocation of these resources to execute jobs and
monitoring of the execution of these jobs.
|
|
|
pljobmgr
|
The Job Manager daemons are run on each render farm node and artist
workstation which provides processing resources to the Pipeline
job execution queue. This daemon collects information about the
dynamic resources available on a host, launches jobs and monitors
the execution of these jobs.
|
|
|
|
The configuration of Pipeline involves selecting the best host to run
each of these daemons in order to achieve the best possible performance
at a given site. Although there are large number of possible
configurations, most facilities will want to choose a configuration based
on one of the two following strategies.
|
|
|
Local File Manager
|
|
|
If the network file server at your facility runs Linux (or any other UNIX
flavor) and has some spare processing capacity you may want to run the
File Manager daemon on the file server host. This gives Pipeline the
best performance for all file system related operations since they are
performed locally with no NFS overhead.
File system operations usually make up the largest portion of the overall
time it takes Pipeline to perform any node related operation. This is
particularly true for nodes which are associated with multiple file sequences
such as rendered images or simulation data. A substantial increase
in performance can be achieved using this configuration.
The network diagram below (figure 2) shows a
site configured using a local File Manager daemon.
|
|
figure 2
|
|
|
In the diagram above, the Master daemon and Queue Manager daemon are run
on separate hosts. This is not required, but can increase performance
at medium and large facilities by distributing the processor and network
load. Both of these daemons may have large numbers of threads which
can utilize all available processors. The Master daemon maintains
network connections with all Pipeline client programs used by artists.
The Queue Manager daemon communicates with all Job Manager daemons.
When there are large numbers of artists using Pipeline and many hosts
participating in the job queue the difference in performance can be
significant.
|
|
|
Remote File Manager
|
|
|
Sometimes it is not possible to run the File Manager daemon on the network
file server. This is often the case when the file server is a special
purpose network appliance instead of a UNIX host. In some cases,
it may be undesirable to run the File Manager on the file server.
If the file server has little spare processing capacity or free memory, it
may negatively impact performance to run the File Manager locally.
In these cases the File Manager should be run on a host which has network
connection to the file server with low latency and as high bandwidth as
possible.
In configurations where the File Manager and Master daemons are run on the
same host, it is possible run the File Manager daemon as a thread
of the Master daemon instead of as a separate OS level process.
The first network diagram (figure 1) shows an
example of a remote File Manager daemon run as a thread of the Master
daemon. This configuration is faster than running these daemons
separately due to eliminating networking overhead and also uses less
memory since both daemons are being executed by a single Java VM.
In smaller facilities, it may be desirable to run all Pipeline server
daemons on a single host when processor and network load are more modest.
The following network diagram (figure 3)
shows an example of such a configuration.
|
|
figure 3
|
|
|
The Pipeline Configuration Tool
|
|
|
Pipeline configurations are specified using the
Pipeline Configuration
Tool. This program has options for specifying the hosts which
should run each Pipeline server daemon, the network ports used by these
daemons and the amount of memory to dedicate to each daemon. There
are also options for specifying the file system directories used for
production data files, user home directories, temporary files and node
database storage. See the
plconfig(1) and
plid(1) man pages for
usage details.
When executed, the Pipeline Configuration Tool generates a
Site Profile. The Site Profile uploaded to Temerity
Software using the
Customer Registration or
Update Profile pages.
Once we have received your Site Profile, we will build a custom distribution
of Pipeline tailored to the needs of your facility within 24-hours.
A confirmation e-mail will be sent giving the download location of this
custom Pipeline distribution.
|
The Pipeline Configuration Tool is provided as an RPM which can be
downloaded below:
|
