An IncrediBuild environment consists of two major components: The IncrediBuild Coordinator (server) and IncrediBuild Agents (clients). These combine server-based and peer-to-peer protocols to achieve a robust, effective and scalable distribution environment.

IncrediBuild Agents

IncrediBuild Agents are client components responsible both for initiating distributed jobs as well as for participating in distributed jobs initiated by other Agents as remote computing resources.

At its most basic level of functionality, an IncrediBuild Agent acts as a "Helper", executing computing tasks from distributed jobs initiated by other Agents. XGE (Xoreax Grid Engine) Virtualization technology ensures a task will be executed exactly as if it were being executed on the computer that initiated the job - regardless of the remote Agent's file system, installation base and environment. Output files are similarly created on the initiating computer's file system.
Additional processors can be utilized to improve processing power, with each CPU carrying out a different task simultaneously.

IncrediBuild Agents are also able to initiate distributed jobs, comprising of tasks (a command line process, such as the compilation of a C++ source file). Throughout the distributed job execution, the Coordinator assigns remote ("Helper") Agents to the executing job. The initiating Agent communicates with these Agents, managing task assignment. Relevant input files are transferred upon demand to remote Agents executing the different tasks, while output is transferred from the remote Agents to the local file system.

IncrediBuild Coordinator

The IncrediBuild Coordinator is the server component in an IncrediBuild environment, responsible for keeping track of system status and for assignment of computing resources (Agents) to distributed jobs. The Coordinator maintains a dynamic, real-time view of the IncrediBuild environment, combining hardware and software profiles of every Agent with their current availability and state. This information is used to ensure every distributed job is assigned the best possible combination of computing resources.

Easily handling 500+ connected Agents, the Coordinator can scale up with an existing IncrediBuild environment as more nodes are added to the grid. Additional Coordinator workgroups can be used side-by-side for further scalability.

Another crucial role performed by the Coordinator is that of a single-point configuration and management center for connected Agents. Using the Coordinator Monitor, version upgrades, maintenance jobs and modification of user settings can be centrally performed with ease.

A Backup Coordinator may be set up to ensure near-100% system availability. The Backup Coordinator assumes control whenever the primary Coordinator becomes unavailable for any reason, alerting users of the condition but otherwise maintaining all system functionality. Once the primary Coordinator is restored, normal operation is resumed.

Using XGE-based distribution, up to a X20 improvement in execution time may be achieved. The following graph depicts the improvement in build time of a Visual Studio project, as IncrediBuild distributes compilation tasks in parallel using more and more computers:

Performance-Related Features

XGE technology incorporates a combination of techniques designed to maximize the efficiency of task distribution over the network. Here are a few examples:

• File Caching: In order to cut down on network traffic, a file cache is created on every Agent's file system, storing recently accessed files. Intelligent file swapping, hard-linking and metadata refresh algorithms are utilized in order to bring file synchronization to a minimum. File caching is particularly effective when source files used by distributed tasks are stored on network storage devices, which are typically slower to access in comparison to local file storage.
• Real-Time Compression: To further reduce network traffic, data transferred over the network is compacted using real-time compression/decompression algorithms.
• Idle CPU Operation: Office workstations are typically used for tasks such as text editing, surfing the Internet and data-based applications. Since these tasks consume very little CPU, they make office workstations excellent resources for accelerating CPU-intensive jobs through network distribution. IncrediBuild achieves this by executing distributed tasks on workstations using idle CPU cycles, meaning anyone using a workstation practically doesn't notice when it is being utilized as a computing resource. Installing IncrediBuild Agents on office workstations thus helps utilize an office's existing hardware while eliminating costs for dedicated Grid hardware.
• Adaptive Resource Assignment: Because availability of computers in the Grid constantly varies, XGE employs adaptive assignment logic that responds to changes in nodes' availability. As soon as a node's CPU/Network availability falls below a minimum level (as a result of other activity on that node), it automatically stops participating as a remote resource in the grid, and any uncompleted tasks being executed on that node are transferred to another, more available Agent. This behavior improves the distributed job performance while keeping remote nodes responsive.
• Multiple CPU/Core Utilization: IncrediBuild Agents can utilize additional processors and cores whenever available. CPUs/Cores can be used as independent computing resources, each capable of executing a task in parallel with others. As a result, each computer's hardware is fully utilized.
• Task Doubling: This unique feature optimizes performance whenever an Agent executing a distributed job is left waiting for a single remote Agent to complete a task. As soon as this state is reached, the node that initiated the distributed job starts executing the pending task in parallel, "racing" against the remote Agent. As soon as one of the Agents completes the task, output from that Agent is applied to the distributed job, which can consequently either complete execution or continue executing any remaining tasks which depended on the completed task.

Virtualization is the cornerstone of XGE (Xoreax Grid Engine) technology. XGE Virtualization technology guarantees that a computational task will always generate reliable results, regardless of which node is executing that task. Uniquely, this is achieved without having to create and manage system images required by different types of tasks.

Whenever a distributed job is initiated, it is essential that all tasks comprising that job will be executed using the initiating node's environment, file system, registry, etc. For example, a C++ compiler task run on a remote Agent should always use the initiating node's file system to ensure that the correct copies of source code files are compiled. XGE makes this possible through the use of a "Virtual Environment".

XGE Virtual Environment

Tasks distributed to IncrediBuild Agents are executed using a special mode of operation, which uses a "Virtual Environment" for these tasks to operate in. This Virtual Environment replicates the initiating node's environment and file system, causing tasks executing within the Virtual Environment to always access the initiating node's environment and file system, while utilizing the remote node's CPU and memory resources.

Tasks executing on a remote node similarly cannot alter the remote node's file system or registry. Instead, any modification to the file system is transferred to the initiating node, ensuring the integrity of the distributed process while keeping the remote node's environment unaffected.

The concept of a Virtual Environment means that no pre-configuration of remote nodes is required in order for an Agent to perform distributed tasks. Instead, all synchronization is performed automatically and safely by the XGE.

Managing and maintaining a Grid of interconnected computers is a complex task deserving special consideration. IncrediBuild has been designed with the notion of simplifying common maintenance tasks such as version upgrades and remote node administration:

Installation

1. Install a Coordinator.
2. Install an IncrediBuild Agent on client nodes.

Central Administration

Network Monitoring

Version Upgrades