TOPs and PDG

Distributed computing and procedural iteration in Houdini

What Are TOPs and PDG?

TOPs (Task Operators) and PDG (Procedural Dependency Graph) make up one of the most ambitious systems added to Houdini in recent years. Officially introduced in Houdini 18, they fundamentally change how a production pipeline is built: instead of processing tasks sequentially, TOPs/PDG lets you orchestrate them in parallel, distribute them across multiple machines, and link them through explicit dependencies.

The central idea is simple but powerful: every operation - simulation, rendering, baking, geometry generation - becomes a node in a dependency graph. That graph, the PDG, knows exactly which order to run tasks in, which ones can run in parallel, and how one task's results feed the next. It's a genuine conductor for the entire production chain.

But TOPs isn't limited to distributed rendering. It's a general-purpose procedural computation tool: use it to generate hundreds of variations of an asset (a rock with different random parameters, say), to launch batch simulations, to bake textures en masse, or to automate complex pipelines spanning multiple pieces of software. In short, TOPs is a universal parallelization and automation system built directly into Houdini.

TOPs vs. PDG: What's the Difference?

PDG (Procedural Dependency Graph) is the underlying engine - the conceptual system that models dependencies between tasks and manages their execution. TOPs (Task Operators) are the visual nodes an artist places in the network to define those tasks. In practice the two terms are often used interchangeably, but the distinction is useful: PDG is the architecture, TOPs is the interface.

Core PDG Concepts

Work Items

The basic unit of PDG is the Work Item. Every task to run - rendering a frame, generating an asset variation, launching a simulation - is represented by a Work Item, which carries attributes (parameters, file paths, numeric values) and a status (pending, running, done, error).

When a TOP node generates multiple Work Items - say, 50 different parameter variations - each is processed independently and potentially in parallel. That mechanism is what turns a classic sequential workflow into a massively parallel one.

The Dependency Graph

TOP nodes are linked together in a graph that defines dependencies. If node B depends on node A, Houdini makes sure every Work Item from A finishes before B's begin. Dependencies can be simple (one task after another) or complex (several parallel branches merging into a single node).

Schedulers

A Scheduler decides how and where Work Items run. Houdini offers several depending on context:

  • Local Scheduler - runs tasks on the local machine using multithreading; the default for everyday work.
  • HQueue Scheduler - distributes tasks across a network of machines via HQueue; ideal for local render farms.
  • Deadline Scheduler - integration with Thinkbox's Deadline render manager (AWS); an industry standard.
  • Python Scheduler - for implementing custom schedulers targeting any third-party system.

Architecture of a TOP Network

A TOP network is built in Houdini's /tasks context (accessible via the Network Editor menu or from a TOP Network node in the SOP or OBJ context). The construction logic always follows the same pattern: generator nodes as input, processing nodes in the middle, and result or conditioning nodes at the output.

Essential Generator Nodes

  • Generic Generator - generates a single simple Work Item, a basic starting point for testing a network.
  • Wedge - the most important node for variations. Generates N Work Items by varying one or more parameters over a defined range (discrete values, a continuous range, or a value list). The heart of any procedural-iteration approach.
  • File Pattern - generates one Work Item per file matching a pattern; ideal for processing a batch of existing assets or caches.
  • Partition by Frame - splits a frame sequence into individual Work Items for distributed rendering.

Key Processing Nodes

  • Houdini Processor - executes a SOP/DOP/ROP network inside a Work Item; the most versatile node, able to run any Houdini operation.
  • ROP Fetch - triggers rendering of an existing ROP node (/out) within a Work Item's context - a direct link into Houdini's render pipeline.
  • ROP Geometry - exports SOP geometry to an external file (Alembic, USD, bgeo) per Work Item; essential for batch asset-variation generation.
  • Python Script - runs a Python script within the Work Item's context, letting you integrate any external tool or custom logic.
  • Shell Script - runs a shell command, opening access to any command-line tool.
  • Wait for All - a synchronization point that waits for every upstream Work Item to finish before continuing.

Worked Example: Generating Rock Variations

Here's a concrete, instructive example that shows exactly what TOPs/PDG is good for: automatically generating N variations of a procedural rock, each with different parameters (size, roughness, fracture count, random seed), exported to separate files, with an automatic preview render for each variation.

Step 1: Prepare the Rock's SOP Network

Before building the TOP network, you need a SOP network that generates the rock procedurally and parametrically. A typical /obj-context workflow:

  • Sphere SOP - base geometry, high subdivision (icosahedron, 4 subdivisions).
  • Mountain SOP - noise deformation for the rock's overall shape. Exposed parameters: Amplitude, Roughness, Element Size, Seed.
  • PolyReduce SOP - polygon reduction for the rocky, angular look. Exposed parameter: Percentage.
  • Facet SOP - hardens normals for a crystalline/fractured look.
  • Transform SOP - overall scaling. Exposed parameter: Scale.

The parameters exposed at the parent node level (via the Parameter Editor) are the ones TOPs will vary: Amplitude, Roughness, Seed, Scale - exposed via Parameters > Edit Parameter Interface.

Step 2: Build the TOP Network

In the /obj context, create a TOP Network node and double-click to enter it. Build this chain of nodes:

  1. Add a Wedge node - this generates all the variations. Configure the number of variations (say, 16) and the attributes to vary: 
    Attribute 1: rock_seed    | Type: Integer | Min: 0   | Max: 99
Attribute 2: rock_rough   | Type: Float   | Min: 0.1 | Max: 0.8
Attribute 3: rock_scale   | Type: Float   | Min: 0.5 | Max: 2.0
  2. Connect a Houdini Processor to the Wedge - it runs the rock's SOP network for each Work Item, mapping Work Item attributes to Houdini parameters: 
    @rock_seed   -> ch('../geo1/mountain1/seed')
@rock_rough  -> ch('../geo1/mountain1/roughness')
@rock_scale  -> ch('../geo1/transform1/scale')
  3. Connect a ROP Geometry node to the Houdini Processor, exporting the generated geometry to an Alembic or bgeo file per variation: 
    Output File: $HIP/rocks/rock_`@wedgeindex`.abc
  4. Optionally add a ROP Fetch node to automatically render a preview image of each variation with Karma or Mantra.
  5. Connect a Wait for All node to synchronize once every variation is done, before a final step (assembling a catalog, sending a notification, etc.).

Step 3: Run and Monitor the Network

To run the TOP network, click the terminal node's Cook button (or right-click > Cook Network). Houdini launches Work Items according to Scheduler availability. In the TOP viewport, each Work Item shows visually:

  • White circle - Work Item pending.
  • Yellow circle - Work Item running.
  • Green circle - Work Item finished successfully.
  • Red circle - Work Item failed. Right-click > Open Task Graph to diagnose.

The Task Graph View (Windows > Task Graph Table) gives a tabular view of every Work Item, its attributes, status and logs - the main debugging tool for a TOP network.

The Wedge Node: Mastering Iteration

Wedge is probably the most widely used TOP node in production, generating a set of Work Items by varying one or more parameters under different strategies.

Wedge Variation Modes

  • Range (Float/Integer) - varies linearly between a minimum and maximum across N samples. Example: Seed from 0 to 99 over 10 iterations.
  • Value List - varies through a specific list of values. Example: Scale with values [0.5, 1.0, 1.5, 2.0, 2.5].
  • Random - generates random values within a range, producing different results every run.

Combining Multiple Attributes

With several attributes defined in one Wedge, behavior depends on the combination mode:

  • Independent (default) - each attribute varies independently; the total Work Item count is the product of each attribute's variations. With 4 Seed values and 3 Scale values, you get 4 × 3 = 12 Work Items.
  • Simultaneous - attributes vary together, in parallel. With 4 Seed values and 4 Scale values, you get 4 Work Items (each pair paired up).

Reading Wedge Attributes in SOPs

In Houdini expressions on SOP nodes or in the Houdini Processor's parameters, Work Item attributes are accessible via:

detail('opinput:0', 'rock_seed', 0)   # in a VEX or SOP expression
@rock_seed                              # in an Attribute Wrangle
`@rock_seed`                            # in a string parameter (backticks)

Advanced Use Cases

Generating Procedural Asset Libraries

One of the most powerful production uses of TOPs is automatically generating asset libraries. Combining Wedge and Houdini Processor, you can generate hundreds of variations of the same asset overnight (rocks, trees, buildings, wreckage) with controlled random parameters, exported in the desired format with baked textures - what would take weeks to model by hand happens in a few hours of distributed computing.

Batch Simulation Pipelines

TOPs can launch dozens of simulations (fluids, RBD, pyro) with different initial conditions, gather the results, and compare them automatically - simulating 20 RBD destructions of the same building with different impact forces, exporting Alembic caches for each, and generating a preview render to pick the best variation to push to a full render.

Mass Texture Baking

Large-scale procedural baking is a common use case. TOPs can iterate over a list of assets (loaded via a File Pattern), run a Houdini Processor per asset that does UV unwrapping and map baking (normal, AO, roughness, height), and assemble it all into an organized folder structure - fully automating processing for hundreds of assets.

Distributed Rendering and Optimization

For animation projects, TOPs with an HQueue Scheduler distributes rendering frame by frame across a network of machines. But TOPs goes further: it can intelligently manage dependencies (simulate first, then render, then compress images, then assemble video), automatically retry failed frames, and send a notification when everything's done.

Integrating External Software

Via Shell Script and Python Script nodes, TOPs can orchestrate tools outside Houdini: calling Substance Painter to texture an asset, launching Blender for a specific operation, triggering Nuke scripts for compositing, or interacting with web APIs. TOPs becomes the conductor for an entire multi-software pipeline.

Attributes and Data Flow

Communication between TOP network nodes happens mainly through Work Item attributes. Understanding how attributes are created, passed and modified is essential for building robust TOP networks.

Attribute Types

  • Integer/Float/String - simple scalar values, the most common type for variation parameters.
  • File - a path to an input or output file; file-type attributes let Houdini automatically manage file dependencies.
  • @pdg_input / @pdg_output - special attributes holding a Work Item's input and output files, used by Houdini to build the dependency graph automatically.

Passing Attributes Along

By default, every TOP node inherits the incoming Work Item's attributes and can add new ones. The wedgeindex attribute (the Work Item's index within the Wedge, starting at 0) is particularly useful for uniquely naming output files:

$PDG_DIR/rock_v`@wedgeindex`_seed`@rock_seed`.abc

The $PDG_DIR variable is the PDG's working directory, configurable in the TOP Network's settings. $PDG_TEMP is a temporary directory Houdini manages automatically for intermediate files.

Best Practices

Organizing a TOP Network

  • Use Null nodes to name stages - as with SOP networks, name your milestones (OUT_wedge, OUT_sim, OUT_render) for readability and easy connection from other contexts.
  • Comment generously - complex TOP networks can get unreadable fast; comments (the C key) and node colors are essential.
  • Break into subnetworks - for complex pipelines, use TOP Subnetwork nodes to encapsulate functional groups of nodes.

Debugging Effectively

  • Test with 1 or 3 Work Items first - before launching 100 variations, temporarily cut the Wedge down to 1 or 3 iterations to confirm the network works.
  • Use the Task Graph Table - Windows > Task Graph Table to inspect every Work Item's attributes and logs.
  • Right-click > Open Failed Work Items - quickly see which Work Items failed and access their detailed logs.
  • Dirty and selective re-cook - right-click a node > Dirty > Dirty All Below to force recalculation only from a specific point in the network.

Performance and Network Paths

  • Use absolute UNC paths - for network-distributed rendering, every file path needs to be reachable from every machine (\\server\projects\...).
  • Prefer bgeo.sc over text formats - for intermediate caches, compressed bgeo.sc reads and writes much faster than ASCII formats.
  • Limit preview resolution - during the variation-exploration phase, render at low resolution (512×512) to iterate quickly, then re-render at full resolution once you've picked a variation.
  • Avoid local paths (C:\Users\) - they will never work on a distributed network's slave machines.

Resources

← Previous: Clouds, Atmosphere and Environment Lighting