  Node graph systems are used ubiquitously for material design in computer
graphics. They allow the use of visual programming to achieve desired effects
without writing code. As high-level design tools they provide convenience and
flexibility, but mastering the creation of node graphs usually requires
professional training. We propose an algorithm capable of generating multiple
node graphs from different types of prompts, significantly lowering the bar for
users to explore a specific design space. Previous work was limited to
unconditional generation of random node graphs, making the generation of an
envisioned material challenging. We propose a multi-modal node graph generation
neural architecture for high-quality procedural material synthesis which can be
conditioned on different inputs (text or image prompts), using a CLIP-based
encoder. We also create a substantially augmented material graph dataset, key
to improving the generation quality. Finally, we generate high-quality graph
samples using a regularized sampling process and improve the matching quality
by differentiable optimization for top-ranked samples. We compare our methods
to CLIP-based database search baselines (which are themselves novel) and
achieve superior or similar performance without requiring massive data storage.
We further show that our model can produce a set of material graphs
unconditionally, conditioned on images, text prompts or partial graphs, serving
as a tool for automatic visual programming completion.
