Learning to program by learning to read

In the age of deep networks, "learning" almost invariably means "learning from examples". Image classifiers are trained with large datasets of (labeled or unlabeled) images, machine translation systems with corpora of translated sentences, and robot policies with demonstrations. But when human learners acquire new concepts and skills, we often do so with richer supervision, especially in the form of language---we learn new concepts from exemplars accompanied by descriptions or definitions, and new skills from demonstrations accompanied by instructions. In natural language processing, recent years have seen a number of successful approaches to learning from task definitions and other forms of auxiliary language-based supervision. But these successes have been largely confined to tasks that also involve language as an input and an output. What will it take to make language-based training useful for other learning problems? In this talk, I'll present some recent results on using natural language to guide both search and library learning in inductive program synthesis, and discuss connections to the role of language in human concept learning.

Basics of Neurosymbolic Architectures

This part overviews the design of neurosymbolic architectures and their training. We will begin with an overview of conventional deep learning (i.e., purely neural architectures). Afterwards, we will introduce the concept of a domain specific language (DSL), which can include both symbolic and neural primitives. Any program or architecture can be designed using primitives from this DSL. We will then construct a few explicit neurosymbolic architectures and train their continuous parameters using gradient-based learning.

Program Synthesis Tutorial Part 1

In this first part of the tutorial, students will get an introduction to symbolic program synthesis and will gain some experience with existing synthesis tools.

Neurosymbolic Program Architecture Search

This part introduces two methods for automatically searching for the best program architecture within a DSL. The first method is a basic enumerative (or exhaustive) search. The second is RobustFill. For both methods, we will have hands-on exercises so that students can get a feel for how these methods work in practice. This part will conclude with a brief survey of other methods.

Probabilistic Programming Tutorial Part 1

Competitive Programming with AlphaCode

Programming is a powerful problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent large-scale language models have demonstrated impressive abilities to generate code, however they still perform poorly on more complex tasks that require problem-solving skills, such as competitive programming problems. In this talk we'll present AlphaCode, the motivations of the project and the design decisions we made. AlphaCode is a system for code generation that achieved an average ranking of top 54.3% in simulated evaluations on popular, recent programming competitions on the Codeforces platform. AlphaCode's success stemmed from: large transformer-based models, using a novel combination of architectural, training, and prompting modifications; extensive datasets; efficient large-scale sampling; and filtering and clustering-based sample selection. This marks the first time an artificial intelligence system has performed competitively in programming competitions.

Understanding the world with Code

This talk will describe how neurosymbolic program synthesis can help understand the world.

Neurosymbolic Program Architecture Search (continued)

This part introduces two methods for program architecture search. The first method is on using Admissible Neural Heuristics (NEAR) which treats program architecture search as an informed graph search problem. The second method is DreamCoder, which aims to learn extensions of the DSL (i.e., library learning). This part will conclude with a practical example of neurosymbolic learning for behavior analysis, grounded in real-world applications for behavioral neuroscience.

Understanding the Visual World Through Naturally Supervised Code

Much of our visual world is inherently symbolic: scenes are made of objects; different objects may have the same color or material, with a grid layout; each object can be symmetric and have repetitive parts. How can we infer, represent, and use such symbolic structure from raw data, without hampering the expressiveness of neural networks? In this talk, I will demonstrate that symbolic structure, or code, can be learned from natural supervision. Such supervision can be from pixels, where neuro-symbolic methods automatically discover repetitive parts and objects for scene synthesis. It can be from objects, where humans during fabrication introduce priors that can be leveraged by machines to infer regular intrinsics such as texture and material. It can also be from language, which humans develop for the need of referring to and talking about objects, and where machines may learn grounded visual concepts only by looking at images and reading captions. When solving these problems, symbolic programs and neural nets play complementary roles: symbolic programs are more data-efficient to train and generalize better to new scenarios, as they robustly capture high-level structure; deep nets effectively extract complex, low-level features from cluttered and noisy visual and language data.

Program Synthesis Tutorial Part 2

In this first part of the tutorial, students will get an introduction to symbolic program synthesis and will gain some experience with existing synthesis tools.

Probabilistic Programming Tutorial Part 2