Learning to program by learning to read

In the age of deep networks, "learning" almost invariably means "learning from examples". Image classifiers are trained with large datasets of (labeled or unlabeled) images, machine translation systems with corpora of translated sentences, and robot policies with demonstrations. But when human learners acquire new concepts and skills, we often do so with richer supervision, especially in the form of language---we learn new concepts from exemplars accompanied by descriptions or definitions, and new skills from demonstrations accompanied by instructions. In natural language processing, recent years have seen a number of successful approaches to learning from task definitions and other forms of auxiliary language-based supervision. But these successes have been largely confined to tasks that also involve language as an input and an output. What will it take to make language-based training useful for other learning problems? In this talk, I'll present some recent results on using natural language to guide both search and library learning in inductive program synthesis, and discuss connections to the role of language in human concept learning.

Basics of Neurosymbolic Architectures

This part overviews the design of neurosymbolic architectures and their training. We will begin with an overview of conventional deep learning (i.e., purely neural architectures). Afterwards, we will introduce the concept of a domain specific language (DSL), which can include both symbolic and neural primitives. Any program or architecture can be designed using primitives from this DSL. We will then construct a few explicit neurosymbolic architectures and train their continuous parameters using gradient-based learning.

Bridging Data and Knowledge in Neuro-Symbolic Learning

The key challenge in neuro-symbolic machine learning is to bridge between perception from data and reasoning about symbolic knowledge. In this talk, I will present recent work that finds novel ways to unify these two worlds, by using logical and probabilistic reasoning tools (circuits and SAT solvers) to improve the learning capabilities of deep neural networks. In particular, I will discuss semantic loss functions to enforce constraints in structured output prediction, and counterexample-guided learning of monotonic neural networks.

Tutorial on Deductive Program Synthesis Part 1

This tutorial is an introduction to deductive program synthesis, using as an example synthesis of provably correct pointer-manipulating programs with the SuSLik system.

In the first part of the tutorial, we will introduce the motivation and main ideas behind deductive synthesis and practice specifying programs using formal logic.

Better learning through Programming Languages

This talk will describe the evolution of program synthesis into a tool for learning. The talk will describe the early work on synthesizing models from data and will then show how ideas in program synthesis can form the basis for new neurosymbolic learning algorithms.

Neurosymbolic Program Architecture Search

This part introduces two methods for automatically searching for the best program architecture within a DSL. The first method is a basic enumerative (or exhaustive) search. The second is RobustFill. For both methods, we will have hands-on exercises so that students can get a feel for how these methods work in practice. This part will conclude with a brief survey of other methods.

Probabilistic Programming Tutorial Part 1

Competitive Programming with AlphaCode

Programming is a powerful problem-solving tool. Systems that can assist programmers or even generate programs themselves could make programming more productive and accessible. Recent large-scale language models have demonstrated impressive abilities to generate code, however they still perform poorly on more complex tasks that require problem-solving skills, such as competitive programming problems. In this talk we'll present AlphaCode, the motivations of the project and the design decisions we made. AlphaCode is a system for code generation that achieved an average ranking of top 54.3% in simulated evaluations on popular, recent programming competitions on the Codeforces platform. AlphaCode's success stemmed from: large transformer-based models, using a novel combination of architectural, training, and prompting modifications; extensive datasets; efficient large-scale sampling; and filtering and clustering-based sample selection. This marks the first time an artificial intelligence system has performed competitively in programming competitions.

Moving Beyond the First Portrait of Our Milky Way’s Black Hole by Leveraging Underlying Structure

As imaging requirements become more demanding, we must rely on increasingly sparse and/or noisy measurements that fail to paint a complete picture. Computational imaging pipelines, which replace optics with computation, have enabled image formation in situations that are impossible for conventional optical imaging. For instance, the first black hole image -- published in 2019 of the black hole in the M87 galaxy -- and the recent second portrait of our Milky Way's supermassive black hole published in 2022, were only made possible through the development of computational imaging pipelines that worked alongside an Earth-sized distributed telescope. However, remaining scientific questions motivate us to improve this computational telescope to see black hole phenomena still invisible to us. This talk will describe how the first images of a black hole were computationally captured, and discuss how we are leveraging underlying structure in the data to both recover images without imposing human-designed image statistics, as well as extract the evolving structure of our own Milky Way's black hole over the course of a night in the future. These problems open up new challenges in neurosymbolic programming that could one day help to elucidate new astrophysics.

Neurosymbolic Program Architecture Search (continued)

This part introduces two methods for program architecture search. The first method is on using Admissible Neural Heuristics (NEAR) which treats program architecture search as an informed graph search problem. The second method is DreamCoder, which aims to learn extensions of the DSL (i.e., library learning). This part will conclude with a practical example of neurosymbolic learning for behavior analysis, grounded in real-world applications for behavioral neuroscience.

Understanding the Visual World Through Naturally Supervised Code

Much of our visual world is inherently symbolic: scenes are made of objects; different objects may have the same color or material, with a grid layout; each object can be symmetric and have repetitive parts. How can we infer, represent, and use such symbolic structure from raw data, without hampering the expressiveness of neural networks? In this talk, I will demonstrate that symbolic structure, or code, can be learned from natural supervision. Such supervision can be from pixels, where neuro-symbolic methods automatically discover repetitive parts and objects for scene synthesis. It can be from objects, where humans during fabrication introduce priors that can be leveraged by machines to infer regular intrinsics such as texture and material. It can also be from language, which humans develop for the need of referring to and talking about objects, and where machines may learn grounded visual concepts only by looking at images and reading captions. When solving these problems, symbolic programs and neural nets play complementary roles: symbolic programs are more data-efficient to train and generalize better to new scenarios, as they robustly capture high-level structure; deep nets effectively extract complex, low-level features from cluttered and noisy visual and language data.

Tutorial on Deductive Program Synthesis Part 2

This tutorial is an introduction to deductive program synthesis, using as an example synthesis of provably correct pointer-manipulating programs with the SuSLik system.

In the second part of the tutorial, we will see how deductive synthesis works under the hood. We will get hands-on experience deriving programs from specifications step-by-step using deductive rules and explore how deductive synthesis handles recursion.

Probabilistic Programming Tutorial Part 2