Understanding the World Through Code

Funded through the NSF Expeditions in Computing Program

Type-Directed Synthesis and Program Synthesis

Type-directed synthesis of visualizations from natural language queries chen2022type-ours

Program Synthesis for Robot Learning from Demonstrations patton2024program-ours

ImageEye: Batch Image Processing Using Program Synthesis imageeye-ours

Seq2seq Type Inference using Static Analysis wei2022typet5-ours

Code Transformation and Synthesis

Automated Transpilation of Imperative to Functional Code using Neural-Guided Program Synthesis mariano-ours

Semantic Code Refactoring for Abstract Data Types pailoor2024semantic-ours

Surveys

Neurosymbolic ProgrammingChaudhuriEPSSY21-ours

Neurosymbolic Programming for Science sun2022neurosymbolic-ours

Neurosymbolic Reasoning and Learning

LEMMA: Bootstrapping High-Level Mathematical Reasoning with Learned Symbolic Abstractions li2022lemma-ours

Neurosymbolic transformers for multi-agent communication inala2020neurosymbolic-ours

Contrastive Reinforcement Learning of Symbolic Reasoning Domains poesia2021contrastive-ours

Left to the Reader: Abstracting Solutions in Mathematical Reasoning poesia2022left-ours

Neurosymbolic reinforcement learning with formally verified exploration anderson2021neurosymbolic-ours

Neurosymbolic Reinforcement Learning

How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regression ma2022far-ours

Versatile offline imitation from observations and examples via regularized state-occupancy matching ma2022versatile-ours

Policy optimization with linear temporal logic constraints voloshin2022policy-ours

Eventual Discounting Temporal Logic Counterfactual Experience Replay voloshin2023eventual-ours

Program synthesis guided reinforcement learning for partially observed environments yang2021program-ours

Guiding Safe Exploration with Weakest Preconditions anderson2023guiding-ours

Computably Continuous Reinforcement-Learning Objectives Are PAC-Learnable yang2023computably-ours

Neurosymbolic Safety and Verification

Safe Neurosymbolic Learning with Differentiable Symbolic Execution safe-ours

Interpretable Machine Learning

Learning Differentiable Programs with Admissible Neural Heuristics near-ours

Unsupervised Learning of Neurosymbolic Encoders zhan2021unsupervised-ours

Counterfactual Explanations for Natural Language Interfaces tolkachev2022counterfactual-ours

Neurosymbolic Programming for Science sun2022neurosymbolic-ours

LILO: Learning Interpretable Libraries by Compressing and Documenting Code grand2023lilo-ours

Datasets

The multi-agent behavior dataset: Mouse dyadic social interactions sun2021multi-ours

The MABe22 Benchmarks for Representation Learning of Multi-Agent Behavior sun2022mabe22-ours

Vision

BKinD-3D: Self-Supervised 3D Keypoint Discovery from Multi-View Videos sun2022bkind-ours

Automatic Synthesis of Diverse Weak Supervision Sources for Behavior Analysis tseng2022automatic-ours

Self-Supervised Keypoint Discovery in Behavioral Videos sun2022self-ours

Interpreting Expert Annotation Differences in Animal Behavior annotation-ours

Few-Shot Image Classification: Just Use a Library of Pre-Trained Feature Extractors and a Simple Classifier Chowdhury_2021_ICCV-ours

Leveraging language to learn program abstractions and search heuristics wong2021leveraging-ours

Task programming: Learning data efficient behavior representations sun2021task-ours

Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beat hu2023federated-ours

Natural Language Processing

Parsel: A Unified Natural Language Framework for Algorithmic Reasoning zelikman2022parsel-ours

Certified Reasoning with Language Models poesia2023certified-ours

From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought wong2023word-ours

Program Induction

Combining Functional and Automata Synthesis to Discover Causal Reactive Programs Ria21aiplans-ours

Top-Down Synthesis for Library Learning stitch-ours

Neural Program Generation Modulo Static Analysis neurostatistic-ours

Library Learning

LILO: Learning Interpretable Libraries by Compressing and Documenting Code grand2023lilo-ours

DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning EllisWNSMHCST21-ours

Mathematical Reasoning

Pragmatic Code Autocomplete poesia2021pragmatic-ours

Geoclidean: Few-Shot Generalization in Euclidean Geometry hsu2022geoclidean-ours

Why think step by step? Reasoning emerges from the locality of experience prystawski2023think-ours

Scientific discovery

Synthesizing theories of human language with Bayesian program induction Ellis22Linguistics-ours

Neurosymbolic Programming for Science sun2022neurosymbolic-ours

Chemistry

Torsional diffusion for molecular conformer generation jing2022torsional-ours

Crystal diffusion variational autoencoder for periodic material generation xie2021crystal-ours

Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem trippe2022diffusion-ours

SE (3) diffusion model with application to protein backbone generation yim2023se-ours

Equibind: Geometric deep learning for drug binding structure prediction stark2022equibind-ours

Diffdock: Diffusion steps, twists, and turns for molecular docking corso2022diffdock-ours

MolScribe: Robust Molecular Structure Recognition with Image-to-Graph Generation qian2023molscribe-ours

Biology

Improved modeling of RNA-binding protein motifs in an interpretable neural model of RNA splicing gupta2023improved-ours

SPARLING: Learning Latent Representations with Extremely Sparse Activations gupta2023sparling-ours

Cognitive Science and Human-Computer Interaction

Why think step by step? Reasoning emerges from the locality of experience prystawski2023think-ours

Web question answering with neurosymbolic program synthesis webq-ours

Probability and Statistical Learning

A Probabilistic Framework for Modular Continual Learning valkov2023probabilistic-ours

Diffusion probabilistic modeling of protein backbones in 3D for the motif-scaffolding problem trippe2022diffusion-ours

Crystal diffusion variational autoencoder for periodic material generation xie2021crystal-ours

Torsional diffusion for molecular conformer generation jing2022torsional-ours

Formal Methods and Theorem Proving

A Language-Agent Approach to Formal Theorem-Proving thakur2023language-ours

Synthesizing trajectory queries from examples mell2023synthesizing-ours

Neurosymbolic Grounding for Compositional World Models sehgal2023neurosymbolic-ours

Generative Models and Differentiation

Auto-Differentiation of Relational Computations for Very Large Scale Machine Learning pmlr-v202-tang23a-ours

PFGM++: Unlocking the Potential of Physics-Inspired Generative Models pmlr-v202-xu23m-ours

Restart Sampling for Improving Generative Processes xu2023restart-ours

Language and Code

Sparse Learning and Representation

SPARLING: Learning Latent Representations with Extremely Sparse Activations gupta2023sparling-ours