Actionable Interpretability

It’s easy for the idea of ‘applied’ or ‘actionable’ research to limit our level of ambition. In this talk I want to push against this tendency and suggest that the most actionable interpretability will also be the highest ambition - if you’re fighting for fractional performance gains against black-box baselines, you’ve probably already lost. Instead, I want to propose thinking of ambitious actionable interpretability, which aims at actions that are either qualitatively harder or outright impossible without interpretability. The most exciting of these in my view is incorporating intelligence into the training process. I’ll present an overview of our recent work on this topic, discuss future directions and risks, and consider what priorities in interpretability research this direction raises.

Modern interpretability research has focused almost exclusively on the learned representations of trained models. Such analysis is well suited to scenarios in which the model is a given: one needs to understand, and try to control, a specific artifact. However, in many development scenarios, the model is not a given; we choose the data, architecture, and optimization protocol, and the interactions of these elements ultimately define the model and determine its representations. In this talk, I will argue (along with a chorus of recent voices) that interpretability should move upstream, into model development, and that this will benefit the entire field. Representational analysis tells us what a model has learned, and upstream analysis tells us why it learned those things and what we should do differently next time to achieve better outcomes.