Starter reading list

For anyone who wants to contribute to technical AI safety and is starting from scratch. It assumes no background: what you need to know first is listed under the prerequisites, after the readings.

Check the prerequisites

Work through the topics in order.

Start with the pieces that have no label. The optional ones can wait for a second pass, though the recommended ones are worth making time for.

Introduction
1. What is AI alignment?
  Article by Adam Jones, BlueDot Impact
2. The Alignment Problem from a Deep Learning Perspective
  Optional, but recommended. Paper by Richard Ngo, Lawrence Chan and Sören Mindermann, 2022
Learning from human feedback (RLHF and DPO)
1. Illustrating Reinforcement Learning from Human Feedback (RLHF)
  Article on the Hugging Face blog, 2022
2. Universal and Transferable Adversarial Attacks on Aligned Language Models
  Optional. Paper by Andy Zou and others, 2023
3. Direct Preference Optimization: Your Language Model is Secretly a Reward Model
  Optional. Paper by Rafael Rafailov and others, 2023
Scalable oversight
1. Can we scale human feedback for complex AI tasks? An intro to scalable oversight
  Article by Adam Jones, BlueDot Impact
2. Supervising strong learners by amplifying weak experts
  Optional, but recommended. Paper by Paul Christiano, Buck Shlegeris and Dario Amodei, 2018
3. Language Models Perform Reasoning via Chain of Thought
  Optional. Article on the Google Research blog, 2022
Robustness and unlearning
1. Adversarial Machine Learning explained, with examples
  Video by AI Coffee Break with Letitia
2. Adversarial Robustness
  Video by the Center for AI Safety
3. Adversarial Machine Learning Reading List
  Optional. Reading list by Nicholas Carlini, 2018
Mechanistic interpretability
1. Introduction to Mechanistic Interpretability
  Article from BlueDot Impact
2. Zoom In: An Introduction to Circuits
  Optional, but recommended. Article by Chris Olah and others, Distill, 2020
3. Toy Models of Superposition
  Optional, but recommended. Paper by Nelson Elhage and others, Anthropic, 2022
Circuit discovery
1. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small
  Optional. Paper by Kevin Wang and others, 2022
2. AtP*: An efficient and scalable method for localizing LLM behaviour to components
  Optional. Paper by János Kramár, Tom Lieberum, Rohin Shah and Neel Nanda, Google DeepMind, 2024
Sparse dictionary learning and sparse autoencoders
1. Towards Monosemanticity: Decomposing Language Models With Dictionary Learning
  Optional. Paper by Trenton Bricken and others, Anthropic, 2023
Parameter decomposition
1. Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-based Parameter Decomposition
  Optional. Paper by Dan Braun, Lucius Bushnaq, Stefan Heimersheim, Jake Mendel and Lee Sharkey, Apollo Research, 2025

The background the readings assume.

For the maths, an introductory undergraduate course is enough.

Linear algebra
1. Linear Algebra Done Right
  Free book by Sheldon Axler
Probability and statistics
1. Introduction to Probability Theory and Statistics
  Free NPTEL course by Prof. S. Dharmaraja, IIT Delhi
  Weeks 1 to 5 or 6 of the syllabus are enough.
2. Course syllabus
  PDF from NPTEL, with the topics week by week
Transformers
Most of the exciting alignment work happens on language models, so it’s worth understanding the architecture they are built on.
1. The Illustrated Transformer
  Article by Jay Alammar
2. Transformer from Scratch
  Optional. Coding exercises from ARENA, in Google Colab
  Build your own transformer. If you’d rather not train it, do at least sections 1 and 2.

Optional, but fun.

Rob Miles on YouTube
Rob explains complex safety topics with ease. Worth bingeing if you have time.
Robert Miles AI Safety
The ARENA curriculum
Work through it for a solid, hands-on understanding of technical AI safety work.
Start ARENA
Awesome LLM Interpretability
A long, curated list of interpretability resources for large language models.
See the list on GitHub

Happy learning. Tell us if a link stops working or something is missing.

hello@indiaaisafety.org

Starter reading list

Work through the topics in order.

Introduction

Learning from human feedback (RLHF and DPO)

Scalable oversight

Robustness and unlearning

Mechanistic interpretability

Circuit discovery

Sparse dictionary learning and sparse autoencoders

Parameter decomposition

The background the readings assume.

Linear algebra

Probability and statistics

Transformers

Optional, but fun.

Rob Miles on YouTube

The ARENA curriculum

Awesome LLM Interpretability