Lots of Links
AI Safety is a new and fast-growing research field. This means that no overview of the field will be accurate for very long, and everything will be a bit of a mess. If you are new to the field and feel overwhelmed or don't know where to start, you are always welcome to reach out.
If you think something is wrong or missing from our lists, there is a suggestion form at the bottom of the page.
Contents
Fellowships, Internships, Training etc
News and Community
Alignment Newsletter by Rohin Shah
ML Safety Newsletter by Dan Hendrycks
AI Safety Discussion - "This group is primarily for people who have experience in AI/ML and/or are familiar with AI safety. We encourage beginners to join the AI Safety Discussion (Open) group."
AI Safety Core by JJ Balisanyuka-Smith
Other
Career Advice and Job Search
General Career Advice
How to pursue a career in technical AI alignment By Charlie Rogers-Smith
FAQ: Advice for AI alignment researchers by Rohin Shah
Beneficial AI Research Career Advice by Adam Gleave
PhD Advice
Should you do a PhD? by Linda
Leveraging Academia and Deliberate Grad School by Andrew Critch
A Survival Guide to a PhD by Andrej Karpathy
There are more non-public resources for finding an AI Safety (or AI Safety friendly) PhD position. Contact Linda for more info.
How to Write an Email to a Potential Ph.D. Advisor/Professor
Jobs
#EAjobs on Twitter
Other
Study Guides
Technical AI safety
MIRI’s Research Guide (despite the name it is actually more of a study guide)
Study Guide by John Wentsworth
List of AI Safety Technical Courses, Reading Lists, and Curriculums from Nonlinear
AI governance
Research Maps and Reviews
Research Agendas
Technical AI safety
MIRI: Agent Foundations for Aligning Machine Intelligence with Human Interests (2017) and Alignment for Advanced Machine Learning Systems research agendas
CLR: Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda (+ includes some questions related to AI governance)
Paul Christiano’s research agenda summary (and FAQ and talk) (2018)
Synthesising a human's preferences into a utility function (example use and talk), Stuart Armstrong, (2019),
The Learning-Theoretic AI Alignment Research Agenda, Vanessa Kosoy, (2018)
Research Priorities for Robust and Beneficial Artificial Intelligence, Stuart Russell, Daniel Dewey, Max Tegmark, (2016)
Concrete problems in AI Safety, Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, (2016)
AGI Safety Literature Review, Tom Everitt, Gary Lea, Marcus Hutter, (2018)
AI Services as a Research Paradigm, Vojta Kovarik, (2020)
Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems, Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar, (2020)
AI Research Considerations for Human Existential Safety (ARCHES), Andrew Critch, David Krueger, (2020)
How do we become confident in the safety of a machine learning system? By Evan Hubinger
Research Agenda: Using Neuroscience to Achieve Safe and Beneficial AGI by Steve Brynes
Unsolved Problems in ML Safety By Dan Hendrycks, Nicholas Carlini, J. Schulman, J. Steinhardt
AI governance
AI Impacts: promising research projects and possible empirical investigations
Governance of AI program at FHI: Alan Dafoe's AI governance research agenda
Center for a New American Security: Artificial Intelligence and Global Security Initiative Research Agenda
FLI: A survey of research questions for robust and beneficial AI (+ some aspects also fall into technical AI safety)
Luke Muehlhauser’s list of research questions to improve our strategic picture of superintelligence (2014)
Books, papers, podcasts, videos
(Non exhaustive list of AI Safety material)Books
The Alignment Problem by Brian Christian, 2020
Human Compatible by Stuart Russell, 2019
Reframing Superintelligence by Eric Drexler, 2019
Artificial Intelligence Safety and Security, By Roman Yampolskiy, 2018
Superintelligence by Nick Bostrom, 2014
Other reading
Victoria Krakovna's AI safety resources (contains a list of motivational resources and key papers for some AI Safety sub fields)
List of AI safety courses and resources from Nonlinear
Pragmatic AI Safety by ThomasWoodside
X-Risk Analysis for AI Research by Dan Hendrycks, Mantas Mazeika
Podcasts
Alignment Newsletter Podcast (Robert Miles reads the Alignment Newsletter)
80k's Podcast (Effective Altruism podcast with some AI Safety episodes)
Quinn’s Technical AI Safety Podcast
The Nonlinear Library - a repository of text-to-speech content from the EA Forum, Alignment Forum, LessWrong, and other EA blogs
YouTube
(Most of these channels are a mix of AI Safety content and other content)Robert Miles discuss AI on Computerphile and Robert Miles's own YouTube channel
SlateStarCodex Meetups (recorded talks)
Other Videos
AI Safety Research Groups
(Many of these groups do a combination of AI Safety and other X-risks.)There are many academic and independent researchers, who are interested in AI Safety, and who are not covered by this list. We are not going to list specific individuals publicly, so please contact us if you want to find more AI Safety researchers.
Technical AI safety
Center for Human-Compatible Artificial Intelligence (CHAI), University of California, Berkeley
Future of Humanity Institute (FHI), University of Oxford
Center on Long-Term Risk (CLR), London
Ought, San Francisco
Redwood Research, Berkeley
Anthropic, San Francisco
AISafety.com - A Startup for Aligning Narrowly Superhuman Models
AI governance
The Center for the Study of Existential Risk (CSER), University of Cambridge
Future of Humanity Institute (FHI), University of Oxford
Global Catastrophic Risk Institute (GCRI), various locations
Median Group, Berkeley
Center for Security and Emerging Technology (CSET), Washington
AI companies which also does some safety work, both technical and governance
Meta, forecast, and strategy
Future of Humanity Institute (FHI), University of Oxford
Convergence Analysis, moving around
Leverhulme Center for the Future of Intelligence (CFI), University of Cambridge
Other AI Safety Orgs and Initiatives
AI Safety Support (AISS) <- That's us!
AI Safety Camp (AISC), collaborate with a research mentor during intensive co-working sprints
ML for Alignment Bootcamp (MLAB), three weeks of intense learning in Berkeley
Future of Life Institute (FLI), outreach, podcast and conferences
Berkeley Existential Risk Initiative (BERI), supports university groups
Stanford Existential Risk Initiative (SERI), student and faculty group focused on reducing x-risks
Swiss Existential Risk Initiative (CHER), focused on mitigating global catastrophic risk
AGI4ALL, support for AGI experts
Nonlinear, search for and support high impact strategies to reduce existential and suffering risks
Existential Risk Observatory, collects and spreads information about existential risks
Lightcone Infrastructure building tech, infrastructure and community
Funding
Survival and Flourishing Fund (SFF)
awards and facilitates grants to existing charities.
Survival and Flourishing (SAF)
awards small grants and service contracts for long-termist projects that don't yet have an institutional home.
Open Phil's Early-career funding for individuals interested in improving the long-term future
ACX Grants by Slate Star Codex
Community building related
Housing
CEEALAR / EA Hotel is group house in Blackpool, UK which provides free food and housing for people working on Effective Altruist projects (including AI Safety), for up to two years