Lots of Links

Fellowships, Internships, Training Programs, Study Guides, etc

• AISafety.com Events and Training – a calendar showing all upcoming programs and other events

Programs

• ML Alignment Theory Scholars Program (MATS)

• AI Safety Camp (AISC) – online, part-time research program.

• Principles of Intelligent Behavior in Biological and Social Systems (PIBBSS) 

• Supervised Program for Alignment Research (SPAR)

• Swiss Existential Risk Initiative (CHERI), Research Fellowship 

• The Existential Risk Alliance (ERA), Cambridge Fellowship

• Center for Human-Compatible AI (CHAI), Research Fellowship, Collaborations, Internships

• Center for the Governance of AI (GovAI), Research Fellows, Summer and Winter Fellowships

• LawAI's Summer Research Fellowship

• Center for AI Safety (CAIS), Philosophy Fellowship

• Global Challenges Project 

• AI Safety Intro Fellowship, Cambridge AI Safety Hub 

• ML for Alignment Bootcamp (MLAB) – not currently running, but you can sign up for news on future iterations or request access to their curriculum

• Institute for AI Policy and Strategy (IAPS) Fellowship 

Self studies 

• • The Library — AI Alignment Forum 

  • AI Safety Fundamentals

    • Self Study Curriculum: Alignment

    • Self Study Curriculum: Governance

  • Introduction to ML Safety – Self Study Curriculum

  • Alignment Research Engineer Accelerator (ARENA) – follow links from each module to find full curriculum

  • Levelling Up in AI Safety Research Engineering by Gabriel Mukobi

  • Study Guide by John Wentsworth

  • Agent Foundations for Superintelligence-Robust Alignment (AFSRA) by plex

• Alignment reading list by Lucius Bushnaq

• Developmental Interpretability - Resources 

• MAISI Wi23 Alignment Fundamentals Guide

• Reading Guide for the Global Politics of Artificial Intelligence

• Description of Refine, including initial reading list by Adam Shimi 

• Introduction to AI Safety, Ethics, and Society textbook + lectures by Dan Hendrycks

• "Key Phenomena in AI Risk" - Reading Curriculum  Principal Curator: TJ (particlemania)

  • AI Safety Papers by Arkose 

  • Textbook: AISF Alignment Express [Comment Mode]

See AISafety.com Courses for more

Old self study guides (pre-LLMs)

• • CHAI's Bibliography

  • 80k's AI safety syllabus 

  • MIRI’s Research Guide (despite the name it is actually more of a study guide) 

  • Open AI’s Spinning Up in Deep RL (teaches you AI and not AI safety, but still useful)

Other lists

• List of AI Safety Technical Courses, Reading Lists, and Curriculums from Nonlinear  

• List of some university courses on X-risk | Pablo's miscellany <- Not AI Safety specific

• Top US policy master's programs <- Not AI Safety specific

• Resource Center | Arkose 

Job listings

• Posts tagged "hiring" on LessWrong 

• 80,000 Hours Job Board WARNING: This job board includes listings of jobs at AI capabilities labs such as OpenAI. Please don't apply for these jobs. Even so called safety roles at these labs should not be assumed to be good places to work for someone who cares about AI safety.

• EA Opportunity Board

• Many job openings are posted in #opportunities in the AI Alignment Slack

Mental Health Support

• Mental Health and the Alignment Problem: A Compilation of Resources (updated April 2023) 

• Shay Gestal – Health Coach

• EA Mental Health Navigator

• Rethink Wellbeing

• The Replacing Guilt series by Nate Soares

Research Agendas

Technical AI safety

• MIRI: Agent Foundations for Aligning Machine Intelligence with Human Interests (2017) and Alignment for Advanced Machine Learning Systems research agendas

• CLR: Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda (+ includes some questions related to AI governance)

• Paul Christiano’s research agenda summary (and FAQ and talk) (2018)

• Synthesising a human's preferences into a utility function (example use and talk), Stuart Armstrong, (2019), 

• The Learning-Theoretic AI Alignment Research Agenda, Vanessa Kosoy, (2018)

• Research Priorities for Robust and Beneficial Artificial Intelligence, Stuart Russell, Daniel Dewey, Max Tegmark, (2016) 

• Concrete problems in AI Safety, Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, (2016)

• AGI Safety Literature Review, Tom Everitt, Gary Lea, Marcus Hutter, (2018)

• AI Services as a Research Paradigm, Vojta Kovarik, (2020)

• Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems, Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar, (2020)

• AI Research Considerations for Human Existential Safety (ARCHES), Andrew Critch, David Krueger, (2020)

• How do we become confident in the safety of a machine learning system? By Evan Hubinger

• Research Agenda: Using Neuroscience to Achieve Safe and Beneficial AGI by Steve Brynes

• Unsolved Problems in ML Safety By Dan Hendrycks, Nicholas Carlini, J. Schulman, J. Steinhardt

• Foundational Challenges in Assuring Alignment and Safety of Large Language Models, Usman Anwar, et al.

AI governance

• AI Impacts: promising research projects and possible empirical investigations

• Governance of AI program at FHI: Alan Dafoe's AI governance research agenda

• Center for a New American Security: Artificial Intelligence and Global Security Initiative Research Agenda

• FLI: A survey of research questions for robust and beneficial AI (+ some aspects also fall into technical AI safety)

• Luke Muehlhauser’s list of research questions to improve our strategic picture of superintelligence (2014)

AI Safety Research Groups

There are many academic and independent researchers who are interested in AI safety and are not covered by this list. We are not going to list specific individuals publicly, so please contact us if you want to find more AI safety researchers.

Technical AI safety

• Machine Research Intelligence Institute (MIRI), Berkeley

• Center for Human-Compatible Artificial Intelligence (CHAI), University of California, Berkeley

• Future of Humanity Institute (FHI), University of Oxford

• Center on Long-Term Risk (CLR), London

• Ought, San Francisco 

• Alignment Research Center

• Redwood Research, Berkeley

• Conjecture

• Aligned AI

• FAR AI

• Apollo Research 

• Apart Research

• Center for AI Safety

• Model Evaluation and Threat Research (METR) 

• Alignment Research Center (ARC)

• AE Studio

• Orthogonal (orxl) 

AI governance

• The Center for the Study of Existential Risk (CSER), University of Cambridge

• Future of Humanity Institute (FHI), University of Oxford

• Global Catastrophic Risk Institute (GCRI), various locations

• Median Group, Berkeley 

• Center for Security and Emerging Technology (CSET), Washington

• Centre for the Governance of AI

• AI Standards Lab 

• Rethink Priorities 

Forecast and strategy

• Future of Humanity Institute (FHI), University of Oxford

• Convergence Analysis

• Leverhulme Center for the Future of Intelligence (CFI), University of Cambridge

• AI Impacts

• Metaculus

• Epoch

Organisation working on both technical safety and capabilities

• Anthropic, San Francisco

• OpenAI

• Obelisk

• DeepMind, London

• Cohere AI, Toronto

Funding

Grants 

• The Center on Long-Term Risk Fund (CLR Fund) – S-risk focused

• Long-Term Future Fund (LTFF)

• Survival and Flourishing Fund (SFF)  – awards and facilitates grants to existing charities

• Career development and transition funding | Open Philanthropy 

• Open Philanthropy Undergraduate Scholarship | Open Philanthropy 

• Future of Life Institute - Grants

• CEA Group Support Funding

• AI Safety: Neuro/Security/Cryptography/Multipolar Approaches | Foresight Institute 

Fundraising platforms

• Manifund 

• GiveWiki

Housing

• CEEALAR / EA Hotel – a group house in Blackpool, UK which provides free food and housing for people working on Effective Altruist projects (including AI safety)

• EA Housing by Nonlinear

Other lists

• AISafety.com Funders

• List of historical donors

Other General Resource Lists 

• AISafety.com

• Resources – AI Safety Fundamentals

• Opportunities | Arkose