Lots of Links

Fellowships, Internships, Training programs, etc

• AI Safety Training <- A calendar showing all upcoming programs and other events.

• AGI Safety Fundamentals

  • Self Study Curriculum: AI Aliment 101

  • Self Study Curriculum: AI Aliment 102

  • Self Study Curriculum: AI Governance

  • Apply to join the official program

• Introduction to ML Safety 

  • Self Study Curriculum

• SERI ML Alignment Theory Scholars Program (SERI-MATS)

• Alignment Research Engineer Accelerator (ARENA)

• AI Safety Camp (AISC) <- Online, part-time research program.

• Swiss Existential Risk Initiative (CHERI), Research Fellowship 

• The Existential Risk Alliance (ERA), Cambridge Fellowship

• Center for Human-Compatible AI (CHAI), Research Fellowship, Collaborations, Internships

• Center for the Governance of AI (GovAI), Research Fellows, Summer and Winter Fellowships

• Legal Priorities Project, Summer Research Fellowship in Law & AI

• Center for AI Safety (CAIS), Philosophy Fellowship

• ML for Alignment Bootcamp (MLAB) <- Not currently running, but you can sign up for news on future iterations or request access to their curriculum.

• Top US policy master's programs <- Not AI Safety specific.

News and Community

Other more complete lists

• AI Safety Communities and University groups  by Alignment Ecosystem Development

• Local and online AI safety communities by AI Safety Fundamentals

Newsletters

• AI Safety Newsletter by Center for AI Safety

• AI Safety Opportunities Newsletter by AI Safety Fundamentals 

• Subscribe for updates from AI Safety Training

Community Blogs

• AI Alignment Forum 

• Post tagged "AI" on LessWrong

• Post tagged "AI Safety" on EA Forum

Facebook

• AI Safety & Security

• Careers in AI Safety 

• AI Safety Discussion (Open) 

• AI Safety Discussion - "This group is primarily for people who have experience in AI/ML and/or are familiar with AI safety. We encourage beginners to join the AI Safety Discussion (Open) group."

Twitter

• AI Safety Core by JJ Balisanyuka-Smith

Other 

• AI Safety Google Group

• AI Alignment Slack group

• AI Existential Safety Community from FLI

• Apart Research

• AI Safety Initiative Groningen

Study Guides

Technical AI safety

• CHAI's Bibliography

• 80k's AI safety syllabus

• MIRI’s Research Guide (despite the name it is actually more of a study guide)

• Open AI’s Spinning Up in Deep RL

• Study Guide by John Wentsworth

• List of AI Safety Technical Courses, Reading Lists, and Curriculums from Nonlinear

• MAISI Wi23 Alignment Fundamentals Guide

AI governance

• Reading Guide for the Global Politics of Artificial Intelligence

Research Agendas

Technical AI safety

• MIRI: Agent Foundations for Aligning Machine Intelligence with Human Interests (2017) and Alignment for Advanced Machine Learning Systems research agendas

• CLR: Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda (+ includes some questions related to AI governance)

• Paul Christiano’s research agenda summary (and FAQ and talk) (2018)

• Synthesising a human's preferences into a utility function (example use and talk), Stuart Armstrong, (2019), 

• The Learning-Theoretic AI Alignment Research Agenda, Vanessa Kosoy, (2018)

• Research Priorities for Robust and Beneficial Artificial Intelligence, Stuart Russell, Daniel Dewey, Max Tegmark, (2016) 

• Concrete problems in AI Safety, Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, (2016)

• AGI Safety Literature Review, Tom Everitt, Gary Lea, Marcus Hutter, (2018)

• AI Services as a Research Paradigm, Vojta Kovarik, (2020)

• Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems, Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar, (2020)

• AI Research Considerations for Human Existential Safety (ARCHES), Andrew Critch, David Krueger, (2020)

• How do we become confident in the safety of a machine learning system? By Evan Hubinger

• Research Agenda: Using Neuroscience to Achieve Safe and Beneficial AGI by Steve Brynes

• Unsolved Problems in ML Safety By Dan Hendrycks, Nicholas Carlini, J. Schulman, J. Steinhardt

AI governance

• AI Impacts: promising research projects and possible empirical investigations

• Governance of AI program at FHI: Alan Dafoe's AI governance research agenda

• Center for a New American Security: Artificial Intelligence and Global Security Initiative Research Agenda

• FLI: A survey of research questions for robust and beneficial AI (+ some aspects also fall into technical AI safety)

• Luke Muehlhauser’s list of research questions to improve our strategic picture of superintelligence (2014)

AI Safety Research Groups

There are many academic and independent researchers, who are interested in AI Safety, and who are not covered by this list. We are not going to list specific individuals publicly, so please contact us if you want to find more AI Safety researchers.

Technical AI safety

• Machine Research Intelligence Institute (MIRI), Berkeley

• Center for Human-Compatible Artificial Intelligence (CHAI), University of California, Berkeley

• Future of Humanity Institute (FHI), University of Oxford

• Center on Long-Term Risk (CLR), London

• Ought, San Francisco 

• Alignment Research Center

• Redwood Research, Berkeley

• Anthropic, San Francisco

• AISafety.com - A Startup for Aligning Narrowly Superhuman Models

• Conjecture

• Aligned AI

• FAR AI

AI governance

• The Center for the Study of Existential Risk (CSER), University of Cambridge

• Future of Humanity Institute (FHI), University of Oxford

• Global Catastrophic Risk Institute (GCRI), various locations

• Median Group, Berkeley 

• Center for Security and Emerging Technology (CSET), Washington

• Centre for the Governance of AI

AI companies which also does some safety work, both technical and governance

• OpenAI, San Francisco

• DeepMind, London

• Cohere AI, Toronto

Meta, forecast, and strategy

• Future of Humanity Institute (FHI), University of Oxford

• Convergence Analysis, moving around

• Leverhulme Center for the Future of Intelligence (CFI), University of Cambridge

• AI Impacts

• Metaculus

Funding

• Nonlinear Network

• The Center on Long-Term Risk Fund (CLR Fund) 

• Long-Term Future Fund (LTFF)

• Survival and Flourishing Fund (SFF) 

  • awards and facilitates grants to existing charities.

• Survival and Flourishing (SAF) 

  • awards small grants and service contracts for long-termist projects that don't yet have an institutional home.

• Open Phil's Early-career funding for individuals interested in improving the long-term future

• Open Philanthropy undergraduate scholarship

• Open Phil AI Fellowship 

• Future of Life Institute - Grants

• ACX Grants by Slate Star Codex

• Future Funding List

• AI Safety Impact Markets

• List of historical donors

Community building related

• CEA Group Support Funding

• CGP Fast Funding

Housing

• CEEALAR / EA Hotel is group house in Blackpool, UK which provides free food and housing for people working on Effective Altruist projects (including AI Safety), for up to two years

• EA Housing by Nonlinear