Lots of Links

AI Safety is a new and fast growing research field. This means that no overview of the field will be accurate for very long, and everything is a bit of a mess. If you are new to the field and feel overwhelmed, or don't know where to start, you are always welcome to reach out to one of us.

If you think something is wrong or missing in our lists, please let us know!

Career Advice

General

• Beneficial AI Research Career Advice by Adam Gleave

• 80k's Job Board

• FAQ: Advice for AI alignment researchers by Rohin Shah

PhD Advice

• Should you do a PhD? by Linda

• Deliberate Grad School by Andrew Critch

• A Survival Guide to a PhD by Andrej Karpathy

• There are more non-public resources for finding an AI Safety (or AI Safety friendly) PhD position. Contact Linda for more info.

Research Agendas

Technical AI safety

• MIRI: Agent Foundations for Aligning Machine Intelligence with Human Interests (2017) and Alignment for Advanced Machine Learning Systems research agendas

• CLR: Cooperation, Conflict, and Transformative Artificial Intelligence: A Research Agenda (+ includes some questions related to AI governance)

• Paul Christiano’s research agenda summary (and FAQ and talk) (2018)

• Synthesising a human's preferences into a utility function (example use and talk), Stuart Armstrong, (2019),

• The Learning-Theoretic AI Alignment Research Agenda, Vanessa Kosoy, (2018)

• Research Priorities for Robust and Beneficial Artificial Intelligence, Stuart Russell, Daniel Dewey, Max Tegmark, (2016)

• Concrete problems in AI Safety, Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané, (2016)

• AGI Safety Literature Review, Tom Everitt, Gary Lea, Marcus Hutter, (2018)

• AI Services as a Research Paradigm, Vojta Kovarik, (2020)

• Avoiding Negative Side Effects due to Incomplete Knowledge of AI Systems, Sandhya Saisubramanian, Shlomo Zilberstein, Ece Kamar, (2020)

• AI Research Considerations for Human Existential Safety (ARCHES), Andrew Critch, David Krueger, (2020)

AI governance

• AI Impacts: promising research projects and possible empirical investigations

• Governance of AI program at FHI: Alan Dafoe's AI governance research agenda

• Center for a New American Security: Artificial Intelligence and Global Security Initiative Research Agenda

• FLI: A survey of research questions for robust and beneficial AI (+ some aspects also fall into technical AI safety)

• Luke Muehlhauser’s list of research questions to improve our strategic picture of superintelligence (2014)

AI Safety Research Groups

There are many academic and independent researchers, who are interested in AI Safety, and who are not covered by this list. We are not going to list specific individuals publicly, so please contact us if you want to find more AI Safety researchers.

Technical AI safety

• Machine Research Intelligence Institute (MIRI), Berkeley

• Center for Human-Compatible Artificial Intelligence (CHAI), University of California, Berkeley

• Future of Humanity Institute (FHI), University of Oxford

• Center on Long-Term Risk (CLR), London

• Ought, San Francisco

AI governance

• The Center for the Study of Existential Risk (CSER), University of Cambridge

• Future of Humanity Institute (FHI), University of Oxford

• Global Catastrophic Risk Institute (GCRI), various locations

• Median Group, Berkeley

• Center for Security and Emerging Technology (CSET), Washington

AI companies which also does some safety work, both technical and governance

• OpenAI, San Francisco

• DeepMind, London

Meta, forecast and strategy

• Future of Humanity Institute (FHI), University of Oxford

• Convergence Analysis, moving around

• Leverhulme Center for the Future of Intelligence (CFI), University of Cambridge

• AI Impacts

• Metaculus

Funding

Currently open

• Long-Term Future Fund (LTFF)

  • The funds currently implement three grant rounds per year, with the following deadlines and decision dates:

    • 7 Mar 2021, decision by 2 Apr 2021

    • 13 Jun 2021, decision by 9 Jul 2021

    • 3 Oct 2021, decision by 29 Oct 2021

    • 6 Feb 2022, decision by 4 Mar 2022

    • 5 Jun 2022, decision by 1 Jul 2022

    • 2 Oct 2022, decision by 28 Oct 2022

  • All dates are at 11:59pm, Pacific Time. Please note that these dates are subject to change up to a month before the application deadline.

  • In addition, the Long-Term Future Fund can evaluate time-sensitive applications more quickly.

• The Center on Long-Term Risk Fund (CLR Fund)

  • Applications are currently open on an ongoing basis

• CEEALAR / EA Hotel is group house in Blackpool, UK which provides free food and housing for people working on Effective Altruist projects (including AI Safety), for up to two years

  • Applications are currently open on an ongoing basis

• Survival and Flourishing Fund (SFF) awards and facilitates grants to existing charities. For smaller grants and fiscal sponsorship for long-termist projects that don't yet have an institutional home, visit SAF.

  • Application due date: March 13, 2021 (11:59pm Pacific Time)

Currently closed

• Open Phil AI Fellowship is a fellowship for full-time PhD students focused on artificial intelligence or machine learning.

• Open Phil's Early-career funding for individuals interested in improving the long-term future

• Survival and Flourishing (SAF) awards small grants and service contracts for long-termist projects that don't yet have an institutional home. For larger grants to existing charities, visit SFF