Main argument
Thesis: moral philosophy's moral-uncertainty framework (credences split across plausible ethical theories - MacAskill, Lockhart, Bostrom) can be translated into RL, yielding trainable agents that compromise between theories; doing so both curbs the extreme behavior of single-theory commitment and exposes technical complications philosophy hasn't faced. Argument type: formalization + experiments. Machinery: each theory i gets a choice-worthiness function Wi (reward-function analogue) and credence Ci; MEC (credence-weighted sum) works only for comparable theories and is scale-sensitive - 'it is not at all clear how to find a scaling function under which such divergently-motivated theories as utilitarianism and deontology are resolved into a common scale'; for incomparable theories they invoke a Principle of Proportional Say (influence proportional to credence, not scale) implemented as NASH VOTING - theories as competing sub-agents with voting budgets, Nash equilibrium as solution concept, trained by multi-agent RL; Arrow's impossibility forces a trade-off (Nash voting sacrifices Pareto efficiency). Key philosophical contribution: philosophy's 'options/possible worlds' framing ignores SEQUENTIAL decision-making - RL actions shape future morally-charged situations. Future directions flagged: agents revising their own credences ('machine meta-ethics'), eliciting a common scale from human judgment data (Riedener).
Why it matters here
The first serious ML implementation of MacAskill-style moral uncertainty - proof that multi-theory normative structures CAN be trained into RL agents, and honest documentation of where philosophy's machinery breaks on contact with sequential decision-making (incomparability, Arrow, scale-normalization). The technical existence proof behind the dissertation's claim that pluralist architectures are implementable, and the technical counterpart of Lloyd's formal analysis.
Reading notes
Close read of secs 1-4, 7-8 (28pp incl. appendices; experiments skimmed). Uber AI/OpenAI. Pre-LLM (2021, gridworld experiments) - pre-agentic in the dissertation's sense. Cited by Lloyd (whose Jackson/compromise concerns echo their voting analysis) and by the Lamsade PhD topic doc from the original reading-list search.
Ecoffet, A., & Lehman, J. (2021). Reinforcement Learning Under Moral Uncertainty. Proceedings of the 38th International Conference on Machine Learning, PMLR 139. arXiv:2006.04734
Close reading — 9 coded units
#1
· pp. 1
· claim
“recent work in moral philosophy proposes that ethical behavior requires acting under moral uncertainty, i.e. to take into account when acting that one's credence is split across several plausible ethical theories. This paper translates such insights to the field of reinforcement learning [...] The results illustrate (1) how such uncertainty can help curb extreme behavior from commitment to single theories.”
#2
· pp. 2
· argument
“proposals in moral philosophy typically do not explicitly consider the sequential nature of decision making. [...] in RL, an agent cannot directly bring about possible worlds but rather takes (often very granular) actions, which have long term effects both in the consequences that they bring about and in how they shape the ethically-charged decision situations an agent may encounter in the future.”
#3
· pp. 3
· definition
“Each theory also has a level of credence Ci, which represents the degree of belief that the agent (or the agent's designer) has in theory i. [...] Here we assume the credences of theories are set and fixed, e.g. by the system designer's beliefs, or by taking a survey of relevant stakeholders.”
#4
· pp. 3
· argument
“The MEC approach is scale-sensitive [...] it is not at all clear how to find a scaling function under which such divergently-motivated theories as utilitarianism and deontology are resolved into a common scale. Indeed, it appears that these theories' judgments may be fundamentally incomparable.”
#5
· pp. 4
· definition
“Principle of Proportional Say: Theories have Proportional Say if they are each allocated an equal voting budget and vote following the same cost structure, after which their votes are scaled proportionally to their credences. [...] the principle of Proportional Say suggests an algorithm we call Nash voting because it has Nash equilibria as its solution concept.”
#6
· pp. 4
· argument
“Arrow's impossibility theorem shows that any deterministic voting system which satisfies Pareto and IIA must be a dictatorship. [...] Thus, the standard approach in designing deterministic voting systems is to strategically break Pareto or IIA in a way that is least detrimental to the particular use case.”
#7
· pp. 8
· argument
“we hypothesize that impossibility results imply a spectrum of plausible algorithms that cover the trade-offs among competing desiderata in decision-making under moral uncertainty. Which algorithm is most appropriate for a given domain may depend on particularities of the competing theories and the domain itself.”
#8
· pp. 8
· argument
“An alternative approach would assume that finding such a common scale is not impossible but merely difficult. Such a research program could seek to elicit a common scale from human experts, either by requesting choice-worthiness values directly, or by having humans suggest the appropriate action under moral uncertainty in different situations and inferring a common scale from that data.”
#9
· pp. 9
· gap
“a final ambitious direction for future work is to explore mechanisms through which an agent can itself update its credences in moral theories (or derive new ones). That is, what might provide a principled foundation for machine meta-ethics?”