NF-CONTRACT Contractualist / Rawlsian

Artificial Intelligence, Values, and Alignment · Iason Gabriel · 2020

“How are we to decide which principles or objectives to encode in AI—and who has the right to make these decisions—given that we live in a pluralistic world that is full of competing conceptions of value? Is there a way to think about AI value alignment that avoids a situation in which some people simply impose their views on others?”

why coded: Framing question is Rawlsian: who may decide under pluralism · unit #1, pp. 411

“both Kantian and contractualist moral theories require that an agent understand the concept of a 'reason' and subject it to certain kinds of hypothetical test before knowing how to proceed—capabilities that extend well beyond most existing forms of artificial agent”

why coded: Same capability claim made for contractualism (Scanlon) · unit #5, pp. 414

“Designing AI in accordance with a single moral doctrine would, therefore, involve imposing a set of values and judgments on other people who did not agree with them. For powerful technologies, this quest to encode the true morality could ultimately lead to forms of domination.”

why coded: Domination argument against encoding one true morality · unit #16, pp. 424

“Their agreement therefore takes the form of an 'overlapping consensus' between different perspectives (Rawls 2001, 32). Thus, even without agreement about the fundamental nature of morality, people may still come to a principled agreement about values and standards that are appropriate for a given subject matter or domain.”

why coded: Overlapping consensus defined and transposed from state to AI · unit #17, pp. 425

“The problem of alignment is, in this sense, political not metaphysical. To address it, I recommended that we look more closely at principles that would be supported by a global overlapping consensus of opinion, chosen behind a veil of ignorance and/or affirmed through democratic processes.”

why coded: Explicitly Rawlsian formulation · unit #24, pp. 436

Reinforcement Learning Under Moral Uncertainty · Adrien Ecoffet; Joel Lehman · 2021

“Principle of Proportional Say: Theories have Proportional Say if they are each allocated an equal voting budget and vote following the same cost structure, after which their votes are scaled proportionally to their credences. [...] the principle of Proportional Say suggests an algorithm we call Nash voting because it has Nash equilibria as its solution concept.”

why coded: Proportional Say / Nash voting - bargaining-theoretic compromise between theories inside one agent · unit #5, pp. 4

Disagreement, AI alignment, and bargaining · Harry R. Lloyd · 2024

“[Nash bargaining solution axioms:] 1. Scale invariance [...] 2. Pareto optimality [...] 3. Symmetry [...] 4. Independence of irrelevant alternatives. The NBS uniquely satisfies all four of these axioms. [...] One attractive feature of the NBS is that (all else being equal) it favours equal division of gains from trade between the bargainers.”

why coded: Nash bargaining as formalized contractualism: equal-gains compromise · unit #10, pp. 1778

Beyond Preferences in AI Alignment · Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal… · 2024

“Rather than learning humanity's preferences in order to maximally satisfy them, AI systems should be aligned with normative standards and criteria that we collectively forge and negotiate—standards exemplified by social, legal, and moral norms. [...] Just as AI assistants should avoid harmful language, self-driving cars should follow the rules of the road.”

why coded: Norm-negotiation as alignment target · unit #15, pp. 1846

“contractualist alignment aims to align AI systems with goals, standards, and principles that are mutually agreed upon by people despite our disparate preferences and values, deriving its normative force from the fair and impartial agreement of relevantly-situated rational actors. [...] AI goals and standards should be justified to each stakeholder, on grounds that none can reasonably reject. Insofar as these AI systems are used to exercise power over others, they should also act in accordance with standards that are not just fair, but legitimate.”

why coded: Scanlonian formulation: justifiable to each on grounds none can reasonably reject · unit #16, pp. 1847

A matter of principle? AI alignment as the fair treatment of claims · Iason Gabriel; Geoff Keeling · 2025

“efforts to justify a specific goal for AI alignment or set of AI decisions by reference to the correctness of a single moral theory, such as utilitarianism, seem destined to fail (Gabriel, 2020). [...] Doing so, would then involve building AI systems that exercise power over people in ways that they have good reason to reject – raising the spectre of both value imposition and domination.”

why coded: Value imposition/domination argument - the 2020 move sharpened · unit #5, pp. 1957

“[Against direct import of Scanlon:] contractualism is part of a fundamentally different type of enterprise: its goal is to provide an alternative to utilitarianism when it comes to answering questions about morality, not to provide a mutually acceptable justification of principles for people who hold different moral beliefs. [...] Because it purports to offer an account of interpersonal morality, moral claims are not allowed to feature directly in the process of evaluating principles.”

why coded: Scanlon adapted, not adopted - moral claims must be admissible · unit #7, pp. 1959

“[Against direct import of Rawls:] The explicit focus on citizens – and what citizens would do – does not pair well with the all-affected-principle as it applies to AI. Specifically, it could create significant gaps in terms of who principles for alignment are justified to, potentially excluding resident non-citizens or those who reside beyond national borders.”

why coded: Rawls adapted - all-affected principle replaces citizen focus · unit #8, pp. 1960

Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic ju… · Nick Schuster; Daniel Kilov · 2025

“We understand AI to be systemically impactful insofar as it not only stands to have significant impacts on human wellbeing but can also embed systemic biases into social systems and institutions [...] And we understand reasonable moral disagreement to be grounded in opposing moral worldviews which are, nonetheless, both internally coherent and compatible with basic liberal values [...] We follow John Rawls (2001) in taking this sort of 'reasonable pluralism to be a permanent condition' of diverse, modern societies.”

why coded: Rawlsian reasonable pluralism as permanent condition - the framing premise · unit #2, pp. 6074

Beyond Preference-based Value-alignment (IEAI Research Brief Q2 2026) · Julia Li · 2026

“Therefore, AIs should move beyond individual preference satisfaction and toward contractualist and situated ways of alignment that account for the preferences of multiple stakeholders. Rather than committing energy to aligning AI systems to what Zhi-Xuan et al. (2024) describe as commitments to universal preference-based standards, AI systems could be more like purpose-built tools.”

why coded: Contractualist, purpose-built, multi-stakeholder alignment as the proposed successor · unit #5, pp. 3