VC-INTRA-VALUE Intra-value fragmentation

Reinforcement Learning Under Moral Uncertainty · Adrien Ecoffet; Joel Lehman · 2021

“The MEC approach is scale-sensitive [...] it is not at all clear how to find a scaling function under which such divergently-motivated theories as utilitarianism and deontology are resolved into a common scale. Indeed, it appears that these theories' judgments may be fundamentally incomparable.”

why coded: Util/deont incomparability at the implementation level - MEC scale-sensitivity · unit #4, pp. 3

Artificial Intelligence, Humanistic Ethics (Daedalus 151(2):232-243) · John Tasioulas · 2022

“This pluralism of values abandons the comforting notion that the key to the ethics of AI will be found in a single master concept, such as trustworthiness or human rights. How could human rights be the comprehensive ethical framework for AI when, for example, AI has a serious environmental impact beyond its bearing on anthropocentric concerns? [...] Being parasitic on compliance with more basic values, trustworthiness cannot itself displace those values.”

why coded: No master concept: rights and trustworthiness both fail as comprehensive frameworks · unit #1, pp. 235

“Beyond the pluralism of values is their incommensurability. [...] although some decisions will be superior to others, there may be no single decision that is optimal [...] This incommensurability calls into question the availability of some optimizing function that determines the single option that is, all things considered, most beneficial or morally right.”

why coded: Incommensurability defeats optimizing functions - eligible-alternatives structure · unit #2, pp. 235

STELA: a community-centred approach to norm elicitation for AI alignment · Stevie Bergman; Nahema Marchal; John Mellor; Shak… · 2024

“we also observe potential tensions between different rules specified by the participants, in both scope and specificity (e.g. when being harmless means potentially being less helpful). [...] Resolving these conflicts might require either further engagement with relevant communities or deference to other stakeholders, or both, to help with prioritisation.”

why coded: Elicited rules conflict (harmless vs helpful) - prioritisation unresolved · unit #11, pp. 11

Aesthetic Value and the AI Alignment Problem · Alice C. Helliwell · 2024

“solutions to the value alignment problem target all human values, not only morally relevant ones. Is there a value alignment problem in other domains? In this paper, I explore whether the AI value alignment problem extends [to aesthetic value].”

why coded: Alignment spans non-moral evaluative domains - aesthetic case (tentative) · unit #1, pp. 1

Democratizing value alignment: from authoritarian to democratic AI ethics · Linus Ta-Lun Huang; Gleb Papyshev; James K. Wong · 2024

“we can avoid a response that maximizes a particular value to the extreme, which typically leads to undesirable behaviors. Most importantly, this algorithm will allow for flexibility in assigning weights to different modules, adapting to diverse user value priorities and contexts.”

why coded: Anti-extremization: plural weighted constraints block single-value maximization (cf. Lundgren's loops) · unit #4, pp. 16

Disagreement, AI alignment, and bargaining · Harry R. Lloyd · 2024

“AI ethicists have proposed numerous prima facie plausible 'fairness criteria' for AI systems. Unfortunately, several impossibility theorems have recently demonstrated that no single AI system can jointly satisfy all of these criteria (Chouldechova, 2017; Corbett-Davies et al., 2017; Kleinberg et al., 2016 [...]). In the face of these impossibility theorems, it is inevitable that there will be social disagreement about what is required for fairness in AI systems.”

why coded: Fairness impossibility theorems: the value fragments into jointly unsatisfiable criteria · unit #2, pp. 1760

“MEC and MSEC are only applicable in cases where differences between the choiceworthinesses of the options available according to every moral theory in which the agent or group has credence can be measured on some common scale of value. Intertheoretic expected choiceworthiness is simply undefined in cases where unit comparisons are impossible. [...] imagine trying to compare absolutist deontology against scalar utilitarianism. These two different moral theories don't even use the same deontic categories.”

why coded: Theories with different deontic categories resist common scaling · unit #8, pp. 1767

Beyond Preferences in AI Alignment · Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal… · 2024

“if utility functions are used to represent aggregate value judgments, this effectively assumes that distinct human values are always commensurable in some way, and that our resulting preferences are always complete. Yet, as value pluralists argue, there are contexts where it seems hard or impossible to commensurate our values (Anderson, 1995), resulting in choices where our reasons run short, and we cannot say if one option is ultimately better than another (Chang, 1997).”

why coded: Incommensurability: utility representation assumes away plural values · unit #4, pp. 1824

“a good assistant is aware that some choices are hard, and some options may seem incomparable (Chang, 1997). When helping someone with such a choice, the assistant does not pretend to know which option is better, or try to optimize that person's life; instead, the assistant respects their autonomy, and empowers them to make the most informed choice possible [...] while ultimately remaining agnostic as to which choice is 'best'.”

why coded: Assistant must surface hard choices, not resolve them - incomparability respected in design · unit #12, pp. 1840

Learning the Value Systems of Societies from Preferences · Andrés Holgado-Sánchez; Holger Billhardt; Sascha … · 2025

“social science and humanities literature suggest that it is more adequate to conceive the value system of a society as a set of value systems of different groups, rather than as the simple aggregation of individual value systems. Accordingly, here we formalize the problem of learning the value systems of societies [...] The method learns socially shared value groundings and a set of diverse value systems representing a given society.”

why coded: Society's values = a SET of group value systems, not an aggregate - plurality as formal requirement · unit #1, pp. 1

“Defining human values and value-based preferences (or value systems) is a challenging task because values vary across time and cultures. In addition, at the time of acting, human preferences may be incomplete due to incommensurable values and context-specificity.”

why coded: Incommensurability + context-specificity conceded in an ECAI paper's framing · unit #2, pp. 1

Why human-AI relationships need socioaffective alignment · Hannah Rose Kirk; Iason Gabriel; Chris Summerfiel… · 2025

“the socioaffective perspective calls attention to intrapersonal dilemmas—such as how our goals, judgement and individual identities change due to prolonged interaction with AI systems.”

why coded: Intrapersonal dilemmas: the alignment target fragments WITHIN one person over time · unit #3, pp. 2

Normative conflicts and shallow AI alignment · Raphaël Millière · 2025

“A normative conflict arises when a given set of norms jointly demands that an agent a does 𝜙 and that a does 𝜓, in contexts where a cannot actually do both 𝜙 and 𝜓. In such cases, a must decide between doing 𝜙 or 𝜓, which inevitably involves a prima facie norm violation. [...] Helpfulness and harmlessness conflict when assisting the user is likely to cause harm.”

why coded: Normative conflict formalized: HHH norms conflict, forcing prima facie violation · unit #3, pp. 2044

Kantian deontology for AI: alignment without moral agency · Oluwaseun Damilola Sanwoolu · 2025

“Dancy and Anscombe maintain that the CI cannot tell us how to formulate maxims. They argue that the CI can only tell us how to determine the moral permissibility of maxims that have already been formulated, however, if we want AI systems to apply the FUL, we want it to know how maxims are formed.”

why coded: Particularist challenge: CI tests but cannot form maxims / capture context · unit #11, pp. 5430

Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic ju… · Nick Schuster; Daniel Kilov · 2025

“'RLHF is typically formulated as a solution for aligning an AI system with a single human, but humans are highly diverse in their preferences, expertise, and capabilities...Attempting to condense feedback from a variety of humans into a single reward model without taking these differences into account is thus a fundamentally misspecified problem. Moreover, current techniques model differences among evaluators as noise rather than potentially important sources of disagreement...As a result, when preferences differ, the majority wins, potentially disadvantaging underrepresented groups' (Casper and Davies et al., p. 9).”

why coded: Structured disagreement erased by aggregation - the intra/inter-value structure is invisible to the reward model (tentative) · unit #13, pp. 6082

Disentangling AI Alignment: A Structured Taxonomy Beyond Safety and Ethics · Kevin Baum · 2026

“achieving perfect conformity with alignment targets may be practically infeasible (when considered with respect to a specific standard X) or even logically impossible (when considered in an all-things-considered sense). [...] the various plausible standards—especially when drawn from different normative domains—may impose mutually exclusive requirements.”

why coded: All-things-considered alignment may be logically impossible across mutually exclusive standards · unit #8, pp. 168

Agents, Alignment, and the Many Faces of Autonomy · Roberta Fischli; Matija Franklin; Arianna Manzini… · 2026

“autonomy is a form of self-governance (Christman, 2009; Dagger, 2005), built around three conceptual components: non-interference, preferences, and abilities.”

why coded: Three-component anatomy that generates the fragmentation · unit #4, pp. 3

“Implicit to this discussion is a subtle but important insight: even a single value like autonomy is not a straightforward alignment target; it is a multifaceted concept we need to elaborate on before operationalizing it. [...] value alignment needs to navigate trade-offs not just across, but also within, individual values.”

why coded: The paper's core general lesson · unit #9, pp. 8

“This approach aims to give users meta-autonomy—the freedom to choose the kind of autonomy they want in a specific domain of the human-AI interaction, as expressed through different types of preferences. [...] An AI agent geared towards meta-autonomy would learn the difference between a person's preferences in different domains and tailor its response accordingly.”

why coded: Meta-autonomy: second-order solution to first-order fragmentation · unit #13, pp. 13

Agency and alignment: toward a normative architecture for human-AI interaction · Saša Josifović; Jörg Noller · 2026

“Our central thesis is that alignment does not require a machine's internalization of human values, not least because the very definition of 'human values' is exceptionally difficult. Human values are not static or universally given; they evolve historically and are often shaped by conflict and contestation. Instead, it demands the integration of machine behavior into human normative domains, where actions can be justified, evaluated, and controlled.”

why coded: Values historically evolving, conflict-shaped - not static parameters · unit #2, pp. 2

No value alignment without control · Björn Lundgren · 2026

“One possible interpretation is that the problem for the consequentialist theories considered so far is that they are monistic about values. The reasoning would be that having a singular intrinsic good makes a normative theory more sensitive to interpretations of that singular value [...] which then easily results in a simple-minded goal-satisfaction that is suboptimal.”

why coded: Value monism as the diagnosed weakness - motivates pluralism · unit #10, pp. 8

“a longer list just risks creating more potential for logical conflicts in the system, which then raises the question of how trade-offs ought to be resolved. Second, even if a list of capabilities does not contain the notion of control [...] such a list would not be successful in avoiding the problem [...] unless it included a notion of control that ensures sufficient human control.”

why coded: Longer capability lists just multiply trade-off conflicts - the weighing problem returns · unit #13, pp. 9

Understanding the Process of Human-AI Value Alignment · Jack McKinlay; Marina De Vos; Janina A. Hoffmann;… · 2026

“Value alignment is iterative. Values are highly sensitive to context, and operating contexts will change repeatedly throughout the system's lifetime. Even if the first version of an agent deployed is appropriately aligned, this will not last forever.”

why coded: Iterativity: context-sensitivity and value change make alignment non-final · unit #12, pp. 29

The value alignment problem in advisory AI: a systematic literature review · Loukas Triantafyllopoulos; Evgenia Paxinou; Diama… · 2026

“alignment should not be understood as a static mapping between human preferences and machine outputs but as a dynamic, context-dependent process. This view challenges earlier accounts that treated alignment as the encoding of fixed value sets and instead emphasizes negotiation shaped by interaction, memory, and adaptation.”

why coded: Dynamic/contextual alignment vs fixed-value encoding - SLR-level confirmation · unit #2, pp. 15