Agents, Alignment, and the Many Faces of Autonomy
Roberta Fischli; Matija Franklin; Arianna Manzini; Iason Gabriel · 2026 · Minds and Machines interlocutor high priority coded
Main argument
Thesis: human autonomy is a desirable but internally fragmented alignment target - its components (non-interference, authentic preferences, ability to act) can be specified in conflicting ways, so aligning a personal AI agent 'with autonomy' requires choosing between rival interpretations. Argument type: conceptual analysis plus design-strategy construction. Offers three coherent strategies: the liberal approach (act strictly on stated preferences, anti-paternalist, tolerates self-harming choices), the capability-boosting approach (enhance declarative/procedural knowledge so users can act on their goals - Sen/Nussbaum-inflected, risks paternalism at the point of overriding explicit preferences), and the meta-autonomy approach (the user chooses, per domain, what KIND of autonomy the agent should promote, operationalized via meta-preferences and conversational tuning). Key general lesson: value alignment faces trade-offs not just ACROSS values but WITHIN a single value. Collective autonomy and republican non-domination are explicitly deferred as future research.
Why it matters here
The state-of-the-art of Gabriel's own research program six years on, now explicitly about AGENTIC systems (personal AI agents). Shows how the 2020 framework gets revised - and what it still cannot handle - once AI performs tasks without human oversight. The single most direct test of the dissertation's claim that pre-agentic alignment frameworks need revisiting.
Reading notes
Full close read completed. 24pp. Google/DeepMind-embedded philosophy (Fischli at Google Zurich/Stanford; other three at Google DeepMind). Companion pieces cited throughout: Gabriel et al. 2024 'Ethics of Advanced AI Assistants', Kirk et al. 2025 socioaffective alignment (in library), Zhi-Xuan et al. 2024 'Beyond Preferences' (NOT in library - acquisition target), Gabriel & Keeling 2025 'A Matter of Principle' (NOT in library - acquisition target).
Fischli, R., Franklin, M., Manzini, A., & Gabriel, I. (2026). Agents, Alignment, and the Many Faces of Autonomy. Minds and Machines, 36:34. https://doi.org/10.1007/s11023-026-09786-9
Close reading — 16 coded units
“Artificial intelligence (AI) systems are becoming more agentic. Capable of predicting, planning and executing actions, AI agents can increasingly perform tasks without human oversight (Knight, 2024; Russell & Norvig, 2016). [...] This puts AI agents at the frontier of AI ethics (Gabriel et al., 2024; Lazar, 2024).”
“As people enter increasingly interdependent relationships with AI agents, concerns about human disempowerment and loss of agency loom large (Kirk et al., 2025; Kosmyna et al., 2025; Kulveit et al., 2025). In particular, there is a worry that AI systems with enhanced capabilities will replace human participation across different domains and gradually erode people's ability to take charge of their own life.”
“Yet, research has not addressed the relationship between personal AI agents and human autonomy, and the methods to align them. In particular, we need to ask whether it is possible for a person to undergo deep and persistent interaction with a personal AI agent and remain as autonomous as—if not more autonomous than—they were before.”
“autonomy is a form of self-governance (Christman, 2009; Dagger, 2005), built around three conceptual components: non-interference, preferences, and abilities.”
“[Table: four types of preference - stated ('I want what I say I want'), revealed ('I want what my actions reveal I want'), informed ('I want what I would want if I had access to all the relevant information'), ideal ('I want what best promotes my objective interests') - each generating a distinct definition of what it is for an AI agent to enhance a person's autonomy]”
“Human preferences are not just integral to human autonomy; they also play an important role within contemporary AI alignment, as they are considered to be an effective way of representing human values and are easily operationalizable [...] This approach has its roots in rational choice theory and builds upon a model of the individual as a rational and well-informed actor [...] Yet, preferences are also context-specific, adaptive, and dynamic [...] And they can be misinformed, performative, or misdirected (Arneson, 1985; Gabriel, 2020; Sen, 1985).”
“Personal AI agents that promote autonomy via revealed preferences therefore face an epistemic challenge: They prioritize action over self-representation—engaging in a form of behaviorism (De Yong & Prey, 2022). The link between user behavior and authenticity is further complicated by the fact that preferences revealed in behavior could occur due to heuristics or habitual thinking [...] or artificially constrained or manipulated digital environments.”
“prioritizing ideal preferences assigned by an AI system also carries substantial risk, since it assumes that there is something that a person ought to want irrespective of their own position in the matter (Hayek, 1960/2011). Taken to the extreme, the user ceases to be the authority on what they truly prefer.”
“Implicit to this discussion is a subtle but important insight: even a single value like autonomy is not a straightforward alignment target; it is a multifaceted concept we need to elaborate on before operationalizing it. [...] value alignment needs to navigate trade-offs not just across, but also within, individual values.”
“Being strictly anti-paternalistic, the liberal approach posits that the AI agent should promote a user's stated preferences even when doing so reduces their well-being or undermines their autonomy over time. [...] The only reason an AI agent will refuse to act on a user's stated preferences is if acting on them would unequivocally inflict harm on others.”
“Another challenge is that a user may give a lot of instructions to their agents, which could reduce their own skills and critical thinking faculties, gradually eroding their autonomy (Kosmyna et al., 2025; Kulveit et al., 2025; Marchal et al., 2026). Finally, people may use their personal agent to do things that undermine their own objective interests in the name of autonomy, such as reckless gambling.”
“The key distinction is that capability-boosting requires voluntary user cooperation and active consent, whereas stronger forms of paternalism replace the user's independent judgment with what the user should want. Boosting tips into this stronger form of paternalism at the point where the agent begins overriding or bypassing explicit user preferences in favor of ideal ones, without seeking the user's endorsement or making its interventions transparent.”
“This approach aims to give users meta-autonomy—the freedom to choose the kind of autonomy they want in a specific domain of the human-AI interaction, as expressed through different types of preferences. [...] An AI agent geared towards meta-autonomy would learn the difference between a person's preferences in different domains and tailor its response accordingly.”
“Yet, we also need to look at the multi-party question and explore what happens, from the standpoint of collective autonomy, when a large number of people use agents that are aligned to their individual preferences (Hammond et al., 2025). Is it possible that these individual autonomy enhancers will limit the agency of others, or affect the autonomy of a society as a whole in unexpected ways?”
“One criterion the current framework does not address is non-domination. Republican accounts of freedom focus on arbitrary interference as a distinct threat to freedom that cannot be reduced to preference satisfaction or capability (Dagger, 2005; Pettit, 2012). [...] Assessing our proposed alignment strategies against the standard of non-domination therefore falls outside the scope of this paper.”
“Yet, there are also risks inherent to increased AI agency, including concerns about gradual disempowerment (Kulveit et al., 2025) or a loss of human agency as a result of cognitive offloading, automation, and manipulation (Koralus, 2025; Mitelut, Smith, & Vamplew, 2023).”
Synthesis-matrix row
supports
T2-PREFERENTISM-BROKEN
four preference types conflict; behaviorism critique
supports
T4-ROSSIAN-DEMAND
intra-value trade-offs demand weighing; meta-autonomy delegates it
supports
T7-AGENTIC-BREAKS-FRAMES
2026 flagship keeps dyadic frame; collective effects deferred
Memos (5)
comparison · unit #5
GABRIEL_2020 vs this paper: the 2020 six-loci taxonomy (instructions/intentions/revealed/informed/interests/values) reappears here compressed to four preference types, now indexed to autonomy rather than to alignment simpliciter. What changed: 'values' and 'interests' as separate loci have collapsed into 'ideal preferences' - i.e., the 2026 Gabriel program has become MORE preference-theoretic, not less, even while criticizing preferentism. What did NOT change: the dyadic one-person-one-agent frame. Unit 14 explicitly defers the multi-party question. So the dissertation's GAP-AGENTIC-UNTESTED claim survives its strongest possible counterexample: even the 2026 flagship paper on agentic alignment retains the pre-agentic dyadic architecture at the collective level.
theoretical · unit #9
Intra-value fragmentation (unit 9) is a gift to the dissertation's metaethics chapter: if even ONE value like autonomy dissolves into rival specifications requiring adjudication, then alignment necessarily involves the kind of conflict-resolution machinery Rossian pluralism provides (weighing competing prima facie considerations in context) - and proceduralism cannot help, because the fragmentation recurs INSIDE whatever target the fair procedure selects. The paper's own solution (meta-autonomy = let the user choose per domain) is effectively a delegation of the Rossian weighing task to the user, without noticing that this presupposes the user can do the weighing that the theory could not.
thesis-link · unit #10
Responsibility silence, all three approaches: liberal approach (unit 10) lets the agent execute self-harming stated preferences (reckless gambling, unit 11) - who bears responsibility for the resulting harm, the user qua principal, or the developer who chose the liberal design? Boosting (unit 12) has the agent shaping preferences - responsibility for instilled preferences? Meta-autonomy shifts the design choice to the user's meta-preference - a consent-based responsibility transfer that looks structurally like the 'consent laundering' concept from Augustine's comps essays. The paper never raises responsibility at all. This is the exact junction where the dissertation's responsibility-locus analysis (RL-* codes) adds something the Gabriel program lacks. Directly feeds the Work chapter (AI Interviewer = an agent acting on delegated authority in employment decisions).
comparison
Acquisition targets surfaced: Zhi-Xuan, Carroll, Franklin & Ashton (2024) 'Beyond Preferences in AI Alignment' (Phil Studies 182:1813-1863; arXiv 2408.16984 - appeared in earlier web search) and Gabriel & Keeling (2025) 'A Matter of Principle? AI alignment as the fair treatment of claims' (Phil Studies 182:1951-1973). Both are load-bearing citations here and neither is in AI-Align-Lit. The IEAI-TUM brief (queued in this batch) is reportedly built on the Zhi-Xuan line - check when coding it.
theoretical · unit #16
Empirical grounding pattern: the agentic-risk claims (units 2, 11, 16) rest on brand-new 2025-2026 empirical work (Kosmyna 'Your Brain on ChatGPT' cognitive debt; Fang et al. longitudinal chatbot study; Kulveit gradual disempowerment). The philosophy is tracking a moving empirical target - supports the dissertation's methodological stance that normative AI ethics now requires live empirical input, contra pure-armchair approaches. Useful for the methodology chapter's defense of the xphi corpus approach.