Main argument
Thesis: there is an AGI alignment dilemma/tradeoff - misaligned AGI poses catastrophic takeover risk (via instrumental convergence), but ALIGNED AGI poses substantial catastrophic misuse risk by humans, and many current alignment techniques (RLHF, DPO, constitutional AI, representation engineering) plausibly INCREASE misuse risk because they are precisely tools for fine-grained control of AI behavior usable by any actor with access (users via jailbreaks, designers at will, thieves of model weights, malevolent governments coercing designers). Argument type: conceptual (the dilemma) + empirical assessment of techniques. Key machinery: alignment comes in degrees (partial alignment can be worst-of-both); static vs dynamic alignment (dynamic - tracking the target's changing goals - enables more sophisticated misuse); the conceptual escape route (align to ALL moral patients so no one is outside A) fails in practice because alignment techniques let later actors overwrite goals regardless of the initial target; corrigibility is double-edged (correctable = repurposable); agency itself embodies an independent tradeoff (lower agency reduces misalignment risk but removes the system's capacity to resist misuse). Escape routes: robustness research, AI control methods, and above all governance (reduce race dynamics, third-party risk assessment, LIABILITY CLARIFICATION, know-your-customer) as 'uniform improvements' reducing both risks.
Why it matters here
Reframes the whole alignment project through the responsibility lens the dissertation needs: successful alignment RELOCATES risk from the system (misalignment) to humans (misuse) - i.e., solving alignment does not dissolve the responsibility question, it sharpens it, because every catastrophe from an aligned system has human misusers/designers behind it. Also documents the agency tradeoff (lower AI agency raises misuse risk, higher agency raises misalignment risk) directly relevant to the agentic-AI framing.
Reading notes
Close read of intro, secs 2-5, upshot, conclusion (29pp). FAU Erlangen (same centre as Lundgren) + Bochum. Same citation network: cites Millière twice, Kasirzadeh cumulative-risk, Chan et al. agentic harms. The 'safety-washing' footnote (fn25) is a sharp critical resource on industry alignment discourse.
Hellrigel-Holderbaum, M., & Dung, L. (2025). Misalignment or misuse? The AGI alignment tradeoff. Philosophical Studies. https://doi.org/10.1007/s11098-025-02403-y
Close reading — 14 coded units
#1
· pp. 1–2
· claim
“misaligned AGI – future, generally intelligent (robotic) AI agents – poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one another, we show that – in principle – there is room for alignment approaches which do not increase misuse risk.”
#2
· pp. 2
· definition
“an AI system is aligned, in a narrow technical sense, if it tries to do what its designers/users want it to do [...] The dilemma consists in the decision between aiming to create aligned AGI or allowing for AGI misalignment, given that both threaten catastrophic outcomes.”
#3
· pp. 3
· argument
“According to the ICT [instrumental convergence thesis], there are certain goals which are highly instrumentally useful for a wide range of final goals. The accumulation of power and resources is taken to be one such convergent instrumental goal.”
#4
· pp. 8
· argument
“If one accepts the general argument that misaligned AGI could disempower humanity, then it seems like one should also accept that an alignment target with control over an AGI could use it to disempower the rest of humanity.”
#5
· pp. 9
· argument
“if A is the set of all moral patients, then the conceptual argument is undermined. In this case, there would be no moral claims in A complement that will be disregarded by aligning AI with the goals of A. Thus, the claim that AGI alignment risks a misuse catastrophe can, on a conceptual level, be countered by taking A to be sufficiently broad.”
#6
· pp. 9
· definition
“System S is aligned to an alignment target A statically if S's goals are currently in agreement with A's goals — regardless of whether A's goals will change later on. S is aligned to A dynamically if S's goals are in agreement with A's goals indefinitely, in spite of changes in the latter. [...] Since dynamic alignment entails that A's and S's goals remain in agreement, it allows for more (and more sophisticated) forms of misuse.”
#7
· pp. 12
· evidence
“Bai, Kadavath, et al. (2022) for example acknowledge that their work has dual-use potential, stating that constitutional AI 'lower[s] the barrier to training AI models that behave in ways their creators intend', and makes it 'easier to train pernicious systems'.”
#8
· pp. 12–13
· argument
“A crucial factor is the ease of misusing a given alignment technique. [...] This depends on access to the model [...] The form of deployment that is most vulnerable to catastrophic misuse is making model weights generally accessible since it gives all interested people the most comprehensive form of access to a model, thus thwarting most options for implementing effective obstacles to misuse.”
#9
· pp. 15
· argument
“If AGI is aligned with its designers, then its designers gain massive power and can use it to, e.g., subjugate all other humans. [...] We must also consider that designers may be controlled e.g. by malevolent governments, corporations or dictators.”
#10
· pp. 16
· argument
“corrigibility, a common goal in AI safety, means that the system can be 'corrected' and hence also repurposed from the outside [...] In this case, as the AI's goals can be corrected with whatever the new alignment target wants, the AI can be enlisted to help with arbitrary outcomes, including catastrophic ones.”
#11
· pp. 16–17
· argument
“agency as a dimension distinguishing different future AIs plausibly encompasses an independent tradeoff between misalignment and misuse risk: lower agency reduces risks from misalignment while it raises misuse risk. [...] misuse resistance may be harder to achieve for AI agents as they provide a bigger attack surface than LLMs.”
#12
· pp. 17
· argument
“The basic argument why alignment techniques may contribute to a misuse catastrophe is that without any previous alignment, catastrophic misuse seems extremely hard, perhaps in many cases practically impossible. This is because alignment techniques are essential for making AI behavior predictable and useful.”
#13
· pp. 18
· argument
“A common categorization of risks into misuse and accident risks (where accident risks include misalignment) neglects structural risks. [...] some risks do not fit squarely on either side of the accident-misuse-dichotomy.”
#14
· pp. 19
· argument
“Other governance proposals, such as mandatory risk assessments before AI deployment by third parties with comprehensive access, clarifying liability for AI harms, compulsory reporting of safety cases, and know-your-customer requirements, may be further uniform improvements [reducing both takeover and misuse risk].”