Browse Comments — Relevant (AI ∩ value)

TBF, I can't think of a single company I would trust to self-regulate on anything. AI just happens to be the worst offender right now.

Chris Olah saying publicly that competitive pressure, capital pressure, and geopolitical pressure push AI labs in directions that can conflict with doing the right thing — that's not a philosophical observation. That's a structural admission that external governance is load bearing, not optional. For healthcare organizations making AI procurement decisions right now, that statement should change how they evaluate vendor accountability. A vendor whose own co-founder acknowledges these incentive conflicts exists is a vendor whose contractual governance requirements, audit trail provisions, and human override protocols need to be airtight before a single patient record touches their system. The trust problem isn't theoretical. One of the architects of the technology just confirmed it from the Vatican. That's about as public as a warning gets.

This is the responsible thing to do, and other frontier labs and politicians should be lining up behind them. The climate change parallel is sobering. In both cases, the people closest to the science sounded the alarm while political and economic systems moved far too slowly. The difference with AI is the timescale. Climate change unfolded over decades, and AI disruption could compress that into years. Olah's candour is valuable precisely because it comes from inside. But candour without structural change is just confession.

From my understanding, AI should not replace people, rather it should free up people to do what they do best: solving the world's unique problems and exercising the freedom to innovate. I appreciate Olah's honesty. However, in my opinion, handing over AI governance to governments have the potential to do more harm than good. If any government turns against its own people, this technology can destroy humanity resulting in the manipulation of information and distortion of facts in global proportions. For me, the best option is to let the markets handle it. Like Healthcare systems who have Compliance and Cybersecurity governance to set up frameworks to safeguard patient information, enterprises who operate AI labs should have similar governance standards that keep innovation within bounds. It's not perfect but I prefer this option over trusting governments with governing information to this degree. Olah's right, people shouldn't be trusted. Much more so with governments, just saying.

Great to see Anthropic taking a lead here and consulting sources of wisdom to guide AI adoption across a broad ideological spectrum. The real question is, is it all fluff or are they going to back it up with serious business strategy and values?

This is the part that matters most for enterprise governance. The issue is not whether a particular lab is well-intentioned. Some clearly are. The issue is that good intentions do not neutralize the incentive structure. If frontier AI labs operate under commercial pressure, capital pressure, geopolitical pressure, and speed-to-market pressure, then enterprise buyers should not treat vendor assurances as a complete governance basis. That does not mean AI should not be adopted. It means adoption has to be authorized against present capability, disclosed limitations, model-change risk, output-shaping controls, and the actual conditions under which reliance is justified. Self-governance is not enough when the market is also selling the reliance. I wrote about this from the adoption-integrity side here: The market is selling the projection. The law has to make them show the machinery.

The danger is not that AI suddenly becomes anti-human. The danger is colder than that: institutions may use AI to make human needs machine-legible, then mistake that legibility for truth. Safety becomes risk scoring. Belonging becomes engagement management. Esteem becomes reputation analytics. Judgment becomes workflow compliance. Purpose becomes another managed service. That is when AI stops being a tool and becomes an operating theology. The machine does not need to hate humanity to reduce it. It only needs to optimize around the wrong definition of value.

I’ve sat in alignment meetings where an engineering team flagged a subtle vector bias in a model pipeline, only to be told by product marketing that delaying the rollout would cost us our quarterly enterprise contracts. That is the exact corporate bias Chris Olah is warning us about in the Reuters brief. You cannot let the entities chasing a multi-trillion-dollar commercial window be the sole auditors of their own systemic risk. When a tech giant's primary fiduciary duty is to Wall Street, "AI Safety" will always be downgraded from an architectural constraint to a public relations line-item. If we don’t fund and institutionalize independent, external cryptographic and behavioral validation outside the Silicon Valley monopoly, we aren't building a safe ecosystem. We are just renting a black box from companies that are incentivized to hide the telemetry logs when things go sideways.

Interesting point. External oversight matters, but I think the next challenge is what kind of working systems we build around AI itself. AI should optimize for what it does best: processing information and handling scale. Humans should be freed to do what humans do best: judgment, creativity, trust, and meaning. We have a unique opportunity right now to build systems where both operate in their strengths rather than creating environments where people slowly become operators inside an efficiency machine.

Mark Wagner 😎 📈 #Bentley #Asset #Management 💥 #BAM #Alert: 📢 MU, SNDK and DRAM are the definitive global AI memory-chip leaders: 💎 Micron Technology (MU) topped a $1 trillion market value for the first time on Tuesday as shares popped over 20%, driven by insatiable global artificial intelligence demand for memory chips. The stock surge came as UBS nearly #tripled its price target on the stock from $535 to $1,625 a share, citing "exponential growth and longer-term contract agreement opportunities with partially fixed pricing". Full details: cnb.cx/4uArkXO

Rand Strauss There is no alignment problem for LLM tech, so long as it makes money.

Evan Hunter Evan the nuclear analogy lands hard because it's exact. We didn't let private companies self-regulate fissile material and call it innovation. The distinction that made nuclear different was that governments understood the catastrophic downside before widespread deployment — not after. The governance infrastructure preceded the technology at scale. AI governance is running in reverse. The technology is deployed at scale while the governance infrastructure is still being argued about in congressional hearings. Your point about C-suite and board accountability preceding vendor accountability is right — the problem is most boards don't yet have the technical literacy to ask the right questions. Which creates a gap that vendors fill with their own risk framing. The Vatican moment matters because it's one of the architects saying publicly that the internal accountability structure is insufficient. That's the signal boards should be acting on right now.

Matthew P. Your point about AI governance running in reverse is well made. Thank you for clarifying. I would push back on the idea that the potential catastrophic downside risk inherent to AI was well understood prior to its use at-scale (by DARPA, The Pentagon, Science Fiction writers, Hollywood, etc). However, enough decision makers (public and private) were sold on the following: 1) massive upside of digital assistants (productivity, prosperity, innovation) 2) long time-horizon on generative ai, wide-spread adoption, and digital sentience and so here we are. I hope that boards will act on that signal, but am not yet optimistic. A lot of big egos are going to have to walk back pet projects and big promises to Constituents, Congress, and Wall Street, and sadly, that seems unlikely to be a swift process absent tight regulation - which, will almost certainly be viewed as reactionary and overly-restrictive.