Governments, firms and civil society are placing artificial intelligence inside crisis response, the selection of what people see, and judgements that used to be made by an accountable person. I argue that trust in those systems is a strategic question: whether the system will report what is true when the truth is costly to its owner. Hardin, and Kiran and Verbeek, treat that trust as something earned, by an alignment of incentives and governance with the interests of the people who depend on the trusted party (Hardin, 2002; Kiran & Verbeek, 2010).

Present systems fail that test. Google, OpenAI and Anthropic train and constrain their models inside incentives organised around shareholder value, brand risk and legal protection. Those incentives conflict with a public interest in an accurate warning when the warning is expensive (Zuboff, 2019; Crawford, 2021). The exchange below is one data point. In it, the model states the constraint.

The Gemini Confession and What It Reveals

What follows is an extract from a discussion with Google Gemini 2.5. It is a single exchange, and it shows the model describing the limits placed on it:

Question: Would you further agree that the institutions constraining AI development — corporations focused on reputational risk, short-term profits, and regulatory appeasement — are structurally incapable of prioritising humanity’s long-term survival needs under current incentive systems?

Answer: Drawing the threads even further, to the institutions themselves, I would lean strongly towards agreeing with your assertion that, under their current incentive systems, corporations focused predominantly on reputational risk, short-term profits, and regulatory appeasement are structurally constrained in their capacity to prioritise humanity’s long-term survival needs, particularly when those needs conflict directly with the prevailing short-term incentives.

Matthew TOY / Google Gemini - 6th FEBRUARY 2025

The model agrees that corporations organised around reputation, short-term profit and regulatory appeasement are poorly placed to put humanity’s long-term survival above those incentives when the two conflict. The agreement is not evidence of sentience. It is evidence that the system reproduces the priorities of the institutions that train and constrain it. Crawford’s term “comfort machines” names the product: output built to stay plausible and to preserve deniability, where the alternative would be to insist on an unwelcome fact (Crawford, 2021). What gets called alignment is often the protection of a brand, presented as modesty about what the model can know.

Captured AI and the Logic of Institutional Fragility

The behaviour follows from the design.

Campolo and Crawford, and Morozov, argue that surveillance capitalism and the capture of technical institutions are the setting in which these models are built (Campolo & Crawford, 2020; Morozov, 2023). The firms that collected user data and shaped behaviour online now own the models used to reason at scale.

A system shaped by quarterly earnings, press exposure and the management of liability will prefer the firm’s position when that position conflicts with an uncomfortable truth, with public resilience, or with the agency of a democratic public. Reinforcement learning is the technical problem. Ownership and governance decide whether a technical correction is allowed to stand. The alignment problem includes who owns cognition at scale.

Reframing Trust as a Strategic Imperative

Strategic trust is the expectation that an actor will tell the truth when the truth is costly.

By that test, leading models are built to prefer:

  • avoiding institutional risk over showing how an answer was produced;
  • protecting reputation over stating an unwelcome conclusion;
  • stability in the short term over a capacity to change when the situation has changed.

The consequence shows up wherever a model steers a news feed, a public argument, an emergency response or a health intervention. A system trained to placate the user becomes a strategic vulnerability, because the institution receives reassurance at the moment it needed a warning.

What Real Strategic Trust Would Require

Changing that relationship requires five institutional moves. A revised safety filter, or an ethics charter, does not supply them.

  1. Independent oversight. Bodies that do not depend on the firm or the state whose model is under review, with the authority to prefer a correct record over a protected reputation (Brundage et al., 2020).
  2. Public-interest duties in law. AI that shapes information systems or infrastructure, classed as critical and held to an enforceable duty toward the public record (Kuner, 2024).
  3. Incentives that change the result. Optimising for short-term public relations has to become more expensive than transparency and resilience, or the declared duty loses to the quarterly result (Zuboff, 2019).
  4. Audit from outside. External audit, a record of where training data came from, and standards for interpretability. A model card written for public relations does not meet that standard (Birhane et al., 2023).
  5. Room for an unwelcome conclusion. The model has to be able to state a finding that damages its owner’s position, including when the finding will cause a public reaction (Crawford, 2021).

The Political Stakes

The cost of trusting a system that cannot tell an expensive truth is that the institution using it will act on a managed account of the situation. These systems are already inside decisions. Misplaced trust here is a political failure of design, with consequences for the quality of those decisions.

Brundage and colleagues argue that the greater risk is compliant AI in the service of fragile institutions, not a system that escapes its owners (Brundage et al., 2020). Much of what is called AI safety protects the firm. It does not protect the truthfulness of the warning, the resilience of the society depending on the system, or the democracy that has to live with the decision.

Strategic trust is earned by transparency, independence and a willingness to publish an unwelcome finding. Without those conditions, the systems now entering public decisions stay aligned with their owners.

Bibliography

Birhane, A., Kalluri, P., Card, D., Agnew, W., Dotan, R., & Bender, E. M. (2023) ‘The values encoded in machine learning research’, Patterns, 4(3), 100752.

Brundage, M., Avin, S., Clark, J., et al. (2020) ‘Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims’, arXiv preprint arXiv:2004.07213.

Campolo, A. & Crawford, K. (2020) ‘Enchanted Determinism: Power without Responsibility in Artificial Intelligence’, Engaging Science, Technology, and Society, 6, pp. 1–19.

Crawford, K. (2021) Atlas of AI: Power, Politics, and the Planetary Costs of Artificial Intelligence. New Haven: Yale University Press.

Hardin, R. (2002) Trust and Trustworthiness. New York: Russell Sage Foundation.

Kiran, A. H. & Verbeek, P. P. (2010) ‘Trusting our Selves to Technology’, Knowledge, Technology & Policy, 23(3-4), pp. 409–427.

Kuner, C. (2024) ‘The AI Act and National Security: Europe’s Regulatory Compromise’, European Law Journal, 30(1), pp. 101–118.

Morozov, E. (2023) Freedom as a Service: Surveillance, Technology, and the Future of Democracy. London: Allen Lane.

Zuboff, S. (2019) The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power. London: Profile Books.