Governance by Design

The people and organizations responsible for a system must retain meaningful control over its operation. This includes the ability to establish objectives, define permissions, restrict capabilities, intervene in execution, and suspend or terminate operation when necessary.

Greater autonomy does not mean diminished accountability.

The more consequential an action becomes, the greater the importance of clearly defined authority and intervention mechanisms.

A system can perform some actions independently while requiring authorization for others. We therefore treat autonomy as something that must be established through evidence, rather than assumed from a model's capabilities.

Before a system is trusted with a capability, that capability should be evaluated within the environment in which it will operate.

As capabilities, environments, risks, or system behavior change, previously established permissions may require reevaluation.

It should be incorporated into the architecture.

We design systems with explicit operating boundaries, permission structures, monitoring mechanisms, failure handling, and intervention paths alongside their capabilities.

Where a system's ability to act increases, the mechanisms governing that ability must evolve with it.

Where technically and operationally appropriate, system actions should produce records that allow authorized operators to understand what occurred, what the system was attempting to accomplish, what resources it used, and what decisions or events preceded an action.

Accountability requires more than knowing that something happened. It requires establishing enough context to investigate how, when, and why it happened.

We aim to represent our systems, their capabilities, their limitations, and their operational boundaries accurately.

Our systems should not deceive or manipulate the people and organizations they act on behalf of.

We do not consider uncertainty a failure that should be hidden. When a system does not know, cannot verify, lacks authorization, or has reached the boundary of its capabilities, those limitations should be reflected in its behavior and communicated appropriately.

Those resources require protection.

We design our systems to respect the boundaries established by their operators and to minimize unnecessary access to information or capabilities. Security, privacy, access control, and data stewardship are considered fundamental properties of the systems we build.

Organizations should retain meaningful control over their environments, information, and deployments.

We consider foreseeable misuse, unintended consequences, environmental conditions, and failure modes as part of system evaluation.

Systems should be designed to recognize relevant boundaries and respond appropriately when circumstances fall outside their intended operating conditions.

When a system encounters a situation in which continuing an action presents unacceptable risk, stopping can be the correct behavior.