I though I was going to have more ideas, but after sitting around a bit I'll just post what I had in terms of friendliness architectural/design choices that are disjoint from safety choices:

• Moral and procedural self-reflection:
  • Human feedback / human in the loop
  • System should do active inference / actively help humans give good feedback.
  • Situational awareness to the extent that the situation at hand is relevant to self-reflection.
  • Bootstrapping:
    • The system should start out safe, with a broad knowledge about humans and ability to implement human feedback.
    • The "Broad knowledge yet safe" part definitely incentivizes language models being involved somewhere.
    • Implementing feedback, however, means more dangerous technology such as automated interpretability or a non-transformer architecture that's good at within-lifetime learning.
• Moral pluralism / conservatism
  • The central idea is to avoid (or at least be averse to) parts of state space where the learned notions of value are worse abstractions / where different ways of generalizing human values wildly diverge.
  • Requires specifying goals not in terms of a value function (or similar), but in terms of a recipe for learning about the world that eventually spits out a value function (or similar).
  • Easier to do if the outer layer of the system is classical AI, though we probably shouldn't rule out options.
• Dataset / Human-side design:
  • Broad sample of humanity is able to give feedback.
  • Transparency and accountability of the feedback process is important.
•  Capabilities useful for friendliness and problems for safety:
  • Situational awareness
  • Independent planning and action when needed (some amount of consequentialist evaluation of soliciting human feedback/oversight).
  • Generality
  • Within-lifetime learning

My improved (but still somewhat bad, pretty non-exhaustive, and in-progress) list:

• Architecture/design
  • The system is an agent composed of language models (and more powerful agent-scaffolding is better)
  • The system is composed of separable narrow tools
  • The system is legible and process-based
  • The system doesn't have hidden states (e.g. it's feedforward-y rather than recurrent-y)
  • The system can't run on general-purpose hardware
• Data & training
  • [Avoid training on dangerous-capabilities-stuff and stuff likely to give situational awareness. Maybe avoid training on language model outputs.]
  • ?
• Limiting unnecessary capabilities (depends on intended use)
  • The system is tool-y rather than agent-y
  • The system lacks situational awareness
  • The system is myopic
  • The system lacks various dangerous capabilities (e.g., coding, weapon-building, human-modeling, planning)
  • The system lacks access to various dangerous tools/plugins/actuators
  • Context and hidden states are erased when not needed
• Interpretability
  • The system's reasoning is externalized in natural language (and that externalized reasoning is faithful)
  • The system doesn't have hidden states
  • The system uses natural language for all outputs (including interfacing between modules, chain-of-thought, scratchpad, etc.)
• Oversight (overlaps with interpretability)
  • The system is monitored by another AI system
  • The system has humans in the loop (even better to the extent that they participate in or understand its decisions, rather than just approving inscrutable decisions) (in particular, consequential actions require human approval)
  • The system decomposes tasks into subtasks in comprehensible ways, and the interfaces between subagents performing subtasks are transparent and interpretable
  • The system is more supervisable or amenable to AI oversight
    • What low-level properties determine this besides interpretable-ness and decomposing-tasks-comprehensibly?
  • Humans review outputs, chain-of-thought/scratchpad/etc., and maybe inputs/context
• Corrigibility
  • [The system doesn't create new agents/systems]
  • [Maybe satisficing/quantilization]
  • ?
• Incident response
  • Model inputs and outputs are saved for review in case of an incident
  • [Maybe something about shutdown or kill switches; I don't know how that works]

(Some comments reference the original list, so rather than edit it I put my improved list here.)

Maybe relatively safe if:

• Not too big
• No self-improvement
• No continual learning
• Curated training data, no throwing everything into the cauldron
• No access to raw data from the environment
• Not curious or novelty-seeking
• Not trying to maximize or minimize anything or push anything to the limit
• Not capable enough for catastrophic misuse by humans

Another factor for the safest type of AGI is one that can practically be built soon.

The perfect is the enemy of the good. A perfectly safe system that will be deployed five years after the first self-improving AGI is probably useless.

Of course the safest path is to never build an agentic AGI. But that seems unlikely.

This criteria is another argument for language model agents. I've outlined their list of safety advantages here.

Of course, we don't know if language model agents will achieve full AGI. 

Another path to AGI that seems both achievable and alignable is loosely brainlike AGI, along the lines of LeCun's proposed H-JEPA. Steve Byrnes' "plan for mediocre alignment" seems extensible to become quite a good plan for this type of AGI.  

Someone anonymously suggests:

• Keep pretraining closely matching the human text distribution
• Require all rumination to be done in natural language
• Require approval before taking consequential actions
• Only deploy a model with a second adversarial monitor
• Only deploy a model if it's "dumb" in a quantifiable way