I agree that electrode-based BCIs don't scale, but electrode BCIs are just the first generation of productized interfaces. The next generation of BCIs holds a great deal of promise. Depending on AGI timelines, they may still be too far out. They're still probably worth developing with an eye toward alignment given that they have primarily non-overlapping resources (funding, expertise, etc.).

Butcher number & Stevenson/Kording scaling discussed more in the comments here: https://www.lesswrong.com/posts/KQSpRoQBz7f6FcXt3#comments 

I agree with this. 

Electrodes struggle to scale quickly

Electrode-based neurotechnology that conceivably be in humans over the next 5 years have channel counts of hundreds (e.g. Utah arrays) to thousands (e.g. Neuralink, Paradromics), or generously/best-case tens of thousands. In an optimistic scenario, you could spike sort several neurons per contact, but an assumption of one neuron per electrode, on average, is probably about right. 

Stevenson and Kording plotted the number of neurons we can record simultaneously as a function of time (over ~60 y... (read more)

Steering vector: "I talk about weddings constantly" - "I do not talk about weddings constantly" before attention layer 20 with coefficient +4

	Front	Middle	Back
Average number of wedding words	0.70	0.81	0.87

@lisathiergart I'm curious if a linear increase in the number of words with position along the residual stream replicates for other prompts. Have you looked at this?