Epistemic status: unsure
I have a hypothesis about why Zettlekasten provide diminishing returns over time. A corrolary is that others should find even less value in your Zettles. Which ties into some of your points, and shows what is missing from the Zibbaldone. Plus there are some suggestions on how to correct the flaw.
One of the key benefits to the Zettlekasten is that the way you link cards reflects your psyche's understanding of the ideas. Of course other note-taking systems have this advantage. But this isn't baked into them like it is with Zettlekasten.
Traversing the Zettlekasten lets you approximate your past state of mind when working on a problem. Which lets you dredge up whatever your subconscious has come up with on the topic. The seemingly random orgranisations of yesterdays Zettles helps this along a little by providing a glimpse into yesterdays self. So when you wake up in the morning and look over yesterdays cards, a flood of relevant houghts arises. When considering where to place them, you glance through your Zettlekasten. Zettles your mind was working on bring new thoughts to the forefront. Often it will feel trivial to combine and play with all these new ideas, generating even more thoughts.
Unfortunately, past a certain size your Zettlekasten contains too many cards for your brain to be processing at once and too many potential states of mind you could have been in. We expect a gradual reduction in value of the system. And less value to others, who have a different understanding of the ideas. Something similair is true for other note taking systems. There's just a steeper decline in value.
How does this square with peoples' reports that between different note taking systems provides the same early returns they got with the old one? My guess is that the reduced scope of your notes and the context shift is what does it. Your brain realises it no longer has to keep track of all those ideas and can focus on a few relatively simple ones. Which makes the low hanging fruit all the easier to grab.
Can we fix these issues? Maybe. And I think you'd have to go digital to do it. Consider spaced repetition. A memory's strength decays exponentially with time. When reminded of them, your memory is strengthened. Spaced repition takes advantage of the fact that there are optimal timings to strengthen the memory. By analogy, we might say there are optimal times to dredge up ideas from your mind. And likewise, there may be optimal timings to link zettles. Perhaps these timings depend on the "distance" between the zettles.
A digital system could provide all this. A useful format would be the zettle to consider, and a graph surrounding it of the zettles you should link it to. When moving to adjacent zettles, you can see all the zettles it links to in order to provide relevant context.
I call this "anti-spaced repetition": the benefit is from surfacing connections for material you've forgotten (as opposed to reviewing material you still remember so as to strengthen retention). You can optimize time spent reviewing older material by using the spacing effect to estimate which things have been forgotten for the longest - same equation, just optimizing for something else.
I started writing a blog post in response, but that seems a bit much for a comment. Suffice to say, I agree that anti-spaced repetition is a good idea. However, it throws away the context of the notes you made, as well as showing it to you after your mind has totally forgotten about it. And as I wrote, those seem to be major factors in the value of the Zettlekasten method!
Why can't any individual 'item' be shown with context like a dozen lines before/after (eg fading out)?
Before or after what? If it is a passage in a book, or an article you wrote, I agree that's enough. But what about a nebulous concept you struggled to put into words? Or an idea which seemed to have suprising links to other thoughts, which you didn't pursue at the time. If you write all this stuff down explicitly, then fine. If not, and you're writing style is like mine, then it seems better to link to other cards and leave it to your future self to figure it out.
Plus, links provide the system extra information with which it can auto-suggest other relevant ideas that you weren't even aware you were considering.
You have to do aggressive recompression of your past stuff. This takes time but pays off in efficiency gains in your internal representations IME.
Do you do this in a piecemeal way, or do you assign a few days to re-organising your thoughts when you learn some important new principle?
That'd be an interesting structure for sure. Some kind of spaced repetition, presenting you with two (or three? or more?) prior thoughts to read simultaneously, to reinforce not just the information, but the relationship between the different ideas; not just in isolation, but reinforcing the network itself... maybe with some kind of highlight markup to indicate where the parallel is strongest.
With regards to the Zettelkasten containing too many cards to keep up with, I think card agglomeration above a certain threshold might be useful. Rather than hyperlinking between 500 cards with one sentence each on them, it may be preferable to link to particular sentences or sections within 50 different cards. The hyper-deconstruction of the original index card system was I think more a limitation of index card size, and how you might best utilize a physical system. Hypertext and NLP could identify links, and keeping a human in the loop ensures they're the kind of parallels you actually want to be forming... to some degree. Being presented with particular links still might still have some subtle undesirable biases. But the prospect of "combine these cards?" might make things more manageable.
I think a manual consolidation step would at least prompt for better structure for future consumption. If you know the order you'll always want to view the notes in, turn them into the one note with headings. That could keep the knowledge bank a bit more manageable and navigable. I don't think the current graph view in systems like Roam or Obsidian is enough - you need a text preview of those notes. Maybe for each new note, checking the graph structure remains planar, avoiding messy crossovers in visualisations of it. Or maybe that's a terrible idea, I have no idea! There's lots of low hanging fruit in the space, and whoever makes the biggest fruit basket's going to win here.
Yeah, I had some ideas concerning how to keep track of Zettlekasten as well as the right way to display graphs. Reinforcing the network is definetely a worthwhile idea. The entire point is to suggest good links, but also give you the freedom to traverse your graph. RE the hyperlinks: I agree about the worry of biases. But more than that, it seems the network should not automate link suggestion without leaving the option to create links yourself. As you say, the worth of the Zettlekasten method is largely in instilling virtuous mental hanits. What you suggested seems like it could instil laziness in the user.
Zettelkästen (slip-box) and Zibaldone (commonplace book) serve two different purposes. That’s why they have a different structure. One is a web, the other a notebook. Can you link things in a notebook structure? Yes, but it’s much easier to use a web structure.
Also, while using an AI as a personal memory assistant may be useful, I guess, looking at past innovations, that it could damage learning process.
As to the narrative, the more you link to a node the more it’s pivotal. Usually it’s the center of a major theme. That’s a simple way to see paths.
Personally, I use both for the aforementioned reasons. Also, I have to say that the Zettelkästen has given me higher returns than the Zibaldone for research.
External is becoming more disintegrated, seems reasonable to aim for dramatically higher internal integration to partially make up for it.
Less Zettelkasten! More Zibbaldone!
A Zettelkasten requires you to intricately interconnect and crosslink your thoughts, figuring out exactly how each concept relates to every other concept. A Zibbaldone is writing down whatever random thing comes to mind - an omnisubject diary of sorts. That's what people are throwing into their Zettelkasten, with their careful annotations and interrelations. What if everyone could access everyone else's Zettelkasten, and interrlate them into a glorious omniscient noospheric substrate? Wow! Whatta thought!
Buuuuuuullshit. This is semantic web stuff all over again - and right at the cusp of it having been clearly and entirely outmoded by automatic natural language processing. We don't need to manually mark up pages when Google will index them all for us anyway! Humans might think in that chaotic, random Zettelkastian way, but explicit sentence composition and task completion is temporally linear, and so we need to output our thoughts in much the same way. Not that we wouldn't love to manually annotate our every passing thought, but god, who has time for that?
What we're really missing is a tool that will take whatever random trash I throw out of my brain, figure out how it relates to historical junk I threw out of my brain, and point out the connections for me. Hell, ideally mid-composition: a modern ersatz Clippy pops up - "Hey, it looks like you're talking about Wittgenstein again! Do you want me to autofill what you already think about it, or have you changed your mind about the topic?"
Not only that: once you get your thoughts out of your head bereft of order, an AI could rephrase your thoughts back to you more clearly, more concisely, help interrogate what you mean, point out logical contradictions, challenge your ideas. We need only Zibbaldone out our stream of consciousness and it can be autonomously interrlated into a glorious omniscient noospheric substrate! Wow! Whatta thought!
...just ignore the risks of mentally coupling with an opaque box, whose biases you don't understand, whose intentions may be short term and not long term, who may network you with exactly the wrong people. Zibbaldone to Zettelkasten mapping seems like a middling-difficulty problem with existing tools, and whoever solves it is going to make a lot of money. When we can auto-generate a Zettelkasten from a blog - or dozens of them - you'll have a whole world of brilliance to tap into!
...or bias. Or noise. Or exactly the sort of thing you're trying to escape from - the internet, where the awful people are, with the Bad Opinions. And what you end up is just another wiki to mindlessly trawl through, to process, to "integrate". I'm starting to think the important part of a Zettelkasten isn't making the links for your future self so much as training yourself to notice the links at all. The Zettelkasten's just external practice for what should be an internal and subconscious process - seeing the fnords, noticing the interconnectedness, it's just training to be a holistic discordian - getting you past the madness of your surface layer thoughts and into the deeper crystalline method.
Of course, everyone will think you're a little crazy - and of course, they'll be a little right. But when the going gets weird, the weird turn pro, and when insanity's the zeitgeist, nothing's more professional than crazy talk. Have fun making bank on the Zibbaldone-Zettelkasten mapping, crafting neurological parasites, offloading the part of our brain with connections and opinions to an external cognitive artifact! No way that can go wrong.
Really, the harder problem's going the other way - give me a web, which path should I walk? What is the narrative of a human mind? You can't read a Zettelkasten effectively, it has too many strands and streams and errant thoughts; you have to craft a narrative out of it, even if you're consuming it out of order, but figuring out how to knit the threads together into a cohesive chunk for consumption's the hard part. What's the mapping then? Zibbaldone to Zettelkasten to... Roman-Fleuve, ministructure to the metastructure? Zettel-fleuve, Zibbal-fleuve? Zittal, Zabbal, Zettaldone, Zibbalkasten, - god, we're all turning into lunatics aren't we? The jargon's making zibbering lunatics of us all - just write something down, anything, throw it to the aether and be zibbaldone with it all.