On context rot and dementia
What is context rot?
The quality degradation that happens as an LLM fills its context window.
I’ve been thinking of the similarities between context rot and dementia.
What is dementia?
A general decline in cognitive abilities that affects a person's ability to perform everyday activities. This typically involves problems with memory, thinking, behavior, and motor control.
Are there lessons we are learning from managing context rot in LLMs that can be used in mitigating, or even reversing, dementia and its symptoms?
I know, it is a wacky idea, but getting mold to develop a miracle drug to kill bacterias was also a wacky idea, and we got penicilin out of it, so maybe wacky ideas is how we get to move forward in this world - sometimes.
So what are we doing to manage context rot?
External memory
Instead of remembering everything and in the end, because of it, nothing, we offload the memory to an external fixed storage location, and retrieve it when needed, we might keep a short summary with us at all time, but not the full detail.
- Will that work for our brains too?
- I believe our brains already have long and short memory capacity, is dementia the inability to store the former or use the latter?
- Is journaling / blogging a form of external memory?
Semantic caching
It’s when two questions are similar, so you use the response of the first for the second, rather than thinking it through again.
Our brains do this all the time, fortunes have been build exploiting this pattern, and is so pervasive that many campaigns to change this mechanism have ultimately failed (discrimination is a prime example of this in action).
Compaction
Similar as external memory, as we keep a summary in local memory, but unlike external memory, nothing gets stored in long term storage, we simply destroy information, and replace it with something smaller but “good enough”.
Keep going
If anything, it’s all food for thoughts, and maybe that, keeping the gears grinding, no matter what technic we need to employ to keep doing so, is the way to keep dementia at bay.