Is there some sort of pattern detecting thing, whose name perhaps includes something like "markov" or "kolmogorov" or "bayesian", that could automatically take a time series data and predict the next values based on an unknown, complex model?
Prediction isn't magic -- you need relevant information. In this case, if information about the causes of your sleep patterns are not in the time series, you won't be able to accomplish much with this approach.
I note that you include info about caffeine and other drug usage in your data. These are obvious possible causal agents, so it's necessary but not sufficient to have information about them. I suggest the following approach:
Comments:
On point 1, I'd say the odds are better than 50:50 that you've already done the relevant lit review. I still included it in the plan of attack in case you hadn't.
On point 2, I suggest months-long exhaustive data collection only because you've already shown that you are motivated to do it. Also, I think it's important to collect data on your eating habits whether or not it shows up as potentially causal in the lit review.
I notice that the karma on this post has fluctuated a bit. I don't care about the karma per se, but I do care that someone indicated they don't want to see more like this. So I invite any criticism here and now, that I may improve.
I have karma display turned off (greasemonkey script). It stresses me out. I think your comment could certainly expand on point 3/4. Really what I was looking for as a response to the post is a good pointer on what sort of algorithms or tools could potentially give me good results on this problem to direct my studying, and perhaps what textbooks or introductions I should be reading.
But point 1 is good. I hadn't thought to do that. I was just going to go on common sense, and a kitchen sink approach.
Would it have improved the comment if I had stated explicitly at the start that my reply was not directly responsive to your request but rather addressed an oversight/implicit assumption?
Is there some sort of pattern detecting thing ... that could automatically take a time series data and predict the next values based on an unknown, complex model?
There are lots of tools like this. The problem is getting enough clean data. If you don't have enough data, you won't be able to use a complex model without overfitting. If the underlying phenomena (i.e. your sleep cycles) is itself complex, then you won't be able to obtain a good description. On the other hand, it can't hurt to try. How much data do you have?
I have six months of past sleep data, though nothing current, with sleep and wake times. I could easily augment that with other potentially relevant variables, like daily caffeine intake or whatnot.
As a starter method, I would try Adaboost. AdaBoost is nice because it is easy to implement, gives some protection against overfitting, and allows you a lot of liberty to define whatever context functions/predictors you want. Try to predict whether a given hour will be sleep or not. Use whatever information like caffeine intake you can as predictors, and use as many of them as you can dream up: AdaBoost will figure out which ones are the most important.
You could ask this guy. I'm saying nothing of the "worthiness" of credentials, just that he's studied and written quite a bit about sleep. Here's an article HERE.
He advocates for biphasic sleep (longer 7-9hr "core" + ~30min nap) and thinks it's how the body prefers to sleep.
Maybe write to him and include your data and see if he can offer you any input? He may very well get contacted by people in your situation and have some information about what helped them? I do know that he advocates for people having trouble to attempt "free running sleep" if possible -- basically sleeping when tired, waking when not. Obviously this is impractical for most, but if your situation allowed for it, you could really give a shot to shutting out distractions and just sleep and keep a log.
Anyway, not much of a help, but I stumbled on his work a bit back and thought I'd at least toss it out as a suggestion. At least contacting him might prove helpful. See his "Apology" link near his contact information. He sounds tough to get a hold of. I'd make it a "provocative" subject line, like "Need serious sleep assistance. 5 mos of data attached."
I use Supermemo daily, and have read everything Wozniak has written about sleep. I've talked to him a couple times about other things (1-2 month response time). I may ask him about this.
Wow -- I'm quite impressed! You've really tried a helluva lot of things. I'd see what he says. One never knows. Good luck finding assistance.
What information are you hoping to get out of this? "I just slept for 7 hours, I can expect to be awake for X hours with probability pX"? Or "my sleep patterns repeat this 125 hour cycle"?
If you have the data in a convenient format, I could take a look at it more easily than I could explain how to take a look at it.
More like, "here's the times I went to sleep and woke up in the previous month. What can I expect today?" Hopefully including the effects of caffeine, delayed sleep, early awakening, etc. My sleep may sort of follow a cycle, but it's not regular enough that knowing the cycle would be that useful.
Here's the raw data for 6 months or so last year: Data.
EDIT: I was unemployed during this period, and not using an alarm regularly, so I was sleeping exactly when I felt like it. If I was working it would look much different.
Now we're getting somewhere! One thing really jumps out at me in that log:
3-22 back was hurting. need new mattress
4-01 Woke up with horribly aching back. (I blame my bed).
4-02 I hate my mattress.
4-05 got up with aching back. god damn bed.
I see no mention in the log of it having been replaced. That is a big deal. It may not be the entire cause of your sleep problem, but it is at least a very major contributing factor. You need to replace that mattress immediately, or find somewhere else to sleep.
It was replaced shortly after, and my back problems promptly dissipated. I had only been sleeping on that mattress for a few weeks at the time, having just thrown away another.
Side note: I thought my mattress was giving me lots of back problems (and incidentally sleep problems), and replaced it, and everything was fixed. Some months later I resumed sleeping with a certain very comfortable, too-small blanket. I hadn't realized I had stopped when I switched mattresses. The back problems returned. I got rid of the blanket, they went away. The too-small blanket had caused me to curl up into too tight a ball while sleeping.
Ok, here's what I just did. I looked through that log, collected all the names of pharmaceuticals in there (Klonopin, Vyvanse, Lunesta, Melatonin), and searched each one's wikipedia page for sleep-related effects and side-effects. And I found this:
While benzodiazepines induce sleep, they tend to produce a poorer quality sleep than natural sleep. Benzodiazepines such as clonazepam [Klonopin] suppress REM sleep.
So that's probably a net negative, sleep-wise, although there might still be non-sleep-related reasons to take it.
Wikipedia has this to say about Lunesta:
In the longest, largest phase 3 trial, patients in the Lunesta group reported falling asleep an average of 15 minutes faster and sleeping an average of 37 minutes longer than those in the placebo group. However, on average, Lunesta patients still met criteria for insomnia and reported no clinically meaningful improvement in next-day alertness or functioning.
Vyvanse has insomnia listed as a side effect. It's not clear from the log whether you just used it once, in which case that's unimportant, or for a long time in which case it is. Melatonin is sometimes used as a sleeping aid, but in my experience it's pretty weak. It may be effective against a particular cause of insomnia, but it didn't do much for me and if it didn't do much for you, well, that's not that surprising.
So, the good news is that the reason using drugs to help you sleep hasn't worked is that they're lousy drugs. The next step is to go through the wikipedia page for insomnia, and collect a list of candidate substances. Then read the page for each one, cross off the ones that sound bad (mainly based on side effect risk), and get a list of candidates to try. Bring this list to your physician, let him veto any subset, get prescriptions for any remaining ones that require prescriptions, and then try each one in turn.
Vyvanse has insomnia listed as a side effect.
Well, Vyvanse is modified amphetamine, so yeah. I also have serious focus problems. I was only on it for a month or so, and found it ineffective for the same reasons as other stimulants. I think in the sleep log I had just taken an isolated pill I had left.
But your advice is good. Going through the options very thoroughly might turn up something.
Not really an answer to your specific question, but have you tried, or considered trying, something radically different, like a polyphasic sleep cycle (e.g. a 20-minute nap every four hours and nothing else)?
Polyphasic sleep suggestions are quite popular on the internet, but they also do seem quite irrational. The negative risk of trying it is very high, you could seriously damage your health by doing it (risk of the fatal car accident could possibly increase) sleep deprivation, while reward is low to medium at best and depends if you can take advantage of more time available to you. And i'm just scratching the surface here.
Also consider:
High entry barriers - minimum 2 weeks just to change your habits
Requires very disciplined approach
Very little science supporting it, mainly anectodal evidence
Maybe a good analogy would be to suggest someone who has financial problems to try a radical approach of robbing a bank.
I see what you're saying, certainly. But we're talking about someone who is already having a lot of sleep problems, and has exhausted all the other options they could find. They may find they're better able to keep up a consistent polyphasic regime than a more standard sleep pattern, and if their sleep problems are already bad enough, it may be worth the drawbacks to give it a shot.
If you want a model, how about this for a simple but not very good first attempt:
What I'm wondering with a markov process is whether it could be embellished to include other potentially relevant variables. From 5 minutes reading wikipedia, it seems like I'd have a combinatorial explosion of states, and the more states, the more data needed to train the model.
So I'd have like 48 states, for each half-hour of the day, times 3-4 for 8-11 hours long sleep? Would it work to have ordered pairs where the first item is measured in time since my last awakening?
I find it impossible to wake up at a consistent time every day (+/- 8 hours)
By this, do you mean that the time you wake up varies over a full range of 16 hours around its median?
I wouldn't exactly call it a median. It trends forward every day, eventually wraps around, but it doesn't spend much time at all around 2-8 AM, due to sunlight keeping me awake when I'd otherwise go to sleep in late morning or afternoon.
You're Harry Potter!
On a more serious note, how would you deal with a constantly forward-moving wake time even if you could predict it?
It trends forward every day, eventually wraps around
This sounds a lot like Non-24-hour sleep-wake syndrome. The defining symptom for Non-24 is (from Wikipedia) "a chronic steady pattern comprising one- to two-hour daily delays in sleep onset and wake times in an individual living in society". Your delays seem to be longer than 1-2 hours, but it may be a similar problem. I don't know how much you've looked into this, given the impressive extent of your other searches, but it may be something to look into.
Have you tried light therapy? Wikipedia (and this study) recommend it, perhaps in combination with melatonin, as the most effective treatment of Non-24.
Not sure how valid this is, but it might be worth looking into, if you haven't already.
For a while I thought I had delayed sleep phase syndrome (which is more easily treated with light therapy), and that it's just so severe that the morning sunlight late in my day tends to make it go crazy. It's not quite regular enough for non-24. Or it could be completely irregular.
In any case, light therapy doesn't seems to help at all. I tried it for about a month or two with this and saw no effects. Also, it's a /huge/ inconvenience.
If it's really that bad, you should use a mild sedative to force yourself to fall asleep at a consistent time.
My sleep is seriously disturbed. ... I find it impossible to wake up at a consistent time every day (+/- 8 hours), despite years of trying, which makes it extremely hard to hold down a job, or do dozens of other normal things. There could be a profession that I could make my sleep work with, but I'm still searching for it.
If it's really that bad, you should use a mild sedative to force yourself to fall asleep at a consistent time.
These don't work for me. The details are boring.
The details may be boring, but they matter a great deal. I believe you when you say you tried using sedatives and it didn't work, but there are a lot of different ones to try and a lot of different ways to use them. Which ones have you tried, and in what way(s) were they unsuitable? Have you actually run out of compounds to try, or did you just get discouraged by a few bad results?
I have tried sedatives, melatonin, melatonin-inducing sleeping aids, traditional sleeping aids, and Ambien (whatever that is). Some have no effect, some put me to sleep but leave me unrested, and some put me to sleep and leave me unrested and incredibly groggy for the rest of the day. Generally speaking, trying to shift your sleep schedule by more than 1-2 hours using sleep aids doesn't work. If your circadian rhythm keeps advancing anyway, the results are just like a normal person trying to go to bed at noon using sleep aids.
a lot of different ways to use them
Can you expand on this?
a lot of different ways to use them
Can you expand on this?
You can vary the dosage, the timing, and the preconditions (ie use it only when you predict you'll fail to fall asleep otherwise). You can use the same compound with different release profiles. You can mix some combinations with each other (but not all combinations; X and Y being safe individually does not always mean coadministration is safe).
Edit: I humbly remove this. As not to simply "run and hide", it was an off-color comment. I had already debated with myself about whether to even post it but thought it might produce at least something humorous to the OP. It's been downvoted and I'm pulling it before many more have to see it.
I apologize.
Edit2: were this to be left up, I could see it getting to -5-10 depending on traffic, so feel free to continue down vote to that point.
I'm aware that you're asking how to predict, rather than how to control. Unfortunately the ideas I have on hand are about how to control.
First there's Ferris' hack of taking a cold bath, item 2 of:
http://www.fourhourworkweek.com/blog/2008/01/27/relax-like-a-pro-5-steps-to-hacking-your-sleep/
I have not tried that myself. Second, there's orange glasses:
http://www.cleveland.com/healthfit/index.ssf/2010/06/goggles_with_orange_lenses_blo.html
These seem good-enough and they're cheap:
http://www.amazon.com/gp/product/B0001YXG8S
My own self-experimentation says they seem to help. Wear them when it's dark outside so artificial light doesn't push around your sleep schedule.
I'd start by playing around with the data, trying to find different ways to organize it, identifying and defining relevant variables, and making graphs to look for patterns. Then you can start to look at relatively simple tests of relationships, and once you have a better idea of how the data are structured and what sorts of relationships seem to be present then you can try fancier math (if necessary).
For instance, you linked to your data organized by sleep cycle, starting a new row of data each time you fell asleep. I made a spreadsheet here using that organization (but slightly different formatting which was easier for me to work with) and used it to make this graph, which shows how much time you slept in blue and then the following amount of time you spent awake in orange, and then starts the next sleep period on the row below it. Two things that jump out from the graph are that you have lots of short cycles (under 5 hours, even) and that most of the cycles are under 24 hours. If you tend to find yourself on a cycle of over 24 hours (going to bed later and later each day), that must be from sleeping multiple times in a single calendar day.
Another way to organize the data is by calendar day. You could have one row for each calendar day (possibly starting each day at a time other than midnight, like the time when you are most often awake or when you're most often asleep) and make a two-color graph of when you're asleep & awake, like the one I linked to (except this time it would be a rectangle). With the data organized that way, you could look at questions like: how often am I awake at each time of day? If I am awake at a certain moment, how likely am I to be awake 24 hours later? If I am asleep? Or x hours later - you could make a graph where the x-axis is the number of hours later, and the y-axis is the probability of being awake that long later (give that you're awake/asleep at t0). How does this change depending on time of day?
There are also various correlations that you could look at, or graphs that you can plot of 2 variables, like bedtime, length of sleep, wakeup time, length of awakenness, sleep debt (percent of time asleep during previous 72 hours? during previous 4 sleep cycles?). You might need to do something about the short cycles (leave them out? combine them with adjacent sleep/wake periods?). Here is a graph of amount of sleep vs. bedtime, which seems to show a pattern (you sleep longer when you fall asleep earlier) if we ignore the short sleeps.
Or, come up with some definition of a "normal" sleep cycle (e.g. one lasting close to 24 hours, which includes close to 8 hours of sleep) or a normal day (e.g. sleeping at least 6 hours from 10pm-10am and then being awake for at least 9 hours from 10am-10pm). How common are these "normal" cycles/days, are there any patterns to when they occur, do they come in streaks, and what seems to happen to precipitate the end of a streak of normality?
There are a bunch of other things like these that you could try. If you've already done some, you could share them here.
Have you been collecting the data? There are all sorts of fun things you can do to make predictions but the hard part is collecting it in the first place. Sleep times, wake times. Then start adding in potentially relevant information - like food, exercise, subjective feelings of fatigue and social events. Extracting a pattern after that tends to be easy.
I think you're taking the fundamentally wrong approach. Rather than trying to simply predict when you'll be sleepy in the near-term, you should try to actively get your sleeping patterns under control.
Besides, having a tool that could forecast my sleep patterns given different variables would allow me to understand the interactions of those variables and ultimately would allow me to take control of my sleep patterns.
"I find it impossible to wake up at a consistent time every day (+/- 8 hours), despite years of trying"
In other words, I've tried everything else.
I suppose I could shop around for a doctor willing to prescribe modafinil for my sort of sleep problems. I have thought of trying it in the past, but that's pretty far off-label.
"Everything" includes having read all current medical literature, which all says that severe circadian rhythm disorders are basically untreatable, and having one sleep doctor basically give up. I could also try more sleep doctors, I suppose.
I suppose I could shop around for a doctor willing to prescribe modafinil for my sort of sleep problems.
Do it. Even if your underlying condition is incurable some of the symptoms can be managed. And Modafinil is outright brilliant for managing fatigue and supplying wakefulness.
I have thought of trying it in the past, but that's pretty far off-label.
Compared to the most prominent usages of modafinil (performance enhancement) your usage would be pretty damn close to the label all things considered. But forget the label. Tell the doctor whatever is convenient to make him sign stuff for you. (You do keep your doctors separate, right? The ones who give you actual useful advice and the ones you use as gatekeepers to the system. Lie to the latter.)
It doesn't sound unreasonable to me given the severity of your symptoms. But I'm not a sleep doctor.
Consider also that there are other ways to procure drugs like this, i.e., shady online vendors from overseas. Just make sure you do your research on the vendors first. There are people who have ordered various drugs from these vendors, chemically verified that the drugs were in fact what they were advertised to be, and then posted their results in various places online for the benefit of others. Bottom line: some companies are more trustworthy than others--do your homework. And obviously you should exercise due caution when taking a new drug without a doctor's consent.
My sleep is unpredictable. Not in a technical sense, but a colloquial one. To be literal, I have no idea how to predict my sleep. I just as often sleep through the day as I do through the night. My sleep itself, as far as a sleep study can tell, is normal. I can vaguely say, 60% confidence, if I'm likely to fall asleep in a given 3-4 hour period, and occasionally I will be fairly sure, 80% confidence, 6-10 hours beforehand, of a 1-2 hour period. I can similarly predict the length of my sleep (which is relatively normal--generally distributed 7, 8, 9.5, 13 hours at .1, .4, .6, .9).
My sleep is seriously disturbed. Without understanding the process behind my sleep, without being able to predict it days beforehand and understand the variables behind it, I find it impossible to wake up at a consistent time every day (+/- 8 hours), despite years of trying, which makes it extremely hard to hold down a job, or do dozens of other normal things. There could be a profession that I could make my sleep work with, but I'm still searching for it.
So I ask you readers: Is there some sort of pattern detecting thing, whose name perhaps includes something like "markov" or "kolmogorov" or "bayesian", that could automatically take a time series data and predict the next values based on an unknown, complex model?
So, I could like enter the times I go to sleep and wake up, and when I have caffeine or I exercise, and maybe other things, and it would puzzle out how my sleep works and forecast my next few sleep cycles?
To have an accurate tool like that would transform my life.
"Hidden Markov models" comes to mind, but at first glance I don't see how a sleep model would count as a Markov process, given that you have to factor in sleep debt, time of day (because of sunlight), and perhaps other variables. But then I know nothing about HMMs.
Also, this is my first post. Is this the sort of thing that goes better in LessWrong or Less Wrong Discussion?