Minerva's hand passed over one of those objects, the one with golden wibblers, her eyes closing briefly.

Heh. From Chapter 17:

Oh, the little fiddly things?" said Dumbledore. "They came with the Headmaster's office and I have absolutely no idea what most of them do. Although this dial with the eight hands counts the number of, let's call them sneezes, by left-handed witches within the borders of France, you would not believe how much work it took to nail that down. And this one with the golden wibblers is my own invention and Minerva is never, ever going to figure out what it's doing."

That's been bothering me ever since chapter 104 came out. It doesn't seem likely that Quirrel would have just missed that, so I see a couple possibilities:

1) Part of Quirrel's plan somehow requires Harry to use the time turner once, so by having Harry go back five hours he limits Harry's use of it as much as possible.

2) It's a trap. Leaving via time turner is an excellent escape mechanism, and Quirrel knows that Harry has a tendency to overuse it for that purpose. So if Harry does manage to do something unexpected, Quirrel's hoping he'll go for the time turner--only to find that it doesn't work.

Why doesn't it work? Because Quirrel wrote the note at 12:04, went back via his own time turner, and handed the note to the anonymous Hufflepuff*. Thus, if Harry were to activate his time turner for the sixth time at 7:00, it would fail in some fashion ("for you couldn't send information further back in time than six hours, not through any chain of Time-Turners" -- chapter 61).

This is a pretty narrow window, though. Quirrel would have to anticipate that Harry might immediately go back from 11:04 to 6:04 once he got the message, so Quirrel wouldn't have written the note any later than 12:04 (if he wrote it at 12:20, for example, and Harry tried to go back to 6:04, it would fail and he'd know something was up.)

Which means that the forbidden period can't end any later than 6:04, and the absolute earliest Harry would want to use the sixth turn would be 6:49 when the whole thing started. So there's a 15 minute period from 6:49 to 7:04 where Harry thinks he can use the time turner but can't. This would be fine if the whole process were to finish in 15 minutes, but that seems unlikely, especially with Quirrel dawdling in chapter 107.

*It's amusing to think that Quirrel went back to 11:04 to hand the note to the Hufflepuff, whose quote "I know you said not to talk to you, but -" would have, if he had not been interrupted by Harry, ended something like "but I don't see why you gave me a note to just hand back to you ten seconds later. Hey--the version of you that handed me the note is still right there under the bleachers, waving at me and smirking." But really, there's no reason Quirrel would have taken that risk when he could have just gone back further and handed the note to the Hufflepuff "earlier".