This is weak on detail, I'm afraid, but I feel fairly sure that I've read accounts of situations in which the presence of a regular human activity has been inferred but is later proved to have been impossible.
Which is to say, there is a regularity to allotment work as there is most work: weekly, seasonal and soothingly repetitive; it lends itself to these scenarios:
Old Fred arrives, disappears into his dilapidated shed for a few minutes, comes out with wellies on and his pipe in his mouth, and sets to work thinning out his peas or what have you. You've seen his figure, heard the rattle of the lock that always sticks and caught a whiff of the smoke so often that it assumes the qualities of a 'background phenomenon' labelled 'pay no heed' in much the same way as mood music is simultaneously present yet imperceptible to consciousness. You might pay more attention to it all from time to time, but for the most part, you yourself are lost in the non-reflective consciousness of a repetitive task or three—like turning over the new patch for the lettuce while half listening to TMS.
And then one week you get home from the allotment to hear from your wife that Old Fred was killed in a road traffic accident earlier in the week—or perhaps it was a Great British heart-attack—but you can't shake the nagging sense that he was there among the rows of potatoes that very morning; at any rate, you could swear that someone was smoking his Old Holborn...
Of course, it's likely all a product of your pattern-recognition equipment making a lazy jump too many, but it's the mechanics of how and why it does so—and where the threshold lies—that can be so interesting.