I'd suggest that there are a few factors.
The extant public fear/thrill/unease over mental illness.
Like, a place where mentally ill patients were resident has been 'contaminated'.
The general thrill of large, rambling buildings.
Perhaps not haunted but inhabited by "hobos", junkies, and other ... urban explorers.
The general image of a medical facility.
Ventnor's (IoW) massive sanatorium which treated "chest diseases" such as TB, has been long gone, and is now a huge and wonderful public park and garden ... but "rumours" of the area being "haunted" persist.
I think it's mostly - not all - concerning the first factor. The taboo of mental illness. It's been used by many films and games* to inspire dread and thrill. This, combined with the natural association between a place where a lot of people died - such as battlefields - means that an old, rambling building, surrounded by local distaste and rumour, associated with medical death, is bound to have ghosts in. Isn't it? (#sarcasm)
There is good argument to note that locations involved with emotional events have many legends of ghosts. In this, ordinary homes where old folk died happy are ... open to phenomena. But a hospital has so much more "emotion"?
Why, then, do many people worry about cemeteries (of those already dead not killed in their precincts), or morgues?
* I'm not having a 'go' at games, of course. But, like cemeteries, they've become cliche with many genre.