If you or someone you know has Ancestry.com, might be worth looking that place up in the Victorian censuses - although that is quite a few decades' worth - it won't take too long if there aren't too many other pubs on the street, (as they're not usually listed by name, just house number). I'd also run a search on a 19thC newspaper database but then I'm a nerd like that. I grew up in a 'badly' haunted house (is the only word I can think for it) and as I research for what I stupidly call a 'living', have often tried to find out about whoever lived in that house in the 19thC (as our ghost too, wore generic 'Victorian' clothing but nothing specific enough to have a rough idea of the decade and like you, I was only a kid when I saw it, so had no sense of clothing history yet). Try as I might, I can't find out about anything ever happening there - and according to the census, it was usually lived in by a farmer (what I think I probably saw, that or a labourer), his family and the occasional servant. But although it looked like a terraced house when I lived in it, we'd always known it used to be a farmhouse (the rest of the terrace got built along from it, until what had been a farm in a field became a street).
You may have more luck and find something. If the building was always a pub - things happened in pubs in the nineteenth century (fights, inquests, people about to be hung stopping by for a last drink, body snatchers used them as bases in Georgian times...
Also once saw a 'mucky' ghost - and he reminded me of nothing so much as the tramps I used to see in Leeds station when I was a kid. Those people were dirty on a whole new level - dirty like you never see now, not even homeless people look like tramps (and probably many people who weren't, in the past) used to look.
