Some children can have very vivid imaginary friends, to the point that the "friends" seem to have a life of their own. I suppose if friends exist, then enemies, or at least sinister figures, can exist too.
Incidentally, your description of the Top Hat Man does sound like the Phantom Raspberry Blower of Old London Town from the comedy serial off The Two Ronnies, broadcast in the 70s but repeated in the 80s too. He's kind of notorious for scaring kids even though he was supposed to be funny. Could he have inspired the look of the brother's "imaginary" enemy?
We must have an imaginary friends thread around here, if anyone can find it... might make you think,
@Seasidepagan ?