In 1821, the 11th earl was at the castle with his wife, awaiting the birth of their first child. According to the legend spread in the village by the midwife who attended Lady Glamis, the child was born extremely deformed, but otherwise healthy. Local suspicion was aroused, then, when the child's death soon after was announced. There were rumors of infanticide, which developed into rumors of the child being hidden away when people realized that there was no grave or headstone for the child found in the family's graveyard.
It became a widely-whispered secret that the 11th earl had locked up his rightful heir in some oppressive secret room rather than let the world see the "monster". If everyone thinks the child to be dead, then the earl's other, non-deformed children can take its place as heir.
Many Victorian visitors decided to investigate the matter while at the castle. People snooped around and claimed to see things, like a shadow walking along the battlements, or a hidden door leading to a long series of tunnels, or even seeing the monster himself, who was "half-human, half-toad." The only evidence to favor that there was some truth to the rumor was a visiting man who reportedly saw something and was soon after pressured into emigrating to Australia, the earl footing the bill.
In 1904, the New York Sun reported: "On one occasion a young doctor, who was staying in the castle professionally, found on returning to his bedroom that the carpet had been taken up and relayed. He noted that the mark [pattern] of the carpet was different at one end of the room. By moving the furniture and raising the carpet, he laid bare a trap door, which he forced open and found himself in a passage. This passage ended in a cement wall. The cement was still soft, leaving the impress of a finger. He returned to his room—and next morning received a cheque for his services with the intimation that the carriage was ready to take him to the station for the first train."
Whether it's true or not, even members of the Bowes-Lyon family got swept up by the rumors about themselves. On his deathbed in 1865, the 12th earl told his brother that, though he had tried to laugh down the story, he felt compelled to pray away its dark influence. When he died and the 13th earl inherited, his first order was to restore the family chapel, where he stayed up praying all night, reputedly for the sins the family had committed against its hidden member. There were a few instances of close family friends who refused to ever stay the night in the castle for reasons they refused to disclose to anyone else.
It was also reputed that on each (male) family member's 21st birthday, they would be initiated into the specifics of the secret in a clandestine family ritual (the women, of course, were too delicate to know anything more than the rumors everyone else knew). However, the 14th earl, having seen the detrimental effect the secret had on his relatives, sternly refused to have anything revealed to him. He wanted to let the secret die out. The "monster", if he ever existed, had been long-dead by this point, his secret chamber reputedly bricked up with the corpse inside. The 14th earl, therefore, saw no reason to keep the knowledge of the mystery going and didn't want his existence to be clouded by family shame.
While it's unclear if the family was ever haunted by more than just rumor and speculation, what is clear is how the later generations felt about it: Queen Elizabeth II's aunt, Rose Bowes-Lyon, who was born in the castle, has stated officially, "We were never allowed to talk about it when we were children. Our parents forbade us ever to discuss the matter or ask any questions about it. My father and grandfather refused absolutely to discuss it.”