If I'm reading it correctly, I think I have a theory ...
The location is in Highland County, Virginia - as remote a county as there is in the state, nestled among the Appalachian ridges and valleys adjacent to what is now West Virginia. Little Mountain is a long ridge running from roughly southwest to northeast. As such, specifying "Little Mountain" does nothing to pinpoint the location of the hermit's cave.
The author of this piece states he first encountered the hermit in 1862 - the "second year of the war", probably in August. In May of that year Stonewall Jackson won his first victory of his Shenandoah campaign in the Battle of McDowell. This battle occurred in Highland County, roughly one or two major ridges east of Little Mountain. This would correlate with the author's statement that a later sighting had been by an acquaintance who lived circa 10 miles away at / on Little Mountain if the author's own home were in the vicinity of McDowell.
Jackson's victory was obtained at the cost of twice as many casualties as the Union forces suffered. After the Battle of McDowell Jackson shifted his attentions east and south to the Shenandoah Valley. I haven't been able to locate any mention of substantial battles in Highland County after the 1862 Battle of McDowell.
The hermit wore a Confederate coat (tunic; whatever ... ) with thread stars sewn onto the collar. A single star on each side of the collar denoted a rank of major. If the collar had been black (which I can't confirm from the article) it would have denoted a major in the medical service (the only rank I find listed for a medical service officer's uniform). The Confederate Army commissioned circa 1,000 surgeons "... with rank, allowances, and emoluments of a major of cavalry." [1]
The only possession discovered at the hermit's cave after his death was a "beautiful" set of surgical instruments with the initials "A. T. H." ("A. T. M."?) engraved in the handles.
The article states that the hermit's body presented an old gunshot wound to the head, and the author suggests this was the cause of the hermit's odd behavior and muteness.
So here's my first cut at a hypothesis ... The hermit was originally a Confederate surgeon wounded at the Battle of McDowell, who wandered off into the mountains permanently dazed by his head wound but retaining enough sense to hang onto his most cherished possession - his monogrammed set of surgical instruments.
[1]
http://www.medicalantiques.com/civilwar/Civil_War_Articles/Confederate_Army_Surgeons.htm