Peering through a magnifying glass beneath his signature deerstalker hat, sucking on a blackened clay pipe, Sherlock Holmes remains the world’s greatest literary detective, immortalised in more than a century of books, films and television. The Baker Street sleuth’s preternatural powers of observation and a rigorously logical and scientific mind mirrored those of his creator, Sir Arthur Conan Doyle.

Yet the greatest mystery of Conan Doyle’s own life has remained unsolved – until now.

At the height of his intellectual powers the author stunned his friends and admirers by embracing the paranormal, trying to communicate with the dead, and declaring his belief in fairies.

“The recognition of their existence will jolt the material 20th century mind out of its heavy ruts in the mud, and will make it admit that there is a glamour and mystery to life,” wrote Conan Doyle in 1920.

It was a shocking assertion from the razor-sharp mind that had crafted the unerringly analytical stories that for the past 136 years have made Sherlock Holmes an unparalleled sleuthing genius admired across the globe.

Conan Doyle’s embrace of fairies came just a few short years after he had shaken London society by declaring his belief in spiritualism, endorsing mediums who claimed to communicate with the deceased in the Great Beyond. “Life is infinitely stranger than anything which the mind of man could invent,” as Sherlock tells his loyal companion Dr Watson in A Case Of Identity.

For the past century Conan Doyle’s credulous faith in fairies and the spirits of the dead speaking from the afterlife has been dismissed as the wishful thinking of an overactive imagination. But a new book, The Real Sir Arthur Conan Doyle, shockingly claims that the creator of the great fictional detective suffered mental illness inherited from his alcoholic father. “It’s long been a mystery why the mentally rigorous Conan Doyle would claim to hear voices, see the dead, and believe in fairies,” says writer Andrew Norman, of Poole, Dorset, who like the author trained as a medical doctor before becoming a writer.

“Conan Doyle experienced hallucinations and delusions, probably as a feature of an inherent tendency to mania. He may well have inherited his delusional disorder from his father Charles, who had exhibited many of the features of a schizophrenic.”

Charles Doyle, a talented artist who spent the last 14 years of his life forcibly incarcerated in mental and medical institutions, may have suffered bipolar disorder, acute schizophrenia, dementia, psychosis and epilepsy, according to Norman, who examined long-forgotten family medical records and Charles’ private diary.

Charles also suffered hallucinations and delusions, “said he was getting messages from the unseen world,” and like his famous son displayed an “intense interest” in goblins, fairies and elf-like creatures.

“Charles was hearing voices – which is indicative of a serious and inherent psychiatric disorder, but which also is instantly reminiscent of the spiritualistic experiences that his son, Conan Doyle, would describe in his later life,” says Norman.

The fear of following his father’s descent into madness haunted the author.

“I think he spent his entire life afraid he might succumb to the same mental illness as his father Charles had done,” says Norman. “Yet when mental illness did finally overcome Conan Doyle he failed to recognise it.”

London-born of Irish ancestry, Charles moved to Edinburgh, and with Scottish wife Mary had ten children, seven surviving to adulthood. He drew the illustrations for his son’s first Sherlock Holmes novel in 1887, A Study In Scarlet, but was plagued by alcoholism and advancing mental illness. Charles’ heavy drinking triggered epileptic seizures, and a likely bipolar disorder led to depression and manic episodes.

“Charles, who displays traits characteristic of both bipolar disorder and schizophrenia, is said to be suffering from a schizo-affective disorder,” which led to his alcoholism, says Norman. His father’s violence and paranoia kept Conan Doyle at boarding school for much of his childhood, even during the holidays. “Charles was dangerous when he got into these psychotic rages,” says Norman.

Despite the mental challenges his father faced, Conan Doyle’s admiration for him was undiminished, saying his father’s life was “full of the tragedy of unfulfilled poets and underdeveloped gifts”.

Charles spent years locked away in the Montrose Royal Asylum, the Royal Edinburgh Asylum, and Crichton Royal Hospital in Dumfries, where he died during an epileptic fit in 1893, at the age of 61. Like a literary detective, Norman finds clues to Conan Doyle’s mental history hidden in his writings, including many of his Sherlock Holmes mysteries.

In The Adventures Of The Cardboard Box, Conan Doyle wrote of “an impulsive man… prone to occasional fits of hard drinking,” who admits he “becomes like a madman when my temper gets loose”.

In The Empty House, Sherlock suggests that an adversary suffers a hereditary mental illness. And in The Sign Of Four, Holmes uses cocaine to sharpen his mind, prompting Norman to wonder if this was a result of Conan Doyle seeing his alcoholic father possibly addicted to opiates. Charles’s wife Mary wrote that not booze but “undue use of stimulants” plunged her husband into madness.

Conan Doyle, a lapsed Catholic, in his later years apparently lost the mental rigour that in his youth prompted him to state: “Never will I accept anything which cannot be proved to me. The evils of religion have all come from accepting things which cannot be proved.”

But the horrors and slaughter of the First World War unleashed a flood of adherents to spiritualism, with many in British society attending seances, consulting clairvoyants, and trying to contact their dearly departed.

“It is treacherous and difficult ground, where fraud lurks and self-deception is possible,” admitted Conan Doyle. Yet his demand for logic and proof faded in the face of his inherited mental illness, Norman believes.

“Conan Doyle wanted to believe in fairies and communicating with the deceased, and he was suffering delusional disorder, seeing things that weren’t there.”

Yet nothing prepared Conan Doyle’s admirers for his 1922 book The Coming Of The Fairies, in which he revealed his faith in the magical winged creatures, elves and gnomes.

He was completely convinced of the authenticity of a series of photographs of fairies frolicking in an English glen, taken by nine-year-old Frances Griffiths and her friend Elsie Wright, aged 16, behind their home in Cottingley near Bradford in 1917.

The Cottingley Fairies may have been dismissed as a childhood prank coupled with clever photographic manipulation, but became world famous when Conan Doyle hailed the images as proof that fairies were real.

Photographic experts at Kodak warned Conan Doyle that the Cottingley fairy pictures could have been “artificially” manipulated by “some clever operator,” but he chose to believe only the expert who told him the snaps “are entirely genuine unfaked photographs of single exposure, open-air work”. Predicting a new era of fairy-human amity, he said: “These little folk, who appear to be our neighbours, with only some small difference of vibration to separate us, will become familiar.”

Ridiculed, Conan Doyle died in July 1930, aged 71. It was another 51 years before Frances Griffiths confessed that the fairies had been illustrations cut out on cardboard and suspended in the bushes with hatpins. “Conan Doyle’s grandiose delusions made him abandon his logical intellectual rigour,” adds Norman. “He was suffering from a disorder, and understanding that is the key to unravelling the mystery that overshadowed his later years.”

• The Real Sir Arthur Conan Doyle by Andrew Norman (White Owl, £20) is out now. Visit expressbookshop.com or call Express Bookshop on 020 3176 3832. Free UK P&P on orders over £25.