“Can we talk about Peep Show now?” asks the 58-year-old with a relieved baritone laugh, after our lengthy discussion about his experiences as a classically-trained black actor in the entertainment industry and the horrors of Bridgerton, more of which later.

“Peep show was the funniest thing you could imagine,” he laughs.

“We just p****d ourselves until we could get a usable take. I wasn’t a professional funny man, so when David Mitchell was sitting in front of me on a toilet seat there was nothing I could do for 15 takes. And that’s how I spend my days on set – just laughing. I only do a few days every year but it’s the thing that most Americans recognise me for.”

After making his name on stage at the National Theatre and with the RSC, then in long-running hospital drama Casualty, in which he played staff nurse Mark Grace over 47 episodes, he was delighted to show audiences he had a “funny bone”.

A successful stage and TV actor, in 2009 he got within touching distance of mega-fame when he was in contention for the part of the 11th Doctor Who. The role ended up going to Matt Smith, but Paterson was close enough to winning the part to be called to audition.

“I was in South Africa, filming the Number One Ladies’ Detective Agency, and woke up to a phone call from my agent saying ‘Don’t answer the phone; they’ve got you two-to-one on Paddy Power as the next Doctor Who’.”Looking back on his Sliding Doors almost-moment, he quickly finds a silver lining.

“At least I didn’t have to do nine months filming in Cardiff,” he jokes, but you sense he would have loved to have been the first black Doctor. That accolade finally went to Ncuti Gatwa, 30, earlier this year.

Paterson, it seems, was slightly too far ahead of his time.

A mainstay of our screens now for three decades, with roles in such series as Green Wing, Survivors, Timeless and Noughts+Crosses, he’s also cut an elegant figure on the big screen in films including Danny Boyle’s adventure drama film The Beach (“four months filming in Thailand, very nice,” he smiles), starring Leonardo DiCaprio.

Soon, he’ll be seen in the film Wonka, a prequel to Roald Dahl’s Charlie And The Chocolate Factory, alongside Timothee Chalamet and Olivia Colman.

But the sort of mainstream recognition that many of his theatrical contemporaries like Simon Russell Beale have enjoyed has eluded Paterson.

“I took the work I thought was interesting, but the outcome for me of doing all that theatre wasn’t the same as it was for other actors of my generation, such as Simon,” he admits. “I wanted to be an actor, not a black actor. The black ‘box’ was something I wanted to avoid.”

And when he went for auditions with companies doing black-directed Shakespeare, he sensed he didn’t belong there either. “I felt like a flamingo – as if I was some aberration. It didn’t feel like home and I felt like I’d be an alien.

“I’d metabolised the whole world of classical theatre, not just the plays but the culture, I wanted to be in a costume drama. I wanted to wear a frock coat, but not as a background figure. I wanted to be the protagonist.”

Frustrated by the lack of roles for black actors, he decided to write a leading role for himself. In the process, he has inadvertently ended up writing a new life for himself.

His one-man play Sancho: An Act of Remembrance – based on the extraordinary life of former slave Charles Ignatius Sancho, who became a composer and man of social standing in 18th Century London, even painted by Gainsborough – was performed in 2015 to huge acclaim.

Born aboard a slave ship crossing the Atlantic, Sancho lived in the Spanish colony of New Granada.

When his parents died, the two-year-old orphan was brought to Britain and gifted to three sisters in Greenwich, south London, where he remained for 18 years before running away to join the household of the 2nd Duke of Montagu, who encouraged him to learn to read.

After spending time as a butler in the Duke’s household, Sancho started his own business as a shopkeeper, while also starting to write and publish essays, music, plays and books.

As a male property-owner, he was legally allowed to vote and – in 1774 and 1780 – he became the first known black Briton to have voted. A hero to the Abolition movement, Sancho died aged 50 or 51 in 1780.

And Paterson’s obsession with the famous Briton led him to turn his story into a novel. Written as the imagined 18th century diaries of Sancho – and already lauded by Stephen Fry as “an absolutely thrilling, throat-catching wonder of a historical novel” – it has introduced Paterson to a new world as Chancellor of Oxford Brookes University. He takes up the ambassadorial role next year.

“This is the by-product of investigating the life of a slave who became a composer in 18th Century London,” he says.

“And it happened because of words. I’m not an actor known for physical strength or beauty; I’m someone people remember for the lines I say. It’s clear I love words and that they mean something to me.” Talking to him, this is abundantly clear.

In the process of finding a character in history who could be “the leading character in my story”, he discovered how blackness has in large parts been “erased” from British history.

“British history is snow-blind, like the Truman show. History and history scholars have curated the story in a certain way to say, ‘This is what this place looks like’ pre-Windrush,” he says.

“But when you explore the 18th century, without the colour taken out, it’s all there and you see the actual truth. You find the slaves who built the industrial revolution; the cotton workers. I started studying, and found huge swathes of history that I didn’t realise contained black faces. Even before I discovered Sancho, I realised I’d been lied to.

“When you look at the thousands of portraits of the great and good, they’re full of negro children and servants. My education emerged through art.”

I wonder what he makes of Bridgerton and its alternate historical world, featuring an integrated mixed cast of white and black actors in prominent roles. It turns out, he finds the escapist colourblind fantasy disturbing rather than liberating.

“Bridgerton is a confection and a fiction,” he says. “I’m a little bit afraid [of it] having toured America with my one man show.

“The idea Britain had no slaves is very common over there and now they have the idea the British scene was all black people mixing in society… I understand the cartoon of it, but what enrages me is that Bridgerton is doing a disservice.”

Talking with this erudite and educated man, it is hard to believe his humble beginnings as the son of a cleaner. “However articulate I am now, if you’d met the mumbler at 14 you wouldn’t have recognised me,” he says.

“The mumbler” is his nickname for his younger self, a boy who “had a social stutter when it came to expressing myself,” and played truant for the best part of two years, spending his time in the local library instead of class.

Until three, he spoke a Caribbean dialect (his parents were emigres from St Lucia to London, where he was born) and little English. “I would search for language, and get it wrong. I know how delicate it can be. My shyness came from insecurity.”

His first name didn’t help.

“I would pronounce the ‘T’ in Paterson, and people would call me posh. I was speaking properly because I listened to the radio a lot!”

His love of books proved his salvation, and for this reason he supports the work of Give A Book, with whom the Daily Express has partnered in our Christmas campaign to raise money to get books into the hardest-to-reach places.

“I always liked words,” he continues. “When I was 12, I picked up The Lion, The Witch and The Wardrobe. I was in my little box room in Kensal Rise and I opened this book and disappeared into it.

“I could feel it transporting me through the wardrobe, feeling the coats, and seeing the snow, and loving Aslan the lion, and crying when he died. Just for a few hours I was away from all the nonsense.

“It was so pure – the magic of words to make you believe.” From then on, he read constantly.

“Good books or bad books, everything from Oscar Wilde to Mills and Boon. The library was a place to find magic for free.”

At college he got into youth theatre, and later trained at drama school. “I found my future, and I found my people. Art is like that.”

Now, all we need is to see him cast in a screen version of his Sancho diaries…

• The Secret Diaries of Charles Ignatius Sancho by Paterson Joseph (Dialogue Books, £16.99) is out now. For free UK P&P, visit expressbookshop.com or call 020 3176 3832