The equerry tried, and failed. It could have been the end of his royal career, but it wasn’t. For Jamie Lowther-Pinkerton went on to be private secretary to Prince William and Catherine, as well as Prince Harry for a time, and was invited to become godfather to Prince George.

But that lunch was a baptism of fire for the former Guards and SAS officer, learning the ropes of how life was lived in the royal court, both the private side and the public.

Sometimes described as the “men in grey suits”, being a courtier is like walking a tightrope – be too obsequious and you are despised; too frank and you’re fired.

Lowther-Pinkerton understood that delicate balancing act which was to prove so helpful to William and Harry in their younger days. “He was honest and straight with them,” a colleague revealed to royal author and journalist Valentina Low.

“He could have conversations with effing and blinding in them. He could be the perfect courtier, and then there’d be the glint in his eye that suggests that he could rip your head off, literally. He had a big vision of what he thought William and Harry could do, and how they could achieve it.”

Indeed, Lowther-Pinkerton learnt – as our courtiers throughout the centuries have discovered – that you win some, you lose some.

Low’s new book, Courtiers, chronicles the ups and downs of these dedicated men and women who give service to the Royal Family, often with little thanks and practically no job security, who bolster our kings and queens but who remain forever in the shadows behind their illustrious employers.

It is a boxful of delights, stuffed with jokes and anecdotes, and insights into what really goes on behind closed palace doors.

It also details at some length, and with absolute authority, the behind-the-scenes dramas surrounding the exit from royal life of Prince Harry and Meghan Markle. If ever courtiers were put through the shredder, it was the Sussex’s brigade.

The names of Ed Lane Fox, Sam Cohen, Sarah Latham, and Jason Knauf have now disappeared, but they were the people who laid their reputations, and sometimes their sanity, on the line in their attempts to keep the increasingly fractious royal couple happy.

Low, in his day job as a royal correspondent, outlined the behavior which came with the arrival of Meghan Markle into the royal fold, the slow peeling away of staff, from nannies to senior courtiers, as the difficult couple took full advantage of the privileged life accorded to senior royals.

We now know that many of Harry and Meghan’s complaints, later aired in the Oprah Winfrey interview, were manufactured – “simply not true”, writes Low.

He quotes Meghan saying: “There’s the family, and there’s the people that are running the institution. Those are two separate things.” All nonsense, of course.

The couple was angry at being hemmed in by the very courtiers trying to protect them.

Before long they chose to escape to California, leaving behind a bewildered Royal Family, a saddened group of discarded staff, and a trail of damage that will take decades to repair.

But these tales form only part of Low’s delicious feast of previously undisclosed royal tales.

Who knew, for example, that it took a Buckingham Palace official to ease the nerves of King Bhumibol of Thailand by suggesting His Majesty stand on his head before an important audience?

His Majesty duly obliged. Or that, when she read through her famous “I will serve you all my days” speech for the first time – written by a courtier – the then 21-year old Princess Elizabeth burst into tears?

Or that private secretary, Sir Martin Charteris, was in love with the Queen?

Courtiers, it has to be said, are an odd bunch. Low describes them as “the hidden power behind the crown” and in some respects they are – their job to rein in erring princes, to coax kings and queens to do things they don’t want, and to defend against attack even when secretly they know the royals were in the wrong.

They seek power, they wield power, and generally, observes Low, have been educated at Eton (though that is changing in the pre-generation) life sent generation). But it’s not a job for life, as the Queen’s private secretary Christopher Geidt recently found out.

In the estimation of many Geidt, effectively chief executive of House of Windsor Plc between 2007 and 2017, was the most successful in that job in several decades.

Yet he was unceremoniously booted out after Prince Charles and Prince Andrew – each with a personal axe to grind – joined forces to see him off.

The Queen, according to one former member of the household, had not the will, at the age of 91, to resist. Yet Geidt was the only sovereign’s private secretary in a century to have been fired. The previous one, the devoted Francis Knollys, moaned in 1913 that he’d been sacked “exactly as an unsatisfactory butler might be dismissed”.

Prince William, writes Low, was furious.

Closer to the Queen than his father, he felt Geidt had done a great job – thus stoking tensions between Kensington Palace and Clarence House which have had to be put to accession one side since Charles’s accession.

If Knollys thought he was treated like a butler, another sovereign’s private secretary, Sir Michael Adeane, equated his task to that of a nanny: “One day you’re writing a letter to the PM. The next you’re holding a small boy’s raincoat.”

But Adeane, who served the Queen from 1953 to 1972, took it in good part.

A livelier courtier altogether was Sir Martin Gilliat, who served as the Queen Mother’s private secretary for 37 years. A distinguished former soldier, he’d been captured during the Second World War and incarcerated in Colditz, the notorious prisoner of war camp.

One day he found himself on a ferry, surrounded by tourists. “Where are you from?” he asked politely. “Oh, Germany – yes, I love Germany! I have spent a lot of time in Germany – wonderful historic houses. Yes, there was one particular castle I just couldn’t tear myself away from.”

It was said without a trace of bitterness, though he’d been a prisoner for almost the whole of the war.

Gilliat was fun, writes Low, although his responses could be unpredictable, given he was an Eton and Sandhurst man. On a tour of Venice with the Queen Mother, the BBC’s Kate Adie, commented: “Isn’t it wonderful, Sir Martin, all these churches?”

He looked at her and in a voice just loud enough for the Queen Mother to hear, said, “Kate, if you’ve seen one f****** church, you’ve seen the lot.”

Her Majesty, looking on, was amused.

Another royal favorite was Brian McGrath, who served Prince Philip in top positions between 1982 and 1995.

McGrath – against the rules, because Buckingham Palace was a corgi-only zone – used to take his labrador, Robert, to work.

One day, returning to his office across the Palace forecourt, he saw the Queen’s car coming out. He stopped and stood to attention, the dog sitting obediently beside him. As the Queen was driven past, she looked out of her window and bowed – not to McGrath, but to the labrador.

Courtiers come in all shapes and sizes, submitting themselves to what political theorist Harold Laski described as “dignified slavery”. Dull indeed was Sir Ralph Anstruther, always stuffily dressed in starched white collar and highly polished lace-up shoes (he dismissed shoes without laces as “bedroom slippers”).

Duller still was Sir Philip Moore. “He bored the Queen stiff,” records Low. Her Majesty was stuck with Moore popping his nose around the door, daily, for nearly ten years as her private secretary during the 80s.

But it also worked the other way around. Pity the poor private secretaries to King

George VI – grandfather of the present king – who was snobbish, irascible and awkward.

“He made it quite clear he did not want to have his life dictated to by his private secretary,” writes Low. Yet his daughter, Queen Elizabeth II, instantly recognized life would be easier if she did just that. She must have been a dream to work for.

That’s not to say Elizabeth wouldn’t put her foot down when she wanted to. The vexed question of whether she should pay income tax or not – which came to a head with the 1992 Windsor Castle fire – was one she felt deeply about.

Queen Victoria had paid tax, and her son Edward VII did, though he tried to get out of it. But both George V and George VI pushed for exemptions and, by the time Elizabeth came to the throne, she paid no tax at all.

Public opinion began to rebel at this inequity, and her private secretary Sir William Heseltine told her she should pay up. He got nowhere. “The resistance came from the Queen herself.

“Her father had told her if they were going to be taxed, they wouldn’t be able to afford to run the show. It needed the Windsor fire [for things to change],” Heseltine told the royal author. Her Majesty, it emerges, wasn’t always right.

Valentine Low, a master of his craft in prose and detail, writes about these palace officials with the wisdom of long experience.

That he chooses to describe Prince Andrew’s dealings with staff as “grotesquely unpleasant” is a sharp reminder that life inside the palaces isn’t all tiaras and curtseys, but can be as bloody and brutal as the Tudor court.

Usually, it’s the courtiers whose heads roll.

• Courtiers by Valentine Low (Headline) is out now. To order for £18 with free UK P&P, visit expressbookshop.com or call 020 3176 3832