As speculation continues over what caused the alleged rift between Prince Harry and the rest of the British royal family, it’s been revealed that the Duke of Sus𝑠e𝑥 apparently turned down a chance to stay at Buckingham Palace during his last visit to London.

According to The Times, King Charles invited his younger son to use the apartments at the palace when Harry went to London in May for the Invictus games, but the duke chose to stay at a hotel instead.

“I don’t know what more the King could have done. He offered Harry to stay at Buckingham Palace but that it seems even that wasn’t good enough,” a “well-placed source” told The Times. “It was far from clear what the duke’s plans were. When it came to it there was no space in the King’s diary in between treatment and other scheduled events.”

Earlier this summer, Harry spoke to Rebecca Barry of ITV News for the special Tabloids on Trial about the harassment he’s endured from the paparazzi. When Barry asked about whether his “determination to fight the tabloids destroyed the relationship with [his] family,” the duke was hesitant to elaborate for fear of “a torrent of abuse from the press” but noted it was a “central piece” of the situation. “I’ve made it very clear that this is something that needs to be done. It would be nice if we did it as a family,” he continued, adding, “I think everything that played out has shown people what the truth of the matter is. For me, the mission continues. But it has caused part of a rift.”

Harry also explained that because of litigations and threats made when wife Meghan Markle lived in the U.K., he doesn’t feel safe bringing the duchess or their two children back to his home country.

“It felt like harassment,” Harry said. “It felt horrible then. It feels horrible now.”