On Wednesday night, music legend Paul McCartney returned to New York’s famous Ed Sullivan Theater 45 years after his first appearance on The Ed Sullivan Show sparked “Beatlemania” in America. During his chat on the Late Show with David Letterman, Paul McCartney opened up about befriending and later drifting apart from the late Michael Jackson.

Paul said he first met Michael in the early ’80s, after the star called asking if he wanted to make some hit songs together.

“You know, he came to my house and we got to know the family and stuff,” Paul told Dave. “We had a really good time. We made a couple of records together ["Say Say Say" and "The Girl is Mine"], did a video and were very good friends.”

The Brit admits that the friendship was never quite the same, though, after Michael bought the Beatles music catalogue and decided not to sell it to Paul.

“He kept saying, ‘That’s just business, Paul, you know,’ so I went, ‘Yeah, it is,’ and waited for a reply. But we never kind of got to it and I thought, ‘Mmmmm,’ so we kind of drifted apart. It was no big bust-up. We kind of drifted apart after that. But he was a lovely man, massively talented and we miss him.”