Patrick Swayze and Jennifer Grey danced each other’s hearts in the 1987 cult classic Dirty Dancing, but the late actor definitely didn’t have the time of his life on the set of the film that made him famous.

On the pages of his memoir The Time of My Life — available Tues., Sept. 29 — Patrick reveals that the co-stars didn’t get along quite so well when the cameras weren’t rolling.

“We did have a few moments of friction when we were tired or after a long day of shooting. [Grey] seemed particularly emotional, sometimes bursting into tears if someone criticized her,” Swayze says of Grey’s needling mood swings. “Other times, she slipped into silly moods, forcing us to do scenes over and over again when she’d start laughing.”

“I was on overdrive for the whole shoot – staying up all night to do rewrites, squeezing in dance rehearsals, shooting various scenes – and was exhausted a lot of the time,” he recalls. “I didn’t have a whole lot of patience for doing multiple retakes…”

Nonetheless, Jennifer managed to impress Patrick with her performance. He credited her with doing “truly phenomenal job,” adding that “in many ways, she made the movie.”

Swayze went from Broadway dancer to Hollywood star in box-office hits like Dirty Dancing and Ghost. He died on September 14 after a nearly two year battle with pancreatic cancer.