The CW’s R-rated teen drama Gossip Girl is drumming up controversy once again, as Manhattan residents complain about crazed fans who have taken over the quiet neighborhood in hopes of catching a glimpse of the stylish cast.

On Tuesday, actress Hilary Duff — who is currently filming episodes as a guest star on the series — shot a dinner party scene at the National Arts Club in Manhattan’s Gramercy Park along with series regulars Penn Badgley and Leighton Meester. The shoot attracted hordes of screaming fans – much to the chagrin of local residents.

A source tells The New York Daily News: “Residents were complaining that they couldn’t even walk on the sidewalks because cables were lying all over the place. Plus, there were trailers and crazed fans everywhere.”