President Barack Obama spoke openly about whether he thinks racism is behind recent political attacks against him on the Late Show with David Letterman on Monday.

Referring to Republican Rep. Joe Wilson’s “You lie!” outburst as Obama addressed health-care reform to the nation two weeks ago, Letterman asked Obama if he believed former President Jimmy Carter’s assertion that the South Carolina congressman is racist.

“Well, first of all, I think it’s important to realize that I was actually black before the election, so…This is true. This is true,” the President laughed. “I actually think that what’s happened is that whenever a president tries to bring about significant changes, particularly during times of economic unease, then there is a certain segment of the population that gets very riled up.”

He continued: “You know, one of the things you sign up for in politics is folks yell at you, but I think that what has been missing from the conversation is that the overwhelming majority of people, Republican or Democrat, I think they just want to see some common sense, they want to see some honesty and integrity in Washington. I think they’re turned off by the shouting and the yelling, and they expect more from their public elected officials.”