Where does free speech factor into it? It's just a way of preventing private monetary interests from affecting the outcome of elections, and allow parties without access to wealthy sponsors to have the same chance of being elected as anyone else.
Money buys elecctions more easily when the masses are easily swayed. Say you have one candidate backed by corporate interests, one candidate backed by union interests, and one candidate representing the little guy with just the federal funding but he's allowed in debates. Isn't this a much better solution than limiting free speech.