If you already know what a public good is, and a bit about the theory of market failure, skip down to And now for my point.
For the rest of you who want an explanation of public goods and "market failure", and don't want to look at the Wikipedia article I list above, a Public Good is an economic good (that is, a product, service, resource, etc.) that has unusual properties that lead us to believe that a free market might underproduce it.
Specifically, it has to be:
Goods that possess both these features are said to have a problem, which is called "market failure". It is said that the free market will not supply as much of the good as would be truly "efficient" (in the economic sense of the word), because suppliers will not be compensated as much as the "real" demand curve for the good would imply. (See this Wikipedia article if you aren't familiar with supply and demand curves.)
Put another way, if you could exclude people from the benefits of a good, people would be forced to pay for it if they wanted it, but since they can enjoy the benefit without paying, and since one person's consumption does not impact another's, the exhibited demand curve is much lower than the "true" curve, and thus supply will be much lower than might be abstractly thought of as "efficient".
Note that you need both properties for something to be a Public Good. A movie theater has a door that locks, and is thus excludable -- I can keep you from entering if you will not pay. There is not an infinite supply of fish, and thus deep sea fishing is (at the limit) not non-rivalrous. (Fishing might exhibit a different kind of market failure popularly called "The Tragedy of the Commons", but that's a story for another day.)
An example that is often given in economics texts for a Public Good is a lighthouse. Everyone benefits from it, but since it is non-excludable why should I pay for it if someone else will? Thus we would naively expect there to be an undersupply of lighthouses. Another example given is raising honeybees — the bees help nearby farmers, but because they can't be stopped from going to any field in the vicinity, you would naively expect that beekeepers would be under-compensated for their work and thus there would be an undersupply of bee hives.
As it turns out, both of these examples are historically false. Lighthouses were historically supplied by private means, and a lively market exists in renting the use of bee hives to pollinate crops. (See the collection "The Theory of Market Failure" if you are interested in details on both these "textbook fallacies".)
As it happens, I don't believe there really are any public goods (or at least, no "market failures" important enough that we should care much about them). However, let us assume that there might be a few. It is argued that one of the functions of government is to "fix" the market failures that a pure free market might have by intervention. For example, lets say that we believed that national defense was a "public good". The government could then provide the good directly (such as by collecting taxes and running an army with them), or could use subsidies or similar mechanisms to "correct" the market failure.
And now for my point.
For the government to actually fix the so-called "market failure", it has to do two things.
First, it needs to somehow assess what the "true" demand curve is. How might it go about doing that? Is there be some amazing scientific method for figuring out the "true" demand? Unfortunately, as Von Mises and Hayek pointed out in their work on the so-called "calculation problem", there is no particularly good way to figure out appropriate supply and demand curves without resorting to market mechanisms.
In practice, then, we end up with the "calculation" being made fairly arbitrarily, as the result of policial mechanisms. Sadly, as James Buchanan demonstrated in his pioneering work on public choice economic theory, the political method is likely to base its "computation" on the interests of powerful actors in the process rather than on any sort of rational basis. Bureaucrats will have a personal interest in the expansion of their fiefdoms, and thus will always argue for increased production of a good. Firms seeking government contracts will have an enormous incentive to lobby for increased production. (Indeed, if contract worth a billion dollars in profit is in danger, why not spend $900M lobbying if it will retain your business?) Individual members of the public, however, each have only a tiny fraction of the cost of any given government program to bear, so one's personal incentive to lobby against any given program is low. On a billion dollar federal budget item, the average American can save only $3.30 by getting the program canceled — the sum hardly makes the effort worthwhile.
We therefore expect that the government will not make rational decisions about the allocation of a public good, but will instead tend to overspend on it — perhaps even vastly overspend on it.
Second, to fix the "market failure", the government must somehow actually act to supply the missing good, either directly or via government contracts. Because there is no market discipline enforcing the efficient delivery of government services, these services are often supplied in a stunningly bad manner. You can't go to the competing DMV — there is none — so you wait on line for hours to get your drivers license. Why should we expect that the efficiency with which, say, national defense or other purported "public goods" will be supplied would be any greater?
So, here is the crux of the problem with the "let the government supply the public goods" argument: there is no evidence the government can supply putative "public goods" with any greater efficiency than the market that has "failed". Indeed, one might even get less efficiency than one started with. Why, then, is government intervention any better than the "market failure" we started with?