The quote can be found in Henry David Thoreau's essay entitled Civil
Disobedience published in 1849. However the quote is often attributed
to Thomas Jefferson, but there is no writing suggesting Jefferson
actually said it.
[Edit: Indeed Jefferson did NOT coin this phrase. This motto
(originally "The best government is that which governs least") was in
fact that of the monthly periodical of the mid-19th century called
"United States Magazine and Democratic Review," it is a most excellent
read too. Archives can be found in the Cornell Library.