Two techniques that make all the difference.
Use the drill press. Use the same type wood to make bungs. Drill the entire board of bungs, then don't snap out the bungs with a screwdriver. This messes up the edge of the bungs. Cut the bungs whole. Meaning use a bandsaw and cut the entire board off en masse with all the bungs being cut off at once.
A high quality bung cutter will give a slightly tapered fit when drilled home in the board. Make sure you set the stops up correctly to get a uniform bung. Makes a whole lot easier to have tight bungs.
DON'T sand the wood smooth. Keep the 'sawn' finish. This way you can see the cut of the bung as you grab it, slather it with glue and plug the hole. The 'saw cut' is at right angles to the grain. Make sure the bung is inserted with the grain parallel to the board. Nothing more obvious than a bung inserted sideways! After the bungs are dried, use the chisel to trim them. Don't try to trim them off in one swipe. Use two (or more) swipes, to determine how the grain is running. It is a 50% chance to get it right on the first swipe. Half the times the grain will chip off and go deeper into the bung, giving a lousy finish.
Also, make sure you buy a 'bung cutter' Not a tenon cutter. Bung cutters are shorter and provide a minimal (but distinct) taper.
A tenon cutter provides NO taper, which does not 'jamb' into the hole for a tight fit.
http://www.jamestowndistributors.com/userportal/show_product.do?pid=501
here's a video of the technique, but the Japanese saw is foreign to me.