Yes, a vent hole without a damper will cost you heating dollars. Generally they should be closed with dampers. However, if the vent hole is not an uncontrolled leak but a planned part of the system necessary to create a high neutral plane, then a damper should not be used. It will cost you much less than you think if the cold air leaks in the house are sealed and weather-stripped and the planned cold-air input in the basement is damper controlled. Besides, one way or another you have to have a minimum amount of air change in the house, so you might as well have it and its potentially damaging moisture under control.
If you are really economy-minded, you can change the ducting and run the exhaust air through a heat-recovery-ventilator -- HRV-- stealing back the heat but getting rid of the bad air.