The traditional fireplace was a part of history that showed us some lessons in physics. When a fire was lit, the warm air creating the current up the chimney tended to also suck air from inside the house to feed the flames, making the house feel cold unless you were sitting directly in front of the fireplace. There's a reason why old buildings had so many of them!
The best way to deal with a fireplace these days is to have it outfitted with an airtight insert, or, if possible, have a vent installed close to the hearth opening to supply air for the fire. An airtight wood stove is a good alternative as well - you can use the existing chimney to run the new chimney for the stove.
Seal air leaks around fireplace chimneys, furnaces, and gas-fired water heater vents with fire-resistant materials such as sheet metal or sheetrock and furnace cement caulk.
Fireplace flues are made from metal, and over time repeated heating and cooling can cause the metal to warp or break, creating a channel for air loss. The older the fireplace, the more likely you're going to see air leaks.
It's best not to mess around too much with your flue assembly, given that there can be fire hazards and creosote to contend with. Consult with a fireplace mason or wood stove dealer who is WETT certified and ask about repairs and new inserts to cut down on leakage.
Note that air sealing alone doesn't eliminate the need for proper insulation to reduce heat flow through the building envelope.