The city is situated between Lake Michigan and the western end of Lake Charlevoix, which drains into Lake Michigan through the short Round Lake/Pine River complex in the heart of downtown Charlevoix.

The Charlevoix South Pier Light Station marks the opening of the channel onto Lake Michigan. Charlevoix's Round Lake has been called the best natural harbor on Lake Michigan.

The only way to get from Lake Michigan to East Jordan, Boyne City and other sites on Lake Charlevoix by boat is via Charlevoix. As a result, much commercial, industrial, and recreational boat traffic passes through Charlevoix.

Charlevoix is named after Pierre François Xavier de Charlevoix, a French explorer who travelled the Great Lakes and was said to have stayed the night on Fisherman's Island one night during a harsh storm. It was during this time that Native Americans were thought to have lived in the Pine River valley.

Soon after its formation in the 1850s, Charlevoix entered into a short lived conflict with Jesse Strang, leader and namesake of the Strangite Mormons, and then king of Beaver Island. Relations between Charlevoix residents and the Strangites were often tense. In 1853, a gunfight broke out between the two groups as the townspeople refused to hand over a man who was called for jury duty on the island, an event known locally as The Battle of Pine River. When Strang was assassinated on June 20, 1856, many believed residents from Charlevoix to be responsible.