The debate on Confederate symbols has focused on a statue of Jefferson Davis at the University of Texas at Austin.
Davis was the president of the Confederacy.
"It represents to me the enslavement the lack of rights for my people during that particular struggle," said Samuel Green, who attended the vigil.
Growing up during the Civil rights movement, Green says it's about time.
"My people died," he said. "Many of them died because of that institution, the institution of slavery."
Flying the Confederate flag on public grounds is a decades-old debate, but was renewed with the Charleston massacre.
Just last week, the U.S. Supreme Court sided with Texas in its decision not to issue specialty license plates with the Confederate battle flag.
The request for the plates was made by the Sons of the Confederate Veterans.
The Sons of Confederate Veterans, which has fought to keep the flag flying, issued a statement on Monday, saying, "Our heritage is under serious attack. Every Compatriot is urged to email the members of the South Carolina Legislature and urge them to keep the flag flying as a memorial to our brave Confederate dead."
Defenders of the flag say it's all about heritage and history. Those who want it removed say it's a symbol of hate.
Dr. Michael Landis, an assistant professor of American History at Tarleton State University, says it represents all three.
"And this is troubling for us to hear," said Landis. "It represents an independent country based on racial and class inequality. The Civil War was an attempt to form a slave-owning republic."
Dr. Landis says when people fly the flag and try to attach a different meaning to it, it creates problems.
"We can have conversations about states' rights, we can have conversations about politics, but to try to use these historical symbols out of their context distracts rather than illuminates," he said.