Some claim one of the speakers is a controversial Islamic and civic leader who's supported some radical views in the past.
While the event will be at a Garland ISD venue, the school district is just leasing it out and has nothing else to do with the conference.
Conference supporters say it's designed to promote peace and that it's anti-terrorist.
However, one of its speakers in particular has become a lightning rod for criticism.
The scheduled keynote speaker at Saturday's conference, Siraj Wahhaj, is an imam from New York, identified by federal authorities as an unindicted co-conspirator in the 1993 World Trade Center bombings.
Wahhaj denied involvement, but did defend convicted plotters, and a New York Post article says he called the FBI and CIA "the real terrorists."
The Garland conference is entitled "Stand with the Prophet In Honor and Respect." It's hosted by a group from Chicago, Sound Vision.
Its flier asks participants if they are "Frustrated with Islamaphobes defaming the Prophet," but also, if they are "Fuming over extremists like ISIS who give a bad name to Islam."
"I don't know if now is the right time," said former Republican Congressman Allen West. "Definitely maybe not anytime is the right time for this type of conference with these type of speakers."
West is now CEO of Dallas-based National Council for Policy Analysis, and critical of Garland ISD for leasing the center.
"There comes a point where we have to start looking and saying, is freedom of speech being turned into seditious speech, speech that can undermine our Constitutional republic and our fundamental principles and values?" said West.
Garland ISD officials say they've been getting dozens of calls and emails, many of which are from people who are misinformed and have been led to believe it's a school sponsored event when GISD has simply leased out the building as it does to many different groups.
"We have to remain non-discriminatory in who we lease the facility to," said Chris Moore with Garland ISD.
"This event is for promotion of peace, for the promotion of understanding," said Alia Salem with the Council for American-Islamic relations.
The Council of American-Islamic Relations is not involved in the conference, but CAIR's executive director calls the protests ironic that the conference seeks to combat hate and terror and that opponents are quoting Wahhaj out of context.
"Perhaps he does have some controversial things that he's said over the years and they're using that as their platform and their galvanizing tool to bring people who are fearful or who hate Islam and who hate Muslims to come and protest against this conference," said Salem.
Garland police say they have been asked to add additional officers for the Saturday event because of the threat that protestors will come and try to stop it.