In the early Middle Ages , Canton had a significant Muslim population as a result of its trade with the Middle East .
Canton never traded with the Middle East and consequently never had much of a Muslim population .