  As Natural Language Processing (NLP) systems are increasingly employed in
intricate social environments, a pressing query emerges: Can these NLP systems
mirror human-esque collaborative intelligence, in a multi-agent society
consisting of multiple large language models (LLMs)? This paper probes the
collaboration mechanisms among contemporary NLP systems by melding practical
experiments with theoretical insights. We fabricate four unique `societies'
comprised of LLM agents, where each agent is characterized by a specific
`trait' (easy-going or overconfident) and engages in collaboration with a
distinct `thinking pattern' (debate or reflection). Evaluating these
multi-agent societies on three benchmark datasets, we discern that LLM agents
navigate tasks by leveraging diverse social behaviors, from active debates to
introspective reflections. Notably, certain collaborative strategies only
optimize efficiency (using fewer API tokens), but also outshine previous
top-tier approaches. Moreover, our results further illustrate that LLM agents
manifest human-like social behaviors, such as conformity or majority rule,
mirroring foundational Social Psychology theories. In conclusion, we integrate
insights from Social Psychology to contextualize the collaboration of LLM
agents, inspiring further investigations into the collaboration mechanism for
LLMs. We commit to sharing our code and datasets (already submitted in
supplementary materials), hoping to catalyze further research in this promising
avenue (All code and data are available at
\url{https://github.com/zjunlp/MachineSoM}.).
