  Bid optimization for online advertising from single advertiser's perspective
has been thoroughly investigated in both academic research and industrial
practice. However, existing work typically assume competitors do not change
their bids, i.e., the wining price is fixed, leading to poor performance of the
derived solution. Although a few studies use multi-agent reinforcement learning
to set up a cooperative game, they still suffer the following drawbacks: (1)
They fail to avoid collusion solutions where all the advertisers involved in an
auction collude to bid an extremely low price on purpose. (2) Previous works
cannot well handle the underlying complex bidding environment, leading to poor
model convergence. This problem could be amplified when handling multiple
objectives of advertisers which are practical demands but not considered by
previous work. In this paper, we propose a novel multi-objective cooperative
bid optimization formulation called Multi-Agent Cooperative bidding Games
(MACG). MACG sets up a carefully designed multi-objective optimization
framework where different objectives of advertisers are incorporated. A global
objective to maximize the overall profit of all advertisements is added in
order to encourage better cooperation and also to protect self-bidding
advertisers. To avoid collusion, we also introduce an extra platform revenue
constraint. We analyze the optimal functional form of the bidding formula
theoretically and design a policy network accordingly to generate auction-level
bids. Then we design an efficient multi-agent evolutionary strategy for model
optimization. Offline experiments and online A/B tests conducted on the Taobao
platform indicate both single advertiser's objective and global profit have
been significantly improved compared to state-of-art methods.
