Experiments such as this one are tedious, because there is no text interface to a commercial program such as Bridge Master or Bridge Baron. As a result, information regarding the sensitivity of GIB's performance to various parameters tends to be only anecdotal.

GIB solves an additional 16 problems (bringing its total to 64.4%) given additional resources in the form of extra time (up to 100 seconds per play, although that time was very rarely taken), a larger Monte Carlo sample (100 deals instead of 50) and hand-generated explanations of the opponents' bids and opening leads. Each of the three factors appeared to contribute equally to the improved performance.

Other authors are reporting comparable levels of performance for GIB. Forrester, working with a different but similar benchmark (Blackwood, 1979), reports 8 that GIB solves 68% of the problems given 20 seconds/play, and 74% of them given 30 seconds/play. Deals where GIB has outplayed human experts are the topic of a series of articles in the Dutch bridge magazine IMP (Eskes, 1997, and sequels). 9 Based on these results, GIB was invited to participate in an invitational event at the 1998 world bridge championships in France; the event involved deals similar to Bridge Master's but substantially more difficult. GIB joined a field of 34 of the best card players in the world, ea player facing twelve su problems over the course of two days. GIB was leading at the halfway mark, but played poorly on the second day (perhaps the pressure was too much for it), and finished twelfth.

The human participants were given 90 minutes to play each deal, although they were penalized slightly for playing slowly. GIB played each deal in about ten minutes, using a Monte Carlo sample size of 500; tests before the event indicated little or no improvement if GIB were allotted more time. Michael Rosenberg, the eventual winner of the contest and the pre-tournament favorite, in fact made one more mistake than did Bart Bramley, the second place finisher. Rosenberg played just quickly enough that Bramley's accumulated time penalties gave Rosenberg the victory. The scoring method thus favors GIB slightly.

Finally, GIB's performance was evaluated directly using records from actual play. These records are available from high levels of human competition (world and national championships, typically), so that it is possible to determine exactly how frequently humans make mistakes at the bridge table. In Figure 4, we show the frequency with which this data indicates that a human declarer, leading to the n th trick of a deal, makes a mistake that causes his contract to become unmakeable on a double-dummy basis. The y axis gives the frequency of the mistakes and is plotted logarithmically; as one would expect, play becomes more accurate later in the deal.

We also give similar data for GIB, based on large sample of deals that GIB played against itself. The error profiles of the two are quite similar.

Before turning to defensive play, let me point out that this method of analysis favors GIB
slightly. Failing to make an information gathering play gets reflected in the above figure, since the lack of information will cause GIB to make a double-dummy mistake subsequently.

But human declarers often work to give the defenders problems that exploit their relative lack of information, and that tactic is not rewarded in the above analysis. Similar results for defensive play appear in Figure 5.