

![0_image_0.png](0_image_0.png)

Figure 6: Screenshot of the tool for entering constraints. The constraint displayed forbids teammates have previously been given information about the user's attendance at the meeting.

language provides the entities a simple way to inform the agent of their specific properties and needs. An entity can use a constraint to forbid the agent from entering specific states or performing specific actions in specific states.  Such constraints can be directly communicated by a user via the tool shown in Figure 6. For instance, in the figure shown the user is forbidding the agent from autonomous action five minutes before the meeting. We define such forbidden-action constraints to be a set, Cfa, where each element constraint is a boolean function, c fg : S × A → { t, f} .  Similarly, we define forbidden-state constraints to be a set, Cfs, with elements, cfs : S → {t, f} . If a constraint returns t for a particular domain element (either state or state-action pair, as appropriate), then the constraint applies to the given element. For example, a forbidden-action constraint, cfg, forbids the action a from being performed in state s if and only if cfa (s, a) = t.

To provide probabilistic semantics, suitable for an MDP context, we first provide some notation. Denote the probability that the agent will ever arrive in state s f after following a policy, P, from an initial state s, as Pr(s, S, S, FIP). Then, we define the semantics of a forbidden-state constraint cfs as requiring Pr(s; - sf [P) = 0. The semantics given to a forbidden-action constraint, c fg , is a bit more complex, requiring Pr(s; $s fAP(s f )=a(P) = 0
(i.e., cfg forbids the agent from entering state sf and then performing action a ). In some cases, an aggregation of constraints may forbid all actions in state s f . In this case, the conjunction allows the agent to still satisfy all forbidden-action constraints by avoiding sf
(i.e., the state sf itself becomes forbidden). Once a state, sf, becomes indirectly forbidden in this fashion, any action that potentially leads the agent from an ancestor state into s f likewise becomes forbidden. Hence, the effect of forbidding constraints can propagate backward through the state space, affecting state/action pairs beyond those which cause immediate violations.