In "mild" cases of self-intersection, with degenerate vertices when three consecutive points are allowed be on the same straight line and form a zero-degree angle, the concept of "interior" still makes sense, but an extra care must be taken in selection of the tested angle. In the given example, imagine point A to lie on segment BC. In this situation the angle ABC and its determinant will be 0, hence useless. A solution is to test consecutive corners along the polygon (BCD, DEF,...) until a non-zero determinant is found (unless all points lie on the same straight line). (Notice that the points C, D, E are on the same line and form a 180-degree angle with zero determinant.)
