I agree with Thomas analyses.

Concerning the difference2 case, the two points are explained by the vertical line at x = 5.0 which comes from the outer shape. The internal representation is a BSP tree and one of this part of the outer boundary creates an hyperplane that splits the inner triangle. When the boundary representation is rebuilt, the two segments are glued together and the points appear there. There is no post-processing that simplifies the representation afterwards.

Concerning the circle test, I guess you mixed the arrays. What is really in the code is that the vertices2 array is build first from outer circle and the vertices1 array is built afterwards from inner circle. So you are really subtracting a big disk from a smaller one. As Thomas explained, computing set2 minus set1 give the expected two boundaries. Another possible change is to build the circles clockwise instead of counter-clockwise, and in this case the two regions are infinite wich a whole at the center, then subtracting set2 from set1 returns a disk with a hole.

There are two test cases here. The first "testdifference1" produces 3 polygons, two of which are correct (one outer polygon and one inner hole polygon). The third polygon is a rectangular region in the lower left corner of the outer polygon and should not be there.

The second test case "testdifference2" is a simpler example. Technically, the results are not incorrect, but they are strange. This example is included because it seems interesting and may help understand the problem. The inner resultant polygon has two extra points that are collinear with two edges on the polygon. The extra points don't change the shape of the polygon, but they don't need to be there.