A、x3-x2-x+1=x2(x-1)-(x-1)=(x-1)(x2-1)=(x-1)2(x+1),故此选项不合题意;
B、x2+y-xy-x=(x2-x)-(xy-y)=x(x-1)-y(x-1)=(x-y)(x-1),故此选项不合题意;
C、x2-2x-y2+1=(x-1)2-y2,故此选项符合题意;
D、(x2+3x)2-(2x+2)2=(x2+3x-2x-2)(x2+3x+2x+2)=(x2+x-2)(x2+5x+2)=(x+2)(x-1)(x2+5x+2),故此选项不合题意;
故选:C.