You have two different attributes with two different sets of values.
Sizes - S, M, L
Size - XS, S, M, L, XL
If you just want to offer S M and L, just pick the one labeled 'sizes' instead of 'size.'
Really you could still use the one with all of the size options, because you don't have to offer an item for every size you have as an available variation.
When your looking at the list of available variations under an attribute, just click the little minus sign beside that variation to delete it from that attributes list.