I don't know about the swords bugs, but designing it so that it ignores speed set is unfair simply because players are paying for that extra speed with reduced stats.

If you want to use to counter speed set, adding some sort of slow would be more fair, that way it does somewhat neutralize speed set, but only temporarily, and it can be countered by stun immunity skills.