Characterizing varieties of colength ≤4