Subvarieties of the Grassmannian $G(1,N)$ with small secant variety