I haven't had that experience. However, rather than drinking your sleeping potion on your island....head to the mainland first then go to sleep, that way you can start your search off on land and maybe you'll find the wispy cotton easier. The spirit world can be a frustrating place. :)