What is wrong with writing passwords down?

It is probably better to create unique, complex passwords for each website, write them down on a piece of paper and keep the piece of paper in your wallet, than it is either to use simple passwords which you can remember in your head, or to have a single complex password which you use for every website.

But what if your wallet is lost or stolen, or you leave the piece of paper lying on your desk and someone makes a note of your password while you are fetching a coffee?