I'm not Jewish but my understanding is that the leaven in bread represents sin, so avoiding leavened bread during Passover shows the avoidance of sin. The exact story comes in the first 20 chapters of Bible book Exodus. Atheist or not, read it, the story is pretty cool. In fact, I think I'll reread it myself, I've forgotten some of the details.