To me it seems it's a matter of definition - a matter of how we define "sex" and what we mean by "no sex" or by "celibacy".

A quick summary of how I see it:

Sexual energy is the key to higher realms of consciousness, which is why we shouldn't waste it. Semen is a concentrate of life energy and it takes months to fully reproduce. Celibacy is about avoiding ejaculation in order to maintain the energy and transform it by moving it up the chakras (and thus increasing its frequency). Without celibacy there's a drain of energy that the body tries to compensate for (it needs to consume more food, it needs to sleep more and it takes more time to process it all). Having that constant drain of energy is not a problem, it simply slows progress. However, by cultivating sexual energy (either by sex or without) and transforming it properly, progress is sped up tremendously.