One reason women sometimes appear to have lower sex drives than men--or at least less interest in casual sex: fear. Oh, not that we're scampering around terrified of men, but for both cultural and real-world reasons, a woman agreeing to casual sex does have to consider some creepy factors. You're going to be alone with this person with your clothes off, and likely you'll even fall asleep in their presence. If they turn out to be a bad person, there's a lot of things they could do to ruin your day.
Of course men could say the same thing, but this is where the cultural part comes in. A man sexually desiring a woman often has overtones of threat in our culture. From street harassment to horror films to PUAs, women learn that someone desiring you doesn't mean they're going to be nice to you. Whereas female desire for men is almost universally seen as flattering. You can even see this in extreme cases--read the comments on any news story about a woman sexually abusing a man or boy, and a good portion of them will be calling the victim a lucky bastard.
As a result, women tend to go on the defensive. It's not our sex drive that's weaker but our trust. We learn to reject men, not because we don't want them, but because we aren't sure that their advances are the friendly kind. Even when I'm horny as hell, I can't turn off the creep-dar, and when a guy is physically attractive to me but sets off the creep-dar in any way, I can't sleep with him. I know this is unfair to guys who aren't creeps, but don't blame women--blame creepy men. They're the ones poisoning the lady-well.
When I decide whether to go home with a new guy, we're both thinking about whether we want to fuck each other, but only I'm thinking about whether or not he'll stop when I say "no."