I ran across an interesting article at Fiction Groupie some time ago about writing male point of view. It provided a checklist of some things that men do and think about...fully admitting that many of these things were stereotypes, but pointing out that the list does have some basis in fact (most stereotypes do, on some level). My first reaction on reading it was that I felt it really didn't apply to most of the male points of view that I write. Was it just that I was avoiding stereotypes? Was it - horrors - that my male characters weren't male enough?
Fortunately, I have male readers who have assured me my male characters are working - but it got me thinking about how I write male points of view. I do this quite a lot, in fact - two of my three published stories have male protagonists, and my novel in progress, For Love, For Power, has three points of view, all of whom are male, for structural reasons.
First I think it's important to think about stereotypical characteristics from the point of view of core vs. peripheral characteristics rather than stereotypes. Core characteristics are those that tend to be possessed by most men we know. Peripheral characteristics are those that can be considered male, but are typically possessed by smaller subgroups of men. One of the things that will cause you to fall into a stereotype is if you give too many of your male characters too many of these characteristics all at once. To go with Roni Loren's list, if they're all action oriented, impatient, visually oriented guys who like to be in charge, project confidence but repress their emotions, say what they mean in order to solve all problems, converse only to exchange information and think about sex all the time... you have a problem. On the other hand, these are all really valuable trends in male behavior in our society that are useful to consider when designing male characters (especially for category romance, which has its own idiosyncratic demands!).
One thing I'd encourage you to remember is that a lot of the characteristics that we consider typically male are based in our society's cultural values - which means that if you're working outside our society and its rules (as I am most of the time) the characteristics of male characters are going to be heavily influenced by the differences in the society around them. Dress varies widely (think Japan versus US men, for example). So does the expression of emotions (think European or Slavic men vs. Englishmen for an alternate example of expressive style). When you're designing your world and the society that operates within it, make sure to think through some of these core gender-role variables and figure out what your society values.
So for the sake of making this more concrete, I'm going to give some examples from my own male characters. I'd say that typically each one has one or two defining characteristics that are "male," but they vary widely on a lot of the other variables.
The most current-society-normative of my characters are the humans from my Allied Systems stories. The young man David Linden doesn't have women to interact with, so sex isn't on his mind at all. He's primarily defined by his need to prove himself to his father as a worthy scientist - which can be done for either gender, but won't seem out of place for a male character. The main character of my story in progress, The Liars, is Adrian Preston. He's married and spends a lot of time thinking about, and negotiating with his wife, but the story doesn't allow a lot of extra time to explore the intimate side of their relationship. He's a man who lives for his work as a linguist and loves it so much that his idea of having fun is working on language.
The idea of the importance of work is one that I didn't see in Roni Loren's piece, but one that I think is common to a great many men. When designing a society you should definitely consider identifying what activities are considered worth dedicating one's life to (work), and which are considered legitimate outlets for emotion and conversation (sports, for example). Even Rulii, my wolflike alien, is very much centered on how his work as Councilor will allow him to achieve his life's goal, which he thinks of in terms of "landing the quarry of my life's hunt."
A more nuanced example from my stories is the character of Imbati Xinta. He lives for his work to the point of fanaticism, and he certainly represses his emotions, but not for the reasons that men in our society would do so. Because he works as manservant to the Eminence of Varin, his job is to stand by and remember everything he hears, and to reveal nothing through his face or movements that would jeopardize his master's secrets. He is a trained bodyguard and martial artist, but in appearance is quite effeminate, and emotionally he is very vulnerable. There are a couple of things going on with this, one of which is that I've known any number of men who go about covering up significant emotional vulnerabilities - and the other of which is that Xinta is expected to repress his own ethics and human feeling, and to be entirely "selfless," since that is considered the ideal state for a member of the servant caste. Xinta self-represses to such an extent that he's not able to connect with anyone emotionally beyond normal politeness, and sex is the last thing on his mind. Which is to say I suppose that I'm using the work focus tendency and the emotional repression tendency to negate the tendency to think about sex in his case. As to his appearance, I'm having him look the way he does - paying close attention to his looks, dressing in bright colors, wearing jewelry, etc. - in part to please the man he works for, and in part to echo that real-world tendency for a "civilized" man to take on more elaborate habits that might be laughed off as effeminate by a member of the lower classes.
I suppose you could say that close observation of the people around you can only go so far, because that will only allow you to see the parameters being used by the people around you. I have found my anthropological studies extremely valuable, because they've given me an eye for paying attention to and interpreting the possible variables behind different styles of social interaction. Particularly if you're worldbuilding, you should try to see foreign movies or read books about people in other times from the point of view of looking at societal models of gendered and romantic behavior (Emma, for example, can be quite an eye-opener for someone used to the permissive ways of modern romance).
When you're writing a male character, you won't want him to be without any male characteristics (those recognizable to the readers). That can be considered a given. But you don't have to cling just to the stereotypes you know. If you cultivate a sense within your world and your reader of what gendered behavior is like, then you can have your male character follow that trend and see it as masculine. Furthermore, female characters can possess Earthly "male" characteristics and still be considered feminine depending on the views of the society you're working in. The most important thing, I think, is to make sure that you've thought through why your character behaves the way he does, why you think he's masculine, and precisely how and why he deviates from the stereotypes that everyone will be looking for, yet fearing to find.
It's something to think about.