This topic came to me thanks to K. Richardson, whom I encountered on the forum at the Critters critique website. Many of you probably know it, but if you don't, and you're seeking critique for works of science fiction, fantasy, or horror, it's a great place to start. The question K. had was how to get her aliens to swear without using swear words from our own world, yet not have the substituted words sound silly.
I would start by asking, "what is swearing for?" Here are some of its purposes in our world:
1. to express emotion in moments of extreme stress or pain
2. to express dissatisfaction (a less extreme version of #1, I think)
3. to give evidence that one is telling the truth ("I swear by the sword of my father, Domingo Montoya")
4. to get attention (over a wide range - everything from making a word stand out to emotionally injuring someone)
5. to express alignment with particular social groups
Here's an example of how a difference in degree on #4 can cause trouble cross-culturally. I come from a cultural group in which the form of swearing I'd call "ugly words" is used to get attention and injure people emotionally. My husband comes from one where the same words are used to draw attention and even give a certain spice of fun to what is said. So when we first met, and before I figured out the difference, I'd be regularly appalled at what was coming out of the mouth of this otherwise entirely nice (and very interesting) guy.
If you're creating an alien society, you can always go with a model that has your aliens using swearing for similar purposes - but of course, the parameter is available to be played with, so there's no need to hold back!
Next, I'd ask about content. Different cultures use different kinds of swearing content, but these can include religious references, scatological or sexual references, ridicule of others by comparison with animals, etc. Some swear words (like the old fashioned "zounds!") are derived from religious sources but have been euphemized (in this case from the expression, "God's wounds!") to avoid blasphemy.
So back to the aliens. Religion-based swearing is going to depend on what their religion is, whether it has the good vs. evil dichotomy, and how their cosmology works. From the point of view of the hypothetical Gegogian, what would be a fate worse than death? And under what circumstances might someone wish it on someone else? If you're looking for ugly words, then what do these people consider ugly? Are sexual body parts considered taboo in the society, and would they be used for expletive purposes? If someone wants to swear truthfulness, what object or concept do they value so highly that they would not want to sully it by lying under oath?
If you're going to pepper your dialogue with swear words or phrases the way we do in English, it's good to keep them short. But if you want to expand the cultural role of swearing and turn it into a lengthy trash-talking contest, then by all means elaborate them. Personally I would hope such a contest would be relevant to the main conflict of a story, though, and not just there for pyrotechnics!
Tonight I don't have a lot of examples on hand from established fiction - I welcome comments, so tell me if you thought swearing was handled well, or in a funny way, or in a less than optimal way, in any of the books you've read. In my own writing I have one culture where they use wordless sounds for extreme emotion, and insult people by talking about their undesirable physical behaviors. I have another where people are expected to swear by one of a family of deities, and which one to use depends on how the situation aligns with the personality and job of the deity in question.
Exploring swearing and its cultural underpinnings is a great way to give spice and dimension to a world.