As an atheist and someone who does not believe in reincarnation I'm strictly in the "No" camp. For karma to work there would need to be an external agent, power, god etc. to punish / reward as necessary and I do not believe such a power exists. Saying that the "universe or cosmos" sorts these things out just doesn't wash with me. IMO it is wishful thinking for people who see a random, unfair world. I remember reading that most Nazis involved in the atrocities in the concentration camps were never brought to justice and presumably died in their sleep in their homes in South America or wherever they fled to. When people do see what they believe to be karma taking place it is merely coincidence, nothing more. As an earlier poster commented 'Karma, scharma'
Careful ! I suspect you entertain a typically Western misconception about karma, here.
Karma is a polysemic sanskrit word which can be translated as "act", "work", or "consequence" depending on the context. As such, it doesn't necessarily bear any notion of divinely imposed morality. Karma is simply the way "actions work out", in other words, how they bear fruit, in certain conditions.
It's akin to the laws of physics. Karma is simply the Indian a way to explain that you need causes and conditions to get consequences. When the causes and condition are met, their "natural" result happens, without any need for a demiurge to intervene and "judge".
All this "superior power" thing you describe comes not from Indian (Hindu, Buddhist, or Jain) culture. It's an application of the Judeo-Christian (or even Egyptian) belief in a weighing of souls. But this is not "karma". This is only an Eastern concept over which we super-impose our own native mythology. Karma is essentially amoral.
This being said, it is true that, even in the East, the concept of karma is often over-simplified to meet moral agendas. Some people speak of "bad karma" or "good karma", and of doing good actions to store "good karma".
In fact, karma is neither good nor bad. What is good or bad is the consequence of an action, from the point of view of the agent, but only when this action has borne fruit.
For instance, when someone says "binge drinking brings bad karma", what he really says is that drinking might feel good until you get hit by a car or until you get a liver cancer. Then, retrospectively it suddenly appears that too much alcohol was not such a good idea ! "Bad karma". "Murdering people" also becomes "bad karma" when you get to the unpleasant consequences of this behaviour. But as you point out, theses consequences might never happen, if the conditions are not met. A murderer who never crosses the way of a police officer may never pay the price of his harmful actions in his lifetime.
As some Buddhists say : You may possess a seed (cause). If you never plant it into the ground (condition 1) and water it (condition 2), it will never grow into a plant [that metaphor comes from the
Salistamba Sutra].
So it's not : "I did this, I'll get that". It's "under certain conditions, if I do this, I'll get that". I cannot control the conditions, but I can control the cause, since I am its agent. So, in order to avoid "consequence X", I'll take care to refrain myself from committing "cause X". That's where morals cross the path of "karma". It's not a matter of godly legislation. It's more like a kind of "heuristics of fear" (but there I am the one who sticks a Western concept on an Asian word ...).