In my career, I've been fortunate to spend a great deal of time involved in system design. Much of that time has been spent implementing or modifying established systems (e.g. Dungeons & Dragons and Fallout's SPECIAL system). Before I was employed in the industry, I spent a lot of time developing my own tabletop systems and modifying the systems of others, so this has always been something I've enjoyed doing.
There are many pitfalls to system design and I believe most designers trip those pitfalls by moving into implementation details too quickly. I believe some keys to success in system design (and for design in general) are to establish clear goals, to frame what those goals will accomplish in terms of player experience, and to continually return to those goals and player experiences to ensure that nothing was lost in the details of implementation.
I believe the most well-executed systems are ones where thoughtful players can accurately discern the designers' goals simply by scrutinizing the systems in action. Though not all players need to be able to do this, the ones who care to do so should be able to. Designers who succeed in creating systems that can be "reverse-engineered" in such a way have captured the soul of elegance in design.
I sometimes look to traditional games for mechanical inspiration. One of the ones I think of most often is chess. Clocking in at over 1,000 years of play around the world, chess has had a lot of iteration time. I'm not an expert on chess strategy and I'm not a particularly good player, but I know chess well enough to take some simple lessons away from it. Two that I often rely on are lessons of obvious value and orthogonally equivalent value. These two lessons can be summarized by examing three chess pieces: the queen, the knight, and the bishop.
When I look at any system, I examine both the system's design as well as the content that uses the system. I believe this is something that system designers should always do. A system is only as good as the content that makes use of it; content that fails to make use of a system (or vice versa) will always create a disappointing experience.
The queen is typically the most powerful piece in chess (though not the most valuable; that role is reserved for the king). The queen's movement capabilities combine the lateral movement of the rook with the diagonal movement of the bishop. Even if you are learning chess for the first time, the fact that the queen combines the movement of two other pieces makes her relative power clear. A rook's ability to perform a castle, the knight's excellence at creating forks, and a the pawn's ability to capture an enemy pawn en passant are all capabilities that take a while for players to appreciate, but not the queen's movement. The queen's value is obvious.
Gameplay consists of players making (more-or-less) informed decisions about what they need to do to overcome an obstacle. It is not enough for the obstacle to be clearly defined and communicated to players. They also need to have a clear understanding of what tools are at their disposal to solve the problem. In chess, the player's primary tools are his or her pieces. Though circumstances determine the value of pieces on any given move, no one needs to advocate the fundamental value of the queen in chess.
As an extreme analogue in video games, it's unlikely that many players need to be told what the value of the HECU RPG is the first time they find one in Half-Life. After being pursued by a relentless Apache helicopter over numerous maps, the player winds up in a cave with the RPG on the ground and the Apache hovering outside. Players typically snatch up the RPG and blast the Apache in moments. Though the HECU is not the "queen" of Half-Life's weapons, it has obvious applicability in the circumstance where it appears.
When designers develop tools, we should strive for clarity of primary purpose in a player's tools. The more obvious we make the value of the tools at a player's disposal, the more quickly the player will spend time fully engaged with the obstacles at hand instead of trying to figure out what they aren't "getting".
Chess has various informal ranking systems for the relative value of pieces. The rankings are not used for scoring, but they are used to give players a rough idea of the strategic (not tactical) value of those pieces. In the most commonly used system, pawns have a value of 1, rooks have a value of 5, and queens have a value of 9. Knights and bishops are both rated at 3. Bishops move diagonally, always staying on their starting color, and knights are the "funny moving" pieces of chess, hopping two squares horizontally or vertically and one square vertically or horizontally, passing over other pieces along the way. Though their tactical applications in any given circumstance are completely dissimilar, the common ranking systems give them equal (or close to equal) strategic value in chess.
Whether chess' numerous contributors intended for them to be equal in value by design or players collectively determined they were equal in value, today's players generally regard them as being so in spite of their radical differences. I.e., players treat them as having orthogonally equivalent value. Knights and bishops are considered equivalent in an orthogonal sense because their mechanics and applications do not overlap but they commonly create the same amount of benefit for players. Though bishops can move infinitely along their color, potentially from corner to corner, they lack the knight's ability to move over pieces.
Dungeons & Dragons commonly presents choices in such a fashion. The most obvious examples are spells, which are grouped by level. In most editions of A/D&D, haste and fireball are 3rd level wizard/magic-user/sorcerer spells. Though the tactical relevance and application of these spells varies wildly, the games' designers established them as being equal.
When we design tools for the player to use -- abilities, gear, options, upgrades -- options with ostensibly orthogonally equivalent value create interesting choices for the player. They also lend themselves to increased clarify of purpose. The more tools overlap in function, the less obvious it is to players why a given tool exists. The less tools overlap in function, the more those tools seem suited to a specific circumstance.
While these are high-level design concepts, creating choices with obvious, easily differentiated values can make the low-level details much easier to execute and build upon. When a player is presented with strategic or tactical choices, he or she is always fundamentally asking the question, "Why do I want to make this choice instead of any of the others?" As designers, we want to communicate the answers to their questions as elegantly as possible. Ideally, the design of the player's tools and the game's content should be self-advocating, allowing players to reverse-engineer our intent and their range of choices without a word of explanation.