Keep It Simple, Stupid, or the way I like to phrase it, make the simple things simple and the complex things possible. This is a very common design mistake. Users ask for tons of features and developers work hard at delivering those features, but often times at the expense of a simple to use application.
Of course by simple, I mean "everything should be made as simple as possible, but no simpler" (quote attributed to Albert Einstein). If the interface doesn't make the most common way of performing a task as simple as possible, the interface has failed the user.
An example for good use of this principle is the Remote Desktop connection interface. When you start Remote Desktop, you see a UI with the absolute bare minimum interface for connecting to another computer (figure 1), a single textbox for entering the name of the computer you want to connect to and some buttons to perform what should be easy to understand commands (at least, easy to understand if you know what the application is supposed to do in the first place).
Of course there are a lot of options that you might want to change when connecting to another computer, if you are an advanced user with a special need you can select the Options >> button. This button opens another window (figure 2) that allows you to change the advanced settings of Remote Desktop, you can even save the settings so that you don't need to go through the options dialog again.
The term for this type of interface is progressive disclosure which simply means removing less frequently used interface elements from the primary screen while still providing a means of accessing those elements. This can be in the form of a secondary screen, such as in Remote Desktop, collapsing regions, or providing an Advanced tab within a tabbed interface (this can be seen in the options dialog in the Remote Desktop screenshot, figure 2).
If you want to see how not to design a UI, take a look at Bulk Rename Utility (figure 3). It seems to provide every possible option on the main form, regardless of how often it is used. Bulk Rename Utility is a useful tool intended for computer savvy people who can probably overlook a bad UI if it provides the features they need, but you would never want to put a UI like this in front of a technophobe.
So how do you know what the simplest interface is for the user? Asking the users is certainly a good start, though not entirely reliable (the mind has a tendency to assign equal importance to exceptional cases and common ones). Other methods include watching the user actually performing the task and logging feature usage and analyzing the results. Feature logging will probably result in the most comprehensive and accurate view of the current system, but it requires you to have a system in place already and for your users to be willing to share the information that is being logged.
By keeping the interface as simple as possible, the user is able to accomplish the task quicker with higher accuracy, less training, and fewer support calls. By providing a simple interface with the ability to perform more complex tasks you have taken nothing away from the user except a barrier to using the application.