A capacitor (often appreviated "cap", just as a potentiometer is often just a "pot") can be likened to a tiny, rechargeable battery which holds a very small amount of charge and can be charged and discharged much more quickly than any battery.
Like a battery, a capacitor has two outside connections. Unlike a battery, however, the polarity of these connections is interchangeable. A battery always has one side positive and the other side negative, and these symbols are permanently labeled on the outside of the battery. A capacitor, however, can have either side be positive or negative; The polarity is determined by how you charge the capacitor. If you hook up the capacitor to a power source, the side connected to the positive side of the source will become positively charged, and the side connected to the negative side of the source will become negatively charged. The two leads from the capacitor are essentially symmetrical when the capacitor comes from the factory, and it can be charged whichever way you like.
A capacitor's capacity is measured in capacitance. (What a sentence, eh?) Capacitance is measured in farads (abbreviated F), named after Michael Faraday, the great chemist who invented the capacitor in the 1800s. One farad (1F) is an uncommonly large amount of electricity for electronics work, so in electronics you'll mostly see capacitance expressed in microfarads (µF), nanofarads (nF), or picofarads (pF).
An interesting property of capacitors is that they will block DC after they become charged, but will (for the most part) allow AC to flow through. When a capacitor is fully discharged, DC can flow through it freely, but as it is doing so, the cap is gradually becoming charged. Finally, when it has reached its storage limit, the cap will not allow any more electricity to flow through it, and will act as a blocker on the circuit. This can be observed if you simply wire a capacitor in series with a simple circuit connecting a battery to a light. When the circuit first comes on, the light will turn on, but after some time (when the cap becomes fully charged) it will turn off. Exactly how long it takes this to happen depends on the capacitance of the cap; With very low-capacitance caps, it will probably happen faster than your eye can perceive, but it still will happen. If you remove the battery and simply leave the cap connected to the light, the light will turn on again as the cap flows into itself, acting like a battery, and the light will stay on until the cap fully discharges.
So a cap blocks DC; If you think about it for a moment, you should be able to understand why they allow AC to pass through: AC keeps reversing its polarity. As long as the AC switches fast enough to prevent the cap from becoming fully charged in any direction, then the cap will partially charge in one direction, and as the AC polarity reverses, the cap will start to discharge, then charge in the opposite direction. Very low-capacitance caps may partially block the AC because they become fully loaded before the AC cycle is complete, however.
Unlike a battery, which has basically only one use (to supply electric power), a capacitor is a surprisingly versatile device which has several applications in common electrical and electronic devices. Just a few of the capacitor's uses are:
To smooth out disruptions in a circuit. This is especially common in power-supply circuits where small power disruptions can instantly crash sensitive electronic equipment like computers; Maintaining a very constant, steady stream of electricity is crucial. For this reason, capacitors are often wired in series with a power supply circuit so that if the power turns off for just a moment, the device will still receive power from the capacitor. Many blackouts or brownouts last only a fraction of a second, and a capacitor can sometimes correct them in the power stream. This use of capacitors is very common in all types of everyday electronics; This is the reason why many radios will keep playing for a few seconds even after you unplug them.
For timing. The amount of time a capacitor can hold its charge is surprisingly precise; So precise, in fact, that a capacitor makes a fairly decent timing device, because at a set rate of discharge and recharge, the time it takes a cap to fully charge up, or fully discharge, is quite predictable, to within a small fraction of a second, and so caps can be used for timing circuits where millisecond-level timing is not critical. For example, a capacitor is used with the 555 timer chip when using the 555 in astable mode; The timing of the 555 basically derives from how quickly that capacitor discharges, and the resultant oscillating circuit is consistent enough in its rate for many applications.
A quick jolt of electricity. As has been mentioned, a capacitor can discharge all of its charge almost instantly, certainly much faster than a typical battery could. This makes a cap especially useful with things like a camera's flash bulb. If you tried to light the flash from the battery, it would be rather dim; You couldn't make an appropriately bright flash with the limited current available from the typical household batteries used in the average camera. So instead, a capacitor is charged from the battery, and when the picture is taken, the capacitor's charge floods into the flashbulb, creating a flash that illuminates the subjects for the photograph. (This is why most cameras have a delay between photos where the flash must be charged; What's actually being charged is a big capacitor inside the camera.)
To block DC. As mentioned previously, caps block DC electricity. If you do not wish for DC to exist at a certain point in a circuit, a cap works great for blocking it.
There are several different types of capacitors as well, but the main types are electrolytic and ceramic. Electrolytic capacitors look like metal barrel shapes; They use a chemical electrolyte inside which makes them relatively unreliable and prone to failure over a period of several years. Ceramic capacitors look like small flat discs, usually brown. They are extremely reliable, solid-state devices that tend to last basically forever, but their downside is that they usually can have only a very small capacitance, and so they cannot be used for every possible application.