A word of warning: As much as overclocking is supported, it is not officially supported. This means that should you damage your card and send it in for a replacement, and the support tech staff is able to determine that you overclocked your card, then your warranty is void. OCing through software makes it very difficult for them to determine this, whereas a flashed BIOS makes it extremely easy.
Ok, so, regardless of why you want to overclock, let's get into the basics of it.
Core means the actual GPU core, where most of the magic happens. The speed, in mhz, that your GPU core is running at is going to greatly affect the texture and pixel fillrate performance, which is a huge bottleneck in gaming.
|I'm not paid for complete testing ;)|
I ran some tests on FurMark, at 1920x1080, showing how both core speed and memory speed can affect performance. As you can see in the graph, the framerate went up as the memory speed increased, with barely any change from core speed. It is important to note that this particular test was run using 4xMSAA. I also did some tests at 0xMSAA in FurMark, and it showed that only core speed had an improvement. Basically I just want to point out that for optimal overclocking, you need to focus on both memory and core speeds.
Voltage is what gives you stability. As the core speed increases, it may need an increase in voltage to remain stable, much like in CPU overclocking. Most higher end GPUs allow voltage modification, however, not all GPUs allow this. In some cases you might be able to also modify the VRAM voltage, but this is fairly uncommon. If you can, then in much the same way as core voltage, it can help you achieve higher clocks.
VRM is the Voltage Regulator Module. This is what adjusts your input voltage to that going into your GPU. These are very prone to over heating when overclocking, especially if you increase your voltage.
How To Overclock
Before you start, make sure that you do NOT have "apply overclock at startup" selected!
The basics of GPU overclocking are very similar to CPU overclocking. Personally, I recommend starting with Memory clock because it gives nice gains and often isn't affected by voltage - since usually you can only change core voltage. To overclock the memory, I like to start with 50mhz jumps, with 5 minutes of FurMark testing at 4xMSAA at full screen - also, let the cards stay heated up a fair bit rather than giving them a long cool down time between tests. You may want to set up your tests using the benchmark feature. Continue this until you get either a lock up, crash, artifacts, or the screen goes blank. Doing a hard reset on your PC isn't fun, but it shouldn't wreck anything. I usually find that ctrl+alt+del still works and you can reset from there.
|FurMark stress test|
Moving on to GPU core overclocking, the process is much the same. You can still start with 50 mhz jumps, but with the testing you should run at 0xMSAA in FurMark. Also try to keep temperatures high throughout the testing phase.
The most common issue of an unstable core is artifacting. This is when pixels essentially get stuck, and you'll notice multiple squares on screen that don't sync up with the image. At this point you will want to stop the test and will also have to reboot the PC. You have a choice here, either to reduce the clock or else increase the voltage. For 24/7 overclocks, I highly recommend reducing the clock and finding the highest stable clock on stock voltage. However, if you want to increase the voltage, I suggest doing it in 0.05V increments. Basically just increase the voltage and retest at the same clocks, and repeat until stable. It is also vitally important to watch all your temperatures, because as I mentioned before, it's possible that you crash because of heat and not instability.
There are, as always, a few other things to consider. FurMark isn't perfect in determining if your GPU is stable. I've passed fairly long sessions in FurMark, as well as a few games, before finally having an issue in one specific game. It didn't take long to realize that my GPU OC was to blame, and dropping my memory clock by only 5mhz made it nice and stable from there on out. So, actually playing games will be your final stress test.
If you run crossfire or SLI, there might be issues with instability. Generally it should be ok, but I've read about issues with crossfire that has to do with the switch between low power and high power, and only occurs when a custom voltage is set. It seems the way around this is to disable ULPS, which you can google for yourself.
Another area of problem is that sometimes after an overclock is applied, the graphics cards won't automatically change to low power state at idle, or else the idle state clocks are higher than they used to be. This is something that is at the mercy of people making the overclock programs, and the only way around this (and other) issues is to manually reflash your GPU BIOS. This is not a recommended practice, but I will just put here that if you have a very stable overclock, it is a handy thing to do because your GPU will essentially think that the overclock settings are it's default settings, so it behaves like a totally stock card. There are tools for changing and flashing the BIOS, and for more info check out techpowerup.com.
If you have concerns about your particular brand or model of graphics card, just google your card and key terms like "overclocking" "highest clocks" "stability" to find a variety of sources. I always advise learning as much as you can before doing something you don't quite understand, and causing damage.