Here is my workflow and tools I use when working on a WordPress site locally.
For WordPress development, you'll need Apache, MySQL and of course PHP.
If you're on a Mac, you can get XAMPP or MAMP, which will let you install these software without much knowledge of configurations and compiling. I tend to use XAMPP because I can use it on port 80 instead of 8888. Use whichever you like the most.
On Windows, you can use WAMP.
Source Control Management
I realize that the use of SCM is quite rare among WordPress users. But with my programming background, it's really hard to not use a SCM. Where I can choose, I always use Git to version my code.
You can either version your wp-content folder, your theme folder or your complete WP install folder. I like to put the whole wordpress install under version control. That way, I'm always sure that I have the WP version I need for my site to work properly.
If you version your whole WP install, you should at least add 'wp-config.php' (without the quotes) to your .gitignore file (if using Git).
Depending on who will add the content to your site, you might want to make Git ignore the 'wp-content/uploads/' folder as well. Just add a .gitkeep inside it so Git will not remove it from your tracked files.
Another obvious thing: in your local wp-config.php, set WP_DEBUG to true. That way you'll better understand why some plugin make your site crash.
The Debug Bar plugin
Install the "Debug Bar" plugin: http://wordpress.org/extend/plugins/debug-bar/
Environments (Development, Staging, Production)
Since you develop locally, it means you have a "Development" environment for working on your site and a "Production" environment where your site is hosted. But if it's a big website and you don't want to mess things up, I strongly advise having also a "Staging" environment on the same server than "Production", but on a different subdomain. This makes it easy for you to see if you have any bug related to the hosting environment (PHP version or whatever), and it makes it easy to show your changes to your client before push them live in "Production".
A no brainer for some people, but sadly I still see a lot of WordPress devs not having this.
Use an automated way to deploy to "Production" and "Staging". Usually, it's something you might call with the command line. Like a Capistrano task. Or you can make your own Shell script. These tools (capistrano or shell scripts) will usually involve deploying via Subversion or Git. But you can use rsync as well.
Your script can be as simple as push your local changes, pulling them in your Staging/Production environment, then changing your wp-config.php file. Or it can be more elaborate, like exporting your local MySQL database and pushing it, then importing it to the server, replacing every occurrence of "http://localhost/mywpdevsite" for "http://staging.mysite.com" in it. Whatever automated tool you'll end up using will beat the heck out of FTP'ing your files manually on your server. FTP is slow, insecure and too manual for my taste.
If Capistrano seems too much for you, but scripting in Makefiles is not your thing, you can always try Fabric or William, which are both DSL's for interacting with your staging/production server via SSH. (Note that William is my pet project and is nowhere near Fabric in terms of completeness or stability. But it's in Ruby!)
Dealing with URLs in your MySQL dumps
Like I mentioned on the previous point, you'll need to modify your exported MySQL dump because WordPress and some of its plugins will store your domain in the database. Personally, I tend to use the "WP Migrate DB" plugin to change every occurrence of the dev URLs to the staging/production URLs. It's done via the admin so it's not automated, but it's still a time saver. Some scripts exist too, like this one, but I didn't try it yet.
Although working locally might seem like an overhead, it doesn't have to be: Automate your deployment and version your code. You'll feel much safer knowing that you can always revert to a previous version of you site if something goes wrong. And it's stress-free development, which means you can go wild with your changes without fear of breaking your site.