Since the PHP devs doesn't see it fit to include an option to uninstall (at least not in v5.3.6) in their source distribution I thought I'd share a quick tip with you.
In order to uninstall what we installed with "make install" we first need to find out what was installed.
"make -n install" will dump the install commands used during installation, at least as long as you haven't rebuilt the config or other wise changed the install files since installation.
You will have to extract out the resulting files on your own since the output of "make -n install" includes commands to create directories.
It should result in a list similar to this one:
/usr/local/bin/php-cgi /usr/local/bin/php /usr/local/bin/phpize /usr/local/bin/phar /usr/local/bin/phar.phar /usr/local/bin/php-config /usr/local/man/man1/php.1 /usr/local/man/man1/phpize.1 /usr/local/man/man1/php-config.1 /usr/local/lib/php/ /usr/local/include/php/
Just remove those files and directories and you are done.