The folders I renamed/removed were:
C:\Users\username\AppData\Local\Microsoft\Internet Explorer
C:\Users\username\AppData\Roaming\Microsoft\Internet Explorer
The only thing it was affected that I had to replace manually was my desktop background. I'm not quite sure how that's linked to Internet Explorer but it appears to be because when I rebooted after doing this my background was just plain black.