Ok I know what you're thinking. Foofles, your game is awesome but the world is completely devoid of anything.
So I created a 3D map format and the export plugin for it for 3DS Max. And with my ultimate skill, I make a very basic map and show collision with the fireballs (and consequently anything else) with the map. It boils down to a sphere vs triangle check, but the faces and parts of the map are seperated into a tree (not really an octree... but something like it) to quickly discard things. It's not perfect yet but so far so good.
The bullet holes right now are a quad that gets placed where the collision took place, and sort of orients itself in the direction of the surface. Again, not perfect. Working on a better decal system.