Yeah, benchmarks of a 580 on a dualcore, quadcore, and an OC'd high end Quadcore for BF3 when it was released showed nearly no change to FPS. Literally 1-2 fps between them (and not necessarily the faster was the better).
I'd guess you've got plenty of grunt. In fact it's really only old games with 1 thread which won't perform as well, if they were cpu limited. Modern games with more than 2 threads will run fine.