We had the same issue while trying to launch Selenium tests from Jenkins. I had selected the 'Start Xvfb before the build, and shut it down after' box and passed in the necessary screen options, but I was still getting this error.
It finally worked when we passed in the following commands in the Execute Shell box.
Xvfb :99 -ac -screen 0 1280x1024x24 &nice -n 10 x11vnc 2>&1 & ...killall Xvfb