I've been fighting with this issue for a long time, and just y'day I figure out how to make it gone and today I can run a 50 threads process calling selenium without seen this issue anymore and also stop crashing my machine with outofmemory issue with too many open chromedriver processes.
- I am using selenium 3.7.1, chromedrive 2.33, java.version: '1.8.0', redhat ver '3.10.0-693.5.2.el7.x86_64', chrome browser version: 60.0.3112.90;
- running an open session with screen, to be sure my session never dies,
- running Xvfb : nohup Xvfb -ac :15 -screen 0 1280x1024x16 &
- export DISPLAY:15 from .bashsh/.profile
these 4 items are the basic setting everyone would already know, now comes the code, where all made a lot of difference to achieve the success:
public class HttpWebClient { public static ChromeDriverService service; public ThreadLocal<WebDriver> threadWebDriver = new ThreadLocal<WebDriver>(){ @Override protected WebDriver initialValue() { FirefoxProfile profile = new FirefoxProfile(); profile.setPreference("permissions.default.stylesheet", 2); profile.setPreference("permissions.default.image", 2); profile.setPreference("dom.ipc.plugins.enabled.libflashplayer.so", "false"); profile.setPreference(FirefoxProfile.ALLOWED_HOSTS_PREFERENCE, "localhost"); WebDriver driver = new FirefoxDriver(profile); return driver; };};public HttpWebClient(){ // fix for headless systems: // start service first, this will create an instance at system and every time you call the // browser will be used // be sure you start the service only if there are no alive instances, that will prevent you to have // multiples chromedrive instances causing it to crash try{ if (service==null){ service = new ChromeDriverService.Builder() .usingDriverExecutable(new File(conf.get("webdriver.chrome.driver"))) // set the chromedriver path at your system .usingAnyFreePort() .withEnvironment(ImmutableMap.of("DISPLAY", ":15")) .withSilent(true) .build(); service.start(); } } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); }}// my Configuration class is for good and easy setting, you can replace it by using values instead.public WebDriver getDriverForPage(String url, Configuration conf) { WebDriver driver = null; DesiredCapabilities capabilities = null; long pageLoadWait = conf.getLong("page.load.delay", 60); try { System.setProperty("webdriver.chrome.driver", conf.get("webdriver.chrome.driver")); String driverType = conf.get("selenium.driver", "chrome"); capabilities = DesiredCapabilities.chrome(); String[] options = new String[] { "--start-maximized", "--headless" }; capabilities.setCapability("chrome.switches", options); // here is where your chromedriver will call the browser // I used to call the class ChromeDriver directly, which was causing too much problems // when you have multiple calls driver = new RemoteWebDriver(service.getUrl(), capabilities); driver.manage().timeouts().pageLoadTimeout(pageLoadWait, TimeUnit.SECONDS); driver.get(url); // never look back } catch (Exception e) { if (e instanceof TimeoutException) { LOG.debug("Crawling URL : "+url); LOG.debug("Selenium WebDriver: Timeout Exception: Capturing whatever loaded so far..."); return driver; } cleanUpDriver(driver); throw new RuntimeException(e); } return driver;}public void cleanUpDriver(WebDriver driver) { if (driver != null) { try { // be sure to close every driver you opened driver.close(); driver.quit(); //service.stop(); do not stop the service, bcz it is needed TemporaryFilesystem.getDefaultTmpFS().deleteTemporaryFiles(); } catch (Exception e) { throw new RuntimeException(e); } }}}
Good luck and I hope you don't see that crash issue anymore
Please comment your success
Best regards,