Built by Maven

Introduction

The WebClient represents the browser when you work with HtmlUnit. To start using HtmlUnit you have to instantiate a new WebClient—like starting a browser in the real world.

WebClient implements AutoCloseable; you should always use it with try-with-resources constructs. After a WebClient is closed (see WebClient.close()), any further use is not supported and might lead to exceptions or incorrect behavior. 


try (final WebClient webClient = new WebClient()) {
    // now you have a running browser, and you can start doing real things
    // like going to a web page
    final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
}

Imitating a specific browser

Often you will want to simulate a specific browser. This is done by passing a org.htmlunit.BrowserVersion into the WebClient constructor. Constants have been provided for some common browsers. 


@Test
public void homePage_Firefox() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
        final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

Specifying this BrowserVersion will change:

  • the user agent HTTP header,
  • the values and order of many other HTTP headers,
  • the list of supported MIME types,
  • the behavior of the web client,
  • the supported JavaScript methods and the behavior of some JS functions, and
  • the default values for various CSS properties

to match real browsers.

In most cases, it should be sufficient to use the predefined BrowserVersion constants.

Using the options to adjust the browser

There are various options available to make fine-grained adjustments to the browser. 


@Test
public void homePage_Firefox() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
         // disable javascript
         webClient.getOptions().setJavaScriptEnabled(false);
         // disable css support
         webClient.getOptions().setCssEnabled(false);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

The default values for most options are similar to the default values of real browsers, but (as always) there is one important exception:

HtmlUnit stops JavaScript execution at the first unhandled exception—browsers do not stop. You can change this by changing the throwExceptionOnScriptError option to false. 


@Test
public void homePage_Firefox() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
         // proceed with the js execution on unhandled js errors
         webClient.getOptions().setThrowExceptionOnScriptError(false);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

For a complete list and more details please have a look at the WebClientOptions API.

Change the browser language / time zone

Changing the language/time zone cannot be done from the options; it must be done before the WebClient is created.

All BrowserVersions are shipped with 'en-US' as the language and 'America/New_York' as the timezone.

To change these default settings, a customized copy of the corresponding BrowserVersion must be created using the BrowserVersionBuilder. This new BrowserVersion can then be used to create a WebClient. 


final BrowserVersion.BrowserVersionBuilder builder = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX);

builder.setSystemTimezone(TimeZone.getTimeZone("Europe/Berlin"));
builder.setBrowserLanguage("de-DE");
builder.setAcceptLanguageHeader("de-DE,de");

final BrowserVersion germanFirefox = builder.build();
try (final WebClient webClient = new WebClient(germanFirefox)) {
    // ...
}

There is no support for changing the language/timezone after the WebClient has been created.

For more details please have a look at the BrowserVersion.BrowserVersionBuilder API.

Change the browser user agent

Changing the user agent is similar to changing language/time zone (see above).

You have to create a customized copy of the corresponding BrowserVersion using the BrowserVersionBuilder. This adapted BrowserVersion can then be used to create a WebClient. 


final BrowserVersion.BrowserVersionBuilder builder = new BrowserVersion.BrowserVersionBuilder(BrowserVersion.FIREFOX);

builder.setUserAgent("Mozilla/5.0 (iPhone; CPU iPhone OS 14_5 like Mac OS X) "
        + "AppleWebKit/605.1.15 (KHTML, like Gecko) FxiOS/128.0 Mobile/15E148 Safari/605.1.15");

final BrowserVersion iosFirefox = builder.build();
try (WebClient webClient = new WebClient(iosFirefox)) {
    // ...
}

For more details please have a look at the BrowserVersion.BrowserVersionBuilder API.

Using HtmlUnit behind a proxy

Using a http proxy

There is a special WebClient constructor that allows you to specify proxy server information in those cases where you need to connect through one. 


@Test
public void homePage_proxy() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX, PROXY_HOST, PROXY_PORT)) {

        //set proxy username and password 
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password");

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}

In case the proxy server requires credentials you can define them on the DefaultCredentialsProvider from the webClient. 


@Test
public void homePage_proxy() throws Exception {
    try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX, PROXY_HOST, PROXY_PORT)) {

        //set proxy username and password 
        final DefaultCredentialsProvider credentialsProvider = (DefaultCredentialsProvider) webClient.getCredentialsProvider();
        credentialsProvider.addCredentials("username", "password", PROXY_HOST, PROXY_PORT);

        final HtmlPage page = webClient.getPage("https://www.htmlunit.org");
        Assert.assertEquals("HtmlUnit – Welcome to HtmlUnit", page.getTitleText());
    }
}