1. Overview
Jsoup is an open source Java library used mainly for extracting data from HTML. It also allows you to manipulate and output HTML. It has a steady development line, great documentation, and a fluent and flexible API. Jsoup can also be used to parse and build XML.
In this tutorial, we’ll use the Spring Blog to illustrate a scraping exercise that demonstrates several features of jsoup:
- Loading: fetching and parsing the HTML into a Document
- Filtering: selecting the desired data into Elements and traversing it
- Extracting: obtaining attributes, text, and HTML of nodes
- Modifying: adding/editing/removing nodes and editing their attributes
2. Maven Dependency
To make use of the jsoup library in your project, add the dependency to your pom.xml:
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.10.2</version>
</dependency>
You can find the latest version of jsoup in the Maven Central repository.
3. Jsoup at a Glance
Jsoup loads the page HTML and builds the corresponding DOM tree. This tree works the same way as the DOM in a browser, offering methods similar to jQuery and vanilla JavaScript to select, traverse, manipulate text/HTML/attributes and add/remove elements.
If you’re comfortable with client-side selectors and DOM traversing/manipulation, you’ll find jsoup very familiar. Check how easy it is to print the paragraphs of a page:
Document doc = Jsoup.connect("http://example.com").get();
doc.select("p").forEach(System.out::println);
Bear in mind that jsoup interprets HTML only — it does not interpret JavaScript. Therefore changes to the DOM that would normally take place after page loads in a JavaScript-enabled browser will not be seen in jsoup.
4. Loading
The loading phase comprises the fetching and parsing of the HTML into a Document. Jsoup guarantees the parsing of any HTML, from the most invalid to the totally validated ones, as a modern browser would do. It can be achieved by loading a String, an InputStream, a File or a URL.
Let’s load a Document from the Spring Blog URL:
String blogUrl = "https://spring.io/blog"; Document doc = Jsoup.connect(blogUrl).get();
Notice the get method, it represents an HTTP GET call. You could also do an HTTP POST with the post method (or you could use a method which receives the HTTP method type as a parameter).
If you need to detect abnormal status codes (e.g. 404), you should catch the HttpStatusException exception:
try {
Document doc404 = Jsoup.connect("https://spring.io/will-not-be-found").get();
} catch (HttpStatusException ex) {
//...
}
Sometimes, the connection needs to be a bit more customized. Jsoup.connect(…) returns a Connection which allows you to set, among other things, the user agent, referrer, connection timeout, cookies, post data, and headers:
Connection connection = Jsoup.connect(blogUrl);
connection.userAgent("Mozilla");
connection.timeout(5000);
connection.cookie("cookiename", "val234");
connection.cookie("cookiename", "val234");
connection.referrer("http://google.com");
connection.header("headersecurity", "xyz123");
Document docCustomConn = connection.get();
Since the connection follows a fluent interface, you can chain these methods before calling the desired HTTP method:
Document docCustomConn = Jsoup.connect(blogUrl)
.userAgent("Mozilla")
.timeout(5000)
.cookie("cookiename", "val234")
.cookie("anothercookie", "ilovejsoup")
.referrer("http://google.com")
.header("headersecurity", "xyz123")
.get();
You can learn more about the Connection settings by browsing the corresponding Javadoc.
5. Filtering
Now that we have the HTML converted into a Document, it’s time to navigate it and find what we are looking for. This is where the resemblance with jQuery/JavaScript is more evident, as its selectors and traversing methods are similar.
5.1. Selecting
The Document select method receives a String representing the selector, using the same selector syntax as in a CSS or JavaScript, and retrieves the matching list of Elements. This list can be empty but not null.
Let’s take a look at some selections using the select method:
Elements links = doc.select("a");
Elements sections = doc.select("section");
Elements logo = doc.select(".spring-logo--container");
Elements pagination = doc.select("#pagination_control");
Elements divsDescendant = doc.select("header div");
Elements divsDirect = doc.select("header > div");
You can also use more explicit methods inspired by the browser DOM instead of the generic select:
Element pag = doc.getElementById("pagination_control");
Elements desktopOnly = doc.getElementsByClass("desktopOnly");
Since Element is a superclass of Document, you can learn more about working with the selection methods in the Document and Element Javadocs.
5.2. Traversing
Traversing means navigating across the DOM tree. Jsoup provides methods that operate on the Document, on a set of Elements, or on a specific Element, allowing you to navigate to a node’s parents, siblings, or children.
Also, you can jump to the first, the last, and the nth (using a 0-based index) Element in a set of Elements:
Element firstSection = sections.first(); Element lastSection = sections.last(); Element secondSection = sections.get(2); Elements allParents = firstSection.parents(); Element parent = firstSection.parent(); Elements children = firstSection.children(); Elements siblings = firstSection.siblingElements();
You can also iterate through selections. In fact, anything of type Elements can be iterated:
sections.forEach(el -> System.out.println("section: " + el));
You can make a selection restricted to a previous selection (sub-selection):
Elements sectionParagraphs = firstSection.select(".paragraph");
6. Extracting
We now know how to reach specific elements, so it’s time to get their content — namely their attributes, HTML, or child text.
Take a look at this example that selects the first article from the blog and gets its date, its first section text, and finally, its inner and outer HTML:
Element firstArticle = doc.select("article").first();
Element timeElement = firstArticle.select("time").first();
String dateTimeOfFirstArticle = timeElement.attr("datetime");
Element sectionDiv = firstArticle.select("section div").first();
String sectionDivText = sectionDiv.text();
String articleHtml = firstArticle.html();
String outerHtml = firstArticle.outerHtml();
Here are some tips to bear in mind when choosing and using selectors:
- Rely on “View Source” feature of your browser and not only on the page DOM as it might have changed (selecting at the browser console might yield different results than jsoup)
- Know your selectors as there are a lot of them and it’s always good to have at least seen them before; mastering selectors takes time
- Use a playground for selectors to experiment with them (paste a sample HTML there)
- Be less dependent on page changes: aim for the smallest and least compromising selectors (e.g. prefer id. based)
7. Modifying
Modifying encompasses setting attributes, text, and HTML of elements, as well as appending and removing elements. It is done to the DOM tree previously generated by jsoup – the Document.
7.1. Setting Attributes and Inner Text/HTML
As in jQuery, the methods to set attributes, text, and HTML bear the same names but also receive the value to be set:
- attr() – sets an attribute’s values (it creates the attribute if it does not exist)
- text() – sets element inner text, replacing content
- html() – sets element inner HTML, replacing content
Let’s look at a quick example of these methods:
timeElement.attr("datetime", "2016-12-16 15:19:54.3");
sectionDiv.text("foo bar");
firstArticle.select("h2").html("<div><span></span></div>");
7.2. Creating and Appending Elements
To add a new element, you need to build it first by instantiating Element. Once the Element has been built, you can append it to another Element using the appendChild method. The newly created and appended Element will be inserted at the end of the element where appendChild is called:
Element link = new Element(Tag.valueOf("a"), "")
.text("Checkout this amazing website!")
.attr("href", "http://baeldung.com")
.attr("target", "_blank");
firstArticle.appendChild(link);
7.3. Removing Elements
To remove elements, you need to select them first and run the remove method.
For example, let’s remove all <li> tags that contain the “navbar-link” class from Document, and all images from the first article:
doc.select("li.navbar-link").remove();
firstArticle.select("img").remove();
7.4. Converting the Modified Document to HTML
Finally, since we were changing the Document, we might want to check our work.
To do this, we can explore the Document DOM tree by selecting, traversing, and extracting using the presented methods, or we can simply extract its HTML as a String using the html() method:
String docHtml = doc.html();
The String output is a tidy HTML.
8. Conclusion
Jsoup is a great library to scrape any page. If you’re using Java and don’t require browser-based scraping, it’s a library to take into account. It’s familiar and easy to use since it makes use of the knowledge you may have on front-end development and follows good practices and design patterns.
You can learn more about scraping web pages with jsoup by studying the jsoup API and reading the jsoup cookbook.
The source code used in this tutorial can be found in the GitHub project.