Поиск  
Always will be ready notify the world about expectations as easy as possible: job change page
Jan 16, 2023

C# parse HTML with AngleSharp

C# parse HTML with AngleSharp
Источник:
Просмотров:
4881

C# AngleSharp tutorial shows how to parse HTML in C# with AngleSharp library.

The library can also parse SVG, MathML, or XML.

AngleSharp GitHub: https://github.com/AngleSharp

Document Object Model (DOM) is a standard tree structure, where each node contains one of the components from an XML structure. Element nodes and text nodes are the two most common types of nodes. With DOM functions we can create nodes, remove nodes, change their contents, and traverse the node hierarchy.

Nodes, elements, and tags are synonyms.

words.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Words</title>
</head>
<body>

<ul>
    <li>sky</li>
    <li>cup</li>
    <li>water</li>
    <li>cloud</li>
    <li>bear</li>
    <li>wolf</li>
</ul>

</body>
</html>

In a few examples, we use this HTML file.

C# AngleSharp parse HTML string

In the first example, we read a HTML document from a string.

Program.cs

using AngleSharp;

var html = @"
<html>
<head>
<title>My Title</title>
</head>
<body>
<h1>Heading I</h1>
<p>Paragraph I</p>
<p>Paragraph II</p>
</body>
</html>";

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

Console.WriteLine(doc.Title);
Console.WriteLine(doc.Body.InnerHtml.Trim());
Console.WriteLine(doc.FirstChild.NodeName.ToLower());
Console.WriteLine(doc.LastChild.NodeName.ToLower());

We print the document's title, body, and first and the last child name.

var config = Configuration.Default;
using var context = BrowsingContext.New(config);

To set up AngleSharp, we define a BrowsingContext to which we pass a Configuration.

using var doc = await context.OpenAsync(req => req.Content(html));

The document is loaded with OpenAsync.

Console.WriteLine(doc.Title);

With Title attribute, we get the title of the document.

Console.WriteLine(doc.Body.InnerHtml.Trim());

Through the Body attribute, we get the inner HTML of the document's body.

Console.WriteLine(doc.FirstChild.NodeName.ToLower());
Console.WriteLine(doc.LastChild.NodeName.ToLower());

We get the names of the first and last children of the document.

$ dotnet run
My Title
<h1>Heading I</h1>
<p>Paragraph I</p>
<p>Paragraph II</p>
html
html

C# AngleSharp QuerySelectorAll

The QuerySelectorAll method returns a list of the elements within the document that match the specified group of selectors.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var html = @"
<html>
<p>words</p>
<ul>
<li>sky</li>
<li>cup</li>
<li>rock</li>
<li>water</li>
</ul>
</html>";

using var context = BrowsingContext.New(Configuration.Default);
using var doc = await context.OpenAsync(req => req.Content(html));
var els = doc.QuerySelectorAll("li");

foreach (var e in els)
{
    Console.WriteLine(e.Text());
}

In the example, we get all li tags. With the Text method, we get the content text of the node.

$ dotnet run
sky
cup
rock
water

C# AngleSharp parse HTML file

In the following example, we load the document from an HTML file.

Program.cs

using AngleSharp;

var html = File.ReadAllText(@"/home/janbodnar/Documents/words.html");

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

var lis = doc.QuerySelectorAll("li");

foreach (var li in lis)
{
    Console.WriteLine(li.InnerHtml);
}

We read the HTML file into a string with File.ReadAllText. Then we select all li tags and print their inner HTML. (In this case it is equal to the Text method.)

C# AngleSharp modify document

In the following example, we load an HTML file and modify it. The modified document is written to the disk.

Program.cs

using AngleSharp;
using AngleSharp.Text;

var html = File.ReadAllText(@"/home/janbodnar/words.html");

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

var wli = doc.CreateElement("li");
wli.TextContent = "smile";

var ul = doc.QuerySelector("ul");
ul.FirstElementChild.Remove();
ul.AppendChild(wli);

var p = doc.CreateElement("p");
p.TextContent = $"New paragraph{Symbols.NoBreakSpace}";
doc.Body.AppendChild(p);

File.WriteAllText("/home/janbodnar/words2.html", doc.ToHtml());

In the example, we add a two elements and remove one element.

var wli = doc.CreateElement("li");
wli.TextContent = "smile";

A new element is created with CreateElement. We set its content using the TextContent attribute.

var ul = doc.QuerySelector("ul");
ul.FirstElementChild.Remove();
ul.AppendChild(wli);

We locate the ul element and remove its first child with Remove. Then we append the newly created element with AppendChild.

var p = doc.CreateElement("p");
p.TextContent = $"New paragraph{Symbols.NoBreakSpace}";
doc.Body.AppendChild(p);

We also add a paragraph at the end of the body.

File.WriteAllText("/home/janbodnar/words2.html", doc.ToHtml());

The modified document is saved to the disk with WriteAllText.

C# AngleSharp parse HTML page

In the next example, we load an HTML page from a simple website.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var config = Configuration.Default.WithDefaultLoader();
using var context = BrowsingContext.New(config);

var url = "http://webcode.me";

using var doc = await context.OpenAsync(url);
// var title = doc.QuerySelector("title").InnerHtml;
var title = doc.Title;

Console.WriteLine(title);

var pars = doc.QuerySelectorAll("p");

foreach (var par in pars)
{
    Console.WriteLine(par.Text().Trim());
}

From the page's HTML document, we get its title and the content of the two paragraphs.

var config = Configuration.Default.WithDefaultLoader();

To enable reading from HTML pages, we call the WithDefaultLoader method.

$ dotnet run
My html page
Today is a beautiful day. We go swimming and fishing.
Hello there. How are you?

C# async parsing of HTML pages

In the next example, we asynchronously parse several HTML pages.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var urls = new[] {
    "http://webcode.me", "http://example.com",
    "http://httpbin.org", "https://google.com",
    "https://www.rust-lang.org/", "https://golang.org/",
    "https://fsharp.org/", "https://clojure.org/",
    "https://www.perl.org/", "https://gnu.org"
};

var config = Configuration.Default.WithDefaultLoader();
using var context = BrowsingContext.New(config);

var docs = new List<Task<IDocument>>();

foreach (var url in urls)
{
    docs.Add(context.OpenAsync(url));
}

Task.WaitAll(docs.ToArray());

foreach (var t in docs)
{
    var res = t.Result;
    Console.WriteLine(res.Title);

    res.Dispose();
}

The example asynchronously loads several web pages and parses their titles.

$ dotnet run
My html page
Example Domain
httpbin.org
Google
Rust Programming Language
The Go Programming Language
F# Software Foundation
Clojure
The Perl Programming Language - www.perl.org
The GNU Operating System and the Free Software Movement

C# AngleSharp search results

In the next example, we submit a request to the Google search engine and process the result.

Program.cs

using System.Text.RegularExpressions;
using AngleSharp;
using AngleSharp.Dom;
using AngleSharp.Html.Dom;

using var context = BrowsingContext.New(Configuration.Default.WithDefaultLoader());
using var doc = await context.OpenAsync("https://www.google.com/");
var form = doc.QuerySelector<IHtmlFormElement>("form[action='/search']");

var term = "F# language";

using var res = await form.SubmitAsync(new {q = term});
var links = res.QuerySelectorAll<IHtmlAnchorElement>("a");

foreach (var link in links)
{
    var val = link.Attributes["href"].Value;
    
    if (!val.Contains("google"))
    {
        var rx = new Regex(@"(/url\?q=)([^&]*)(\&)", RegexOptions.Compiled);
        var match = rx.Match(val);

        if (match.Success)
        {
            Console.WriteLine(match.Groups[2]);
        }
    }
}

The example prints all the links that the search engine returns for the given search term. 

var form = doc.QuerySelector<IHtmlFormElement>("form[action='/search']");

We locate the form.

using var res = await form.SubmitAsync(new {q = term});

The form is submitted with SubmitAsync.

var links = res.QuerySelectorAll<IHtmlAnchorElement>("a");

From the result, we get all the links.

foreach (var link in links)
{
    var val = link.Attributes["href"].Value;
...

We go through the links and get the contents of their href attribute, which has the link.

if (!val.Contains("google"))

We want to exclude the search engine links, such as links to terms, privacy, and similary.

var rx = new Regex(@"(/url\?q=)([^&]*)(\&)", RegexOptions.Compiled);

if (match.Success)
{
    Console.WriteLine(match.Groups[2]);
}

We use a regular expression to process the link. The links we are interested in start with the /url?q= prefix. The search engine appends some parameters to the links after the & suffix. We divide the regex expression into three groups and pick the second one.

$ dotnet run
https://fsharp.org/
https://fsharp.org/learn/
https://fsharp.org/specs/language-spec/
https://fsharp.org/about/
https://fsharp.org/use/windows/
https://docs.microsoft.com/en-us/dotnet/fsharp/what-is-fsharp
https://medium.com/skills-matter/what-is-the-most-underrated-programming-language-fa...
https://www.planetgeek.ch/2020/12/16/c-vs-f/
https://en.wikipedia.org/wiki/F_Sharp_(programming_language)
https://en.wikipedia.org/wiki/F_Sharp_(programming_language)
https://sk.wikipedia.org/wiki/F_Sharp
...

In this article we have parsed HTML documents in C# with AngleSharp.

Похожее
Sep 16
Author: Stanley Ulili
Logging libraries vary in strengths, with some better suited for specific tasks than others. Certain projects demand lightweight solutions, while others require robust capabilities to handle high log data volumes. As a result, choosing a logging library can be challenging,...
Mar 24
Author: Mohammad Hussain
Introduction A common table expression (CTE) is a temporary named result set that you can reference within a SELECT, INSERT, UPDATE, or DELETE statement in SQL Server. CTEs were introduced in SQL Server 2005 and are similar to derived tables...
Apr 18
Author: Michael Shpilt
One of the most used databases these days is PostgreSQL (aka Postgres). Its accomplishments include being the most popular DB [among professional developers] according to Stack Overflow survey of 2022, the database in all of the fastest TechEmpower benchmarks, and...
Dec 1, 2023
Author: Rico Fritzsche
The flight monitor case study: Applying Vertical Slices in a real-world scenario In my last articles I talked about Vertical Slice Architecture and how to organize code in a non-technical way. I showed some simple examples of how to do...
Написать сообщение
Тип
Почта
Имя
*Сообщение