Поиск  
Always will be ready notify the world about expectations as easy as possible: job change page
Jan 16, 2023

C# parse HTML with AngleSharp

C# parse HTML with AngleSharp
Источник:
Просмотров:
5749

C# AngleSharp tutorial shows how to parse HTML in C# with AngleSharp library.

The library can also parse SVG, MathML, or XML.

AngleSharp GitHub: https://github.com/AngleSharp

Document Object Model (DOM) is a standard tree structure, where each node contains one of the components from an XML structure. Element nodes and text nodes are the two most common types of nodes. With DOM functions we can create nodes, remove nodes, change their contents, and traverse the node hierarchy.

Nodes, elements, and tags are synonyms.

words.html

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Words</title>
</head>
<body>

<ul>
    <li>sky</li>
    <li>cup</li>
    <li>water</li>
    <li>cloud</li>
    <li>bear</li>
    <li>wolf</li>
</ul>

</body>
</html>

In a few examples, we use this HTML file.

C# AngleSharp parse HTML string

In the first example, we read a HTML document from a string.

Program.cs

using AngleSharp;

var html = @"
<html>
<head>
<title>My Title</title>
</head>
<body>
<h1>Heading I</h1>
<p>Paragraph I</p>
<p>Paragraph II</p>
</body>
</html>";

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

Console.WriteLine(doc.Title);
Console.WriteLine(doc.Body.InnerHtml.Trim());
Console.WriteLine(doc.FirstChild.NodeName.ToLower());
Console.WriteLine(doc.LastChild.NodeName.ToLower());

We print the document's title, body, and first and the last child name.

var config = Configuration.Default;
using var context = BrowsingContext.New(config);

To set up AngleSharp, we define a BrowsingContext to which we pass a Configuration.

using var doc = await context.OpenAsync(req => req.Content(html));

The document is loaded with OpenAsync.

Console.WriteLine(doc.Title);

With Title attribute, we get the title of the document.

Console.WriteLine(doc.Body.InnerHtml.Trim());

Through the Body attribute, we get the inner HTML of the document's body.

Console.WriteLine(doc.FirstChild.NodeName.ToLower());
Console.WriteLine(doc.LastChild.NodeName.ToLower());

We get the names of the first and last children of the document.

$ dotnet run
My Title
<h1>Heading I</h1>
<p>Paragraph I</p>
<p>Paragraph II</p>
html
html

C# AngleSharp QuerySelectorAll

The QuerySelectorAll method returns a list of the elements within the document that match the specified group of selectors.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var html = @"
<html>
<p>words</p>
<ul>
<li>sky</li>
<li>cup</li>
<li>rock</li>
<li>water</li>
</ul>
</html>";

using var context = BrowsingContext.New(Configuration.Default);
using var doc = await context.OpenAsync(req => req.Content(html));
var els = doc.QuerySelectorAll("li");

foreach (var e in els)
{
    Console.WriteLine(e.Text());
}

In the example, we get all li tags. With the Text method, we get the content text of the node.

$ dotnet run
sky
cup
rock
water

C# AngleSharp parse HTML file

In the following example, we load the document from an HTML file.

Program.cs

using AngleSharp;

var html = File.ReadAllText(@"/home/janbodnar/Documents/words.html");

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

var lis = doc.QuerySelectorAll("li");

foreach (var li in lis)
{
    Console.WriteLine(li.InnerHtml);
}

We read the HTML file into a string with File.ReadAllText. Then we select all li tags and print their inner HTML. (In this case it is equal to the Text method.)

C# AngleSharp modify document

In the following example, we load an HTML file and modify it. The modified document is written to the disk.

Program.cs

using AngleSharp;
using AngleSharp.Text;

var html = File.ReadAllText(@"/home/janbodnar/words.html");

var config = Configuration.Default;
using var context = BrowsingContext.New(config);
using var doc = await context.OpenAsync(req => req.Content(html));

var wli = doc.CreateElement("li");
wli.TextContent = "smile";

var ul = doc.QuerySelector("ul");
ul.FirstElementChild.Remove();
ul.AppendChild(wli);

var p = doc.CreateElement("p");
p.TextContent = $"New paragraph{Symbols.NoBreakSpace}";
doc.Body.AppendChild(p);

File.WriteAllText("/home/janbodnar/words2.html", doc.ToHtml());

In the example, we add a two elements and remove one element.

var wli = doc.CreateElement("li");
wli.TextContent = "smile";

A new element is created with CreateElement. We set its content using the TextContent attribute.

var ul = doc.QuerySelector("ul");
ul.FirstElementChild.Remove();
ul.AppendChild(wli);

We locate the ul element and remove its first child with Remove. Then we append the newly created element with AppendChild.

var p = doc.CreateElement("p");
p.TextContent = $"New paragraph{Symbols.NoBreakSpace}";
doc.Body.AppendChild(p);

We also add a paragraph at the end of the body.

File.WriteAllText("/home/janbodnar/words2.html", doc.ToHtml());

The modified document is saved to the disk with WriteAllText.

C# AngleSharp parse HTML page

In the next example, we load an HTML page from a simple website.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var config = Configuration.Default.WithDefaultLoader();
using var context = BrowsingContext.New(config);

var url = "http://webcode.me";

using var doc = await context.OpenAsync(url);
// var title = doc.QuerySelector("title").InnerHtml;
var title = doc.Title;

Console.WriteLine(title);

var pars = doc.QuerySelectorAll("p");

foreach (var par in pars)
{
    Console.WriteLine(par.Text().Trim());
}

From the page's HTML document, we get its title and the content of the two paragraphs.

var config = Configuration.Default.WithDefaultLoader();

To enable reading from HTML pages, we call the WithDefaultLoader method.

$ dotnet run
My html page
Today is a beautiful day. We go swimming and fishing.
Hello there. How are you?

C# async parsing of HTML pages

In the next example, we asynchronously parse several HTML pages.

Program.cs

using AngleSharp;
using AngleSharp.Dom;

var urls = new[] {
    "http://webcode.me", "http://example.com",
    "http://httpbin.org", "https://google.com",
    "https://www.rust-lang.org/", "https://golang.org/",
    "https://fsharp.org/", "https://clojure.org/",
    "https://www.perl.org/", "https://gnu.org"
};

var config = Configuration.Default.WithDefaultLoader();
using var context = BrowsingContext.New(config);

var docs = new List<Task<IDocument>>();

foreach (var url in urls)
{
    docs.Add(context.OpenAsync(url));
}

Task.WaitAll(docs.ToArray());

foreach (var t in docs)
{
    var res = t.Result;
    Console.WriteLine(res.Title);

    res.Dispose();
}

The example asynchronously loads several web pages and parses their titles.

$ dotnet run
My html page
Example Domain
httpbin.org
Google
Rust Programming Language
The Go Programming Language
F# Software Foundation
Clojure
The Perl Programming Language - www.perl.org
The GNU Operating System and the Free Software Movement

C# AngleSharp search results

In the next example, we submit a request to the Google search engine and process the result.

Program.cs

using System.Text.RegularExpressions;
using AngleSharp;
using AngleSharp.Dom;
using AngleSharp.Html.Dom;

using var context = BrowsingContext.New(Configuration.Default.WithDefaultLoader());
using var doc = await context.OpenAsync("https://www.google.com/");
var form = doc.QuerySelector<IHtmlFormElement>("form[action='/search']");

var term = "F# language";

using var res = await form.SubmitAsync(new {q = term});
var links = res.QuerySelectorAll<IHtmlAnchorElement>("a");

foreach (var link in links)
{
    var val = link.Attributes["href"].Value;
    
    if (!val.Contains("google"))
    {
        var rx = new Regex(@"(/url\?q=)([^&]*)(\&)", RegexOptions.Compiled);
        var match = rx.Match(val);

        if (match.Success)
        {
            Console.WriteLine(match.Groups[2]);
        }
    }
}

The example prints all the links that the search engine returns for the given search term. 

var form = doc.QuerySelector<IHtmlFormElement>("form[action='/search']");

We locate the form.

using var res = await form.SubmitAsync(new {q = term});

The form is submitted with SubmitAsync.

var links = res.QuerySelectorAll<IHtmlAnchorElement>("a");

From the result, we get all the links.

foreach (var link in links)
{
    var val = link.Attributes["href"].Value;
...

We go through the links and get the contents of their href attribute, which has the link.

if (!val.Contains("google"))

We want to exclude the search engine links, such as links to terms, privacy, and similary.

var rx = new Regex(@"(/url\?q=)([^&]*)(\&)", RegexOptions.Compiled);

if (match.Success)
{
    Console.WriteLine(match.Groups[2]);
}

We use a regular expression to process the link. The links we are interested in start with the /url?q= prefix. The search engine appends some parameters to the links after the & suffix. We divide the regex expression into three groups and pick the second one.

$ dotnet run
https://fsharp.org/
https://fsharp.org/learn/
https://fsharp.org/specs/language-spec/
https://fsharp.org/about/
https://fsharp.org/use/windows/
https://docs.microsoft.com/en-us/dotnet/fsharp/what-is-fsharp
https://medium.com/skills-matter/what-is-the-most-underrated-programming-language-fa...
https://www.planetgeek.ch/2020/12/16/c-vs-f/
https://en.wikipedia.org/wiki/F_Sharp_(programming_language)
https://en.wikipedia.org/wiki/F_Sharp_(programming_language)
https://sk.wikipedia.org/wiki/F_Sharp
...

In this article we have parsed HTML documents in C# with AngleSharp.

Похожее
Jan 7
Author: Sebastian Stupak
Few days ago I stopped myself while writing code. I wrote my LINQ filtering wrong. items.Where(x => x > 0).Any(); (Obviously, it’s a pseudo code) I realized my mistake immediately, changed the code, and went on with my day. Then...
Aug 15, 2021
.NET has a large number of built in exceptions. However, there maybe times when none of the built exceptions seem adequate for your particular scenario and you will need to create your own custom (AKA “user defined”) exception. This post...
Aug 18, 2023
Author: Alex Maher
C# Data Structures Handbook! Every decision in programming is a silent influencer of performance and clarity. In C#, one such vital choice is selecting the right data structure. Data structures are foundational pillars. These structures are where data lives, breathes,...
Nov 19, 2020
We will demonstrate how to setup Elasticsearch and learn how to write basics statements. You will learn about the structure, commands, tools in Elasticsearch API and get it up and running using standard settings. Introduction We will be talking about...
Написать сообщение
Тип
Почта
Имя
*Сообщение