One of the most challenging things to do is ‘making the right choice.’ Arriving at a decision becomes even more complicated when there are multiple options in front of you☺. The same is the case with choosing a testing framework for .NET Core. The three major C# Unit testing frameworks are MSTest, NUnit, and xUnit.Net. You should select the most appropriate test framework that suits your project requirements. In this blog, we will see a detailed comparison between NUnit vs. XUnit vs. MSTest.
We have earlier covered all the three frameworks independently and recommend you to refer to those articles for detailed information on these test frameworks. In this article, we will briefly go through the features and compare the frameworks (NUnit vs. XUnit vs. MSTest) so that you can arrive at a decision to select the best-suited framework.
All three frameworks use annotations/attributes to inform the underlying framework about how the source code should be interpreted. Attributes are meta-tags that give more information about the test methods and classes defined in the source code.
We would be demonstrating NUnit vs. XUnit vs. MSTest comparison using cross browser testing using the Selenium framework. When choosing a test framework, you should check the in-house expertise and check whether the framework is well-suited for the ‘type’ of project your team is working on!
NUnit
NUnit is an open-source testing framework ported from JUnit. The latest version of NUnit is NUnit3 that is rewritten with many new features and has support for a wide range of .NET platforms.
NUnit has been downloaded more than 126 million times from NuGet.org. This shows the popularity of NUnit within the .NET user community. As of writing this article, there were close to 24,000 questions tagged as NUnit on Stackoverflow. It is a popular test framework used for Selenium automation testing. If you are planning to perform Test-Driven Development (TDD) with C#, you should have a close look at the NUnit framework.
Automation testing in NUnit can be performed using the Visual Studio GUI and the NUnit console (NUnit.ConsoleRunner). NUnit framework uses an attribute/annotation style system like other test frameworks; shown below are the most popular NUnit attributes used for Selenium automation testing.
| Annotation |
Description |
| [SetUp] |
There should be at least one method per test class. Marks a method that should be called before each test method. |
| [TearDown] |
There should be at least one method per test class. Marks a method that should be called after each test method. |
| [TestFixture] |
Marks a class that contains tests. |
| [Test] |
Marks a method, i.e., an actual test case in the test class. |
| [TestCase] |
Marks a method with parameters and provides the inline arguments. |
| [TestFixtureSetUp] |
Marks a method that is executed once before the execution of any test method in that fixture. |
| [TestFixtureTearDown] |
Marks a method that is executed after the last test method in that fixture has finished execution. |
| [Ignore] |
Marks a test method or test class that should not be considered for execution, i.e., it is ignored. |
| [Category] |
Specify the category for the test |
| [Indicates] |
Marks that the test should be skipped unless it is explicitly run. |
| [OneTimeSetUp] |
Methods that should be called before any of the child tests are executed. |
| [OneTimeTearDown] |
Methods that should be called after the child tests are executed. |
| [Culture] |
Indicates cultures for which a test or test fixture should be executed. |
| [MaxTime] |
Maximum time frame in milliseconds for the successful execution of a method. A method is marked as a fail if the execution does not complete within MaxTime. |
| [Platform] |
Platforms on which the test execution should be performed. |
| [TimeOut] |
Timeout value in milliseconds for the test cases under execution. |
| [Retry] |
Test will re-run if it fails till the time it is either successful or the maximum number of times have reached. |
| [Repeat] |
Specifies the method that needs to be executed/repeated number of times. |
| [ExpectedException(ExceptionType)] |
Marks a test that is expected to raise a particular exception, i.e., ExceptionType. |
| [Random] |
Specifies generation of random values that will be passed as arguments to the parameterized tests. |
The attributes [Test] and [TestCase] are not extensible as those attributes are sealed.
Using NUnit framework, tests can be executed serially as well as parallel. Parallel test execution is possible at assembly, class, or method level. When parallelism is enabled, by default four tests can be executed in parallel. As Parallel test execution is supported in NUnit, it becomes a compelling option for Selenium automation testing and cross browser testing.
XUnit
xUnit.Net is an open-source testing framework based on the .NET framework. ‘x’ stands for the programming language, e.g., JUnit, NUnit, etc. The creators of NUnit created xUnit as they wanted to build a better framework rather than adding incremental features to the NUnit framework.
xUnit is created with more focus on the community; hence it is easy to expand upon. You can download xUnit from NuGet gallery. So far, there are close to 7,500 questions tagged as xUnit on Stackoverflow.
Below are some of the core reasons why the creators of NUnit decided to reconsider the framework and built xUnit.Net.
As of writing this article, the latest version of xUnit was 2.4.1. The framework follows a unique style of testing, and tags like [Test] & [TestFixture], a part of the NUnit framework, are no longer used in the xUnit framework.
The popular attributes [SetUp] and [TearDown] are also not a part of the xUnit framework. As per the creators of NUnit (and xUnit), usage of [SetUp] and [TearDown] led to code duplication, and they wanted to implement the same features in a much more optimized manner in xUnit. For initialization, constructor of the test class is used, whereas, for de-initialization, IDisposable interface is used. This also encourages writing much cleaner tests. This makes this C# unit testing framework a much better option when it comes to Selenium automation testing as it is more robust and extensible.
As far as NUnit vs. XUnit vs. MSTest is concerned, the biggest difference between xUnit and the other two test frameworks (NUnit and MSTest) is that xUnit is much more extensible when compared to NUnit and MSTest. The [Fact] attribute is used instead of the [Test] attribute. Non-parameterized tests are implemented under the [Fact] attribute, whereas the [Theory] attribute is used if you plan to use parameterized tests.
In NUnit and MSTest, the class that contains the tests is under the [TestClass] attribute. This was not a very robust approach hence [TestClass] attribute was also removed in xUnit. Instead, intelligence is built in the xUnit framework so that it can locate the test methods, irrespective of the location of the tests.
Shown below are the most popular attributes/annotations used in the xUnit framework:
| Annotation |
Description |
| [Fact] |
Marks a test method, i.e., actual test in a class |
| Assert.Throws Record Exception |
Verify the raise and raise assert, irrespective of the place in the code where the problem occurs. |
| Constructor |
This is not an attribute but is an ideal replacement for the [SetUp] attribute. The constructor should be parameter-less. |
| IDisposable.Dispose |
This is not an attribute but is an ideal replacement for the [TearDown] attribute. This is where the code for performing necessary cleanup and de-initialization is included.
The decision to do away with [TearDown] was made as the investors of xUnit felt that a lot of unnecessary code was run before every single test execution.
|
| [Trait] |
Used to set arbitrary meta-data on a test |
| [Theory] |
This attribute is used when data-driven tests have to be executed. In such cases, [Theory] has to be used instead of [Fact] attribute |
| [InlineData] |
This attribute is used along with the [Theory] attribute to supply a subset of data against which parameterized tests will be executed. |
| [ClassData] |
This attribute is used when the parameters being passed to the [Theory] tests are not constants.
[Theory] [ClassData(typeof(some-data))] |
| [MemberData] |
This attribute can be used to fetch data for [Theory] from a static method. The most common approach is to load the data from the property of a test class, i.e., using IEnumerable<object[]> |
MSTest
MSTest is the default test framework that is shipped along with Visual Studio. The initial version of MSTest (V1) was not open-source; however, MSTest V2 is open-source. The project is hosted on GitHub. Like other test frameworks, it can also be used for data driven testing. You can download MSTest V2 from Nuget.org.
As of writing this article, there are close to 10,000 questions with the tag MSTest on Stackoverflow. Below are the most commonly used MSTest attributes.
| Annotation |
Description |
| [TestInitialize] |
Marks a method that should be called before each test method. One such method should be present before each test class. |
| [TestCleanup] |
Marks a method that should be called after each test method. One such method should be present before each test class. |
| [TestClass] |
Marks a class that contains tests. |
| [TestMethod] |
Marks the method, i.e., the actual test case in the test class. |
| [DataRow] |
Allows setting the values of the parameters of the tests. Multiple [DataRow] annotations can be present in the code. |
| [DataTestMethod] |
It has the same functionality as the [TestMethod] attribute except that it is used when the [DataRow] attribute is used. |
| [AssemblyInitialize] |
Marks the method that should be called once before the execution of any method in the assembly code. |
| [AssemblyCleanup] |
Marks the method that should be called once after the execution of any method in the assembly code. |
| [Ignore] |
Marks a test method or test class that should be considered for execution, i.e., it is ignored. |
| [TestCategory] |
Specify the category for the test. |
| [ClassInitialize] |
Methods that will be called only once before executing any of the test methods present in that class. |
| [ClassCleanup] |
Methods that will be called only once after executing the test methods present in that class. |
MSTest V2 has cross-platform support and can be extended using custom test attributes & custom asserts. Parallel test execution is supported by the MSTest framework, where parallelism is possible at the Method level or Class level.
The test framework has evolved from being an “in-box” testing framework to a framework that is gaining wide adoption from the developer community.
NUnit vs. XUnit vs. MSTest
We now compare the C# unit testing frameworks from an attribute usage point of view along with a simple example, which demonstrates the code flow.
Attributes in test frameworks
Irrespective of the C# unit testing framework, attributes are used to describe class, methods, properties, etc. Attributes are meta-tags that provide additional information about the implementation under that particular tag.
To make the NUnit vs. XUnit vs. MSTest comparison clearer, we cover the important attributes in each of these frameworks. The NUnit vs. XUnit vs. MSTest attributes comparison will also help in porting the test implementation from one test framework to another.
| Description |
NUnit |
MSTest |
xUnit |
| Marks a test method/individual test |
[Test] |
[TestMethod] |
[Fact] |
| Indicates that a class has a group of unit tests |
[TestFixture] |
[TestClass] |
N.A |
| Contains the initialization code, which is triggered before every test case |
[SetUp] |
[TestInitialize] |
Constructor |
| Contains the cleanup code, which is triggered after every test case |
[TearDown] |
[TestCleanup] |
IDisposable.Dispose |
| Contains method that is triggered once before test cases start |
[OneTimeSetUp] |
[ClassInitialize] |
IClassFixture<T> |
| Contains method that is triggered once before test cases end |
[OneTimeTearDown] |
[ClassCleanup] |
IClassFixture<T> |
| Contains per-collection fixture setup and teardown |
N.A |
N.A |
ICollectionFixture<T> |
| Ignores a test case |
[Ignore(“reason”)] |
[Ignore] |
[Fact(Skip=”reason”)] |
| Categorize test cases or classes |
[Category()] |
[TestCategory(“)] |
[Trait(“Category”, “”) |
| Identifies a method that needs to be called before executing any test in test class/test fixture |
[TestFixtureSetup] |
[ClassInitialize] |
N.A |
| Identifies a method that needs to be called after executing any test in test class/test fixture |
[TestFixtureTearDown] |
[ClassCleanUp] |
N.A |
| Identifies a method that needs to be called before the execution of any tests in Test Assembly |
N.A |
[AssemblyInitialize] |
N.A |
| Identifies a method that needs to be called after execution of tests in Test Assembly |
N.A |
[AssemblyCleanUp] |
N.A |
To demonstrate the usage of attributes and the sequence in which the test code is executed, we take the example of simple test implementation.
NUnit Test Framework
In the example shown above, the implementation under [SetUp] and [TearDown] attributes will be called for each test case.
using NUnit.Framework;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
namespace NUnit_Test
{
class NUnit_Demo
{
[SetUp]
public void Initialize()
{
Console.WriteLine("Inside SetUp");
}
[TearDown]
public void DeInitialize()
{
Console.WriteLine("Inside TearDown");
}
public class TestClass1
{
[OneTimeSetUp]
public static void ClassInitialize()
{
Console.WriteLine("Inside OneTimeSetUp");
}
[OneTimeTearDown]
public static void ClassCleanup()
{
Console.WriteLine("Inside OneTimeTearDown");
}
}
[Test, Order(1)]
public void Test_1()
{
Console.WriteLine("Inside TestMethod Test_1");
}
[Test, Order(2)]
public void Test_2()
{
Console.WriteLine("Inside TestMethod Test_2");
}
}
}
Shown below is the execution log:
Inside SetUp
Inside TestMethod Test_1
Inside TearDown
Inside SetUp
Inside TestMethod Test_2
Inside TearDown
xUnit Test Framework
As seen in the example shown below, the [SetUp] and [TearDown] attributes from the NUnit framework are not present. Instead, a constructor is used for initialization, and a Dispose method is used for de-initialization.
xUnit creates a new instance of the test class for every test. In some time-critical test scenarios, execution of creation & cleanup code may take time and make the tests slower. Class fixtures can be used in such situations where a fixture instance is created before any of the tests are executed, and once the tests have completed execution, cleanup of the fixture object by calling Dispose (if it is present).
In the example, we make use of Class Fixtures which means that even though there are two test classes, initialization, and cleanup code will be called only once.
using System;
using Xunit;
namespace xUnit_Test
{
public class xUnit_Tests : IDisposable
{
public xUnit_Tests()
{
Console.WriteLine("Inside SetUp Constructor");
}
public void Dispose()
{
Console.WriteLine("Inside CleanUp or Dispose method");
}
}
public class UnitTest_1 : IClassFixture<xUnit_Tests>
{
[Fact]
public void Test_1()
{
Console.WriteLine("Inside Test_1");
}
}
public class UnitTest_2 : IClassFixture<xUnit_Tests>
{
[Fact]
public void Test_2()
{
Console.WriteLine("Inside Test_2");
}
}
}
Shown below is the output log:
Inside SetUp Constructor
Inside Test_1
Inside Test_2
Inside CleanUp or Dispose method
MSTest Test Framework
The test code only contains Console.WriteLine, which is put to trace the execution flow. Most of the widely used MSTest attributes are used in the demonstrated test code.
using Microsoft.VisualStudio.TestTools.UnitTesting;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading;
namespace MsTest
{
[TestClass]
public class Initialize
{
[AssemblyInitialize]
public static void AssemblyInitialize(TestContext context)
{
Console.WriteLine("Inside AssemblyInitialize");
}
}
public class DeInitialize
{
[AssemblyCleanup]
public static void AssemblyCleanup()
{
Console.WriteLine("Inside AssemblyCleanup");
}
}
[TestClass]
public class TestClass1
{
[ClassInitialize]
public static void ClassInitialize(TestContext context)
{
Console.WriteLine("Inside ClassInitialize");
}
[ClassCleanup]
public static void ClassCleanup()
{
Console.WriteLine("Inside ClassCleanup");
}
[TestMethod]
public void Test_1()
{
Console.WriteLine("Inside TestMethod Test_1");
}
}
[TestClass]
public class TestClass2
{
[TestInitialize]
public void TestInitialize()
{
Console.WriteLine("Inside TestInitialize");
}
[TestMethod]
public void Test_2()
{
Console.WriteLine("Inside TestMethod Test_2");
}
[TestCleanup]
public void TestCleanup()
{
Console.WriteLine("Inside TestCleanup");
}
}
}
Shown below is the execution log:
Inside ClassInitialize
Inside TestMethod Test_1
Inside AssemblyInitialize
Inside TestInitialize
Inside TestMethod Test_2
Inside TestCleanup
The implementation under [ClassInitialize] & [ClassCleanup] attributes are called respectively once before the methods are executed and once after the execution of the test methods. The implementation under [TestInitialize] and [TestCleanup] attributes are respectively called once before & after executing tests in each class.
Core Differences between NUnit, XUnit, and MSTest
xUnit framework is different from NUnit and MSTest frameworks on multiple fronts. Let’s look at the core differences:
1. Isolation of Tests
xUnit framework provides much better isolation of tests in comparison to NUnit and MSTest frameworks. For each test case, the test class is instantiated, executed, and is discarded after the execution. This ensures that the tests can be executed in any order as there is reduced/no dependency between the tests. Executing each test as a separate instance minimizes the chances of one test causing the other tests to fail!
There is also an option to share setup/cleanup code that can be shared between different tests without the need to share object instances. One way of doing this is via Class Fixtures which we demonstrated earlier. xUnit has excellent documentation on how to share contexts between tests.
2. Extensibility
When we do NUnit vs. XUnit vs. MSTest, extensibility plays an important role in choosing a particular test framework. The choice might depend on the needs of the project, but in some scenarios, extensibility can turn the tables around for a particular test framework.
When compared to MSTest and NUnit frameworks, xUnit framework is more extensible since it makes use of [Fact] and [Theory] attributes. Many attributes that were present in NUnit framework e.g. [TestFixture], [TestFixtureSetup], [TestFixtureTearDown] [ClassCleanup], [ClassInitialize], [TestCleanup], etc. are not included in the xUnit framework.
Introduction of the [Theory] attribute for parameterized tests is one of the prime examples of the extensibility of the framework. This also makes the implementation of custom functionality much easier in the xUnit framework.
3. Initialization and De-initialization
The NUnit uses [SetUp], [TearDown] pairs whereas MSTest uses [TestInitialize], [TestCleanup] pairs for setting up the activities related to initialization & de-initialization of the test code.
On the other hand, xUnit uses the class constructor for the implementation of steps related to test initialization and IDisposable interface for the implementation of steps related to de-initialization.
xUnit starts a new instance per test, whereas, in NUnit & MSTest frameworks, all the tests execute in the same Fixture/Class.
4. Assertion mechanism
xUnit framework makes use of Assert.Throws instead of [ExpectedException] which is used in NUnit and MSTest. The drawback of using [ExpectedException] is that the errors might not be reported if they occur in the wrong part of the code. For example, if assert has to be raised for Security Exception, but Authentication Exception occurs, [ExpectedCondition] will not raise assert.
On the other hand, Assert.Throws raises assert even if the exception is generic. This ensures that assert is raised even after the exception is raised.
5. Parallel test execution
All the three C# unit testing frameworks support parallel test execution and are well-suited for Selenium automation testing as throughput plays a major role in automation testing. Below are the ways in which parallelism can be achieved in each of the test frameworks.
- NUnit – Parallelism is possible at the level of Children (child tests are executed in parallel with other tests), Fixtures (descendants of test till the level of Test Fixtures can execute in parallel), Self (test itself can be executed in parallel with other tests), and All (test & its descendants can execute in parallel with others at the same level).
[Parallelizable(ParallelScope.Children)]
[Parallelizable(ParallelScope.Fixtures)]
[Parallelizable(ParallelScope.Self)]
[Parallelizable(ParallelScope.All)]
- xUnit – Parallelism by putting the test classes into a single test collection or by executing an ‘n’ number of threads in parallel.
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]
[assembly: CollectionBehavior(MaxParallelThreads = n)]
- MSTest – Parallelism at Method level as well as Class level.
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)]
[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.ClassLevel)]
Note: Workers indicate the number of threads that can be executed in parallel.
NUnit vs. XUnit vs. MSTest – Demonstration using Remote Selenium Grid
For the initial rounds of cross browser testing can be performed on the local Selenium as it aids in verifying the functionalities of the test code on ‘certain’ combinations of browsers & operating systems. Selenium automation testing can be performed using either of the three C# unit testing frameworks.
Automation testing using the local Selenium Grid is a good start to cross browser testing activity, but local testing may not be sufficient to attain good test coverage. Having in-house infrastructure with machines having different web browsers and operating systems installed is not a scalable & economic idea.
When cross browser testing is performed using local Selenium WebDriver, the activity can hit a roadblock as you cannot perform Selenium automation testing using innumerable combinations of web browsers, operating systems, and devices. Instead of the local Selenium Grid, the capabilities of the remote Selenium Grid on the cloud can be leveraged so that testing is more fool-proof.
LambdaTest is a cross browser testing platform on the cloud where you can verify the test code on 2000+ combinations of browsers, devices, and operating systems. The effort to port the code from the local Selenium Grid to LambdaTest’s remote Selenium Grid is not much as major changes are infrastructure-related.
After creating an account on LambdaTest, make a note of the user-name and access-key from the Profile Section. The Dashboard consists of all the relevant information about the tests, status, logs, video recordings of the tests, etc. LambdaTest capabilities generator is used to generate the desired browser capabilities against which the cross browser tests will be performed. Shown below are the capabilities for Safari on the macOS Mojave platform:
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.SetCapability("user","Your Lambda Username")
capabilities.SetCapability("accessKey","Your Lambda Access Key")
capabilities.SetCapability("build", "your build name");
capabilities.SetCapability("name", "your test name");
capabilities.SetCapability("platform", "MacOS Mojave");
capabilities.SetCapability("browserName", "Safari");
capabilities.SetCapability("version","12.0");
Implementation (using NUnit C# unit testing framework)
To demonstrate the usage of Remote Selenium WebDriver on LambdaTest Grid, we implement the below test case:
- Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using selected browsers.
- Check the first two items.
- Add a new item – Adding item to the list.
- Click the Add button to add that new item to the list.
Below are the browser and operating system combinations on which the test has to be performed.
| Browser |
Browser version |
Platform/Operating System |
| Chrome |
72.0 |
Windows 10 |
| Internet Explorer |
11.0 |
Windows 10 |
| Safari |
11.0 |
macOS High Sierra |
| Microsoft Edge |
18.0 |
Windows 10 |
The browser capabilities are generated using the LambdaTest capabilities generator. The combination of user-name & access-key is used to access LambdaTest’s remote Selenium Grid.
using System;
using OpenQA.Selenium;
using OpenQA.Selenium.Remote;
using NUnit.Framework;
using System.Threading;
using System.Collections.Generic;
namespace ParallelLTSelenium
{
[TestFixture("chrome", "72.0", "Windows 10")]
[TestFixture("internet explorer", "11.0", "Windows 10")]
[TestFixture("Safari", "11.0", "macOS High Sierra")]
[TestFixture("MicrosoftEdge", "18.0", "Windows 10")]
[Parallelizable(ParallelScope.All)]
public class ParallelLTTests
{
ThreadLocal<IWebDriver> driver = new ThreadLocal<IWebDriver>();
private String browser;
private String version;
private String os;
public ParallelLTTests(String browser, String version, String os)
{
this.browser = browser;
this.version = version;
this.os = os;
}
[SetUp]
public void Init()
{
String username = "user-name";
String accesskey = "access-key";
String gridURL = "@hub.lambdatest.com/wd/hub";
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.SetCapability("user", username);
capabilities.SetCapability("accessKey", accesskey);
capabilities.SetCapability("browserName", browser);
capabilities.SetCapability("version", version);
capabilities.SetCapability("platform", os);
driver.Value = new RemoteWebDriver(new Uri("https://" + username + ":" + accesskey + gridURL), capabilities, TimeSpan.FromSeconds(600));
System.Threading.Thread.Sleep(2000);
}
[Test]
public void LT_ToDo_Test()
{
{
String test_url_1 = "https://lambdatest.github.io/sample-todo-app/";
driver.Value.Url = test_url_1;
String itemName = "Adding item to the list";
System.Threading.Thread.Sleep(2000);
// Click on First Check box
IWebElement firstCheckBox = driver.Value.FindElement(By.Name("li1"));
firstCheckBox.Click();
// Click on Second Check box
IWebElement secondCheckBox = driver.Value.FindElement(By.Name("li2"));
secondCheckBox.Click();
// Enter Item name
IWebElement textfield = driver.Value.FindElement(By.Id("sampletodotext"));
textfield.SendKeys(itemName);
// Click on Add button
IWebElement addButton = driver.Value.FindElement(By.Id("addbutton"));
addButton.Click();
// Verified Added Item name
IWebElement itemtext = driver.Value.FindElement(By.XPath("/html/body/div/div/div/ul/li[6]/span"));
String getText = itemtext.Text;
// Check if the newly added item is present or not using
// Condition constraint (Boolean)
Assert.That((itemName.Contains(getText)), Is.True);
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
}
}
[TearDown]
public void Cleanup()
{
bool passed = TestContext.CurrentContext.Result.Outcome.Status == NUnit.Framework.Interfaces.TestStatus.Passed;
try
{
// Logs the result to Lambdatest
((IJavaScriptExecutor)driver.Value).ExecuteScript("lambda-status=" + (passed ? "passed" : "failed"));
}
finally
{
// Terminates the remote webdriver session
driver.Value.Quit();
}
}
}
}
Parallel test execution is enabled using the Parallelizable attribute, and test combinations are passed via TestFixtures. The actual logic of the test code is under the [Test] attribute. Shown below is the snapshot from the Automation Tab on LambdaTest and Visual Studio:
Implementation (using xUnit C# unit testing framework)
For demonstrating parallel test execution using xUnit framework on remote Selenium Grid, we test the following scenarios:
Test Case 1 – LamdaTest ToDo App
- Navigate to the to-do app https://lambdatest.github.io/sample-todo-app/ using the Firefox WebDriver.
- Mark the first two items as Done, i.e., Check those two items.
- Add a new item – Adding an item to the list.
- Click the Add button to add that new item to the list.
Browsers on which cross browser testing is performed are:
Test Case 1
| Browser |
Browser version |
Platform/Operating System |
| Chrome |
72.0 |
Windows 10 |
| Microsoft Edge |
18.0 |
Windows 10 |
| Firefox |
70.0 |
macOS High Sierra |
| Safari |
12.0 |
macOS Mojave |
Test Case 2 & 3 – Google Search for LambdaTest
Navigate to Google.com.
Search for LambdaTest.
Quit the browser window.
Both the test cases are the same, but the execution will be performed on different web browsers.
Test Case 2
| Browser |
Browser version |
Platform/Operating System |
| Chrome |
72.0 |
Windows 10 |
| Microsoft Edge |
18.0 |
Windows 10 |
| Firefox |
70.0 |
macOS High Sierra |
Test Case 3
| Browser |
Browser version |
Platform/Operating System |
| Microsoft Edge |
18.0 |
Windows 10 |
| Firefox |
70.0 |
macOS High Sierra |
| Safari |
12.0 |
macOS Mojave |
using Xunit;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using System.Threading.Tasks;
using System.Text.RegularExpressions;
using OpenQA.Selenium.Support.UI;
using OpenQA.Selenium.Interactions;
/* For using Remote Selenium WebDriver */
using OpenQA.Selenium.Remote;
[assembly: CollectionBehavior(MaxParallelThreads = 4)]
namespace ParallelLTSelenium
{
public class GlobalVar
{
static int _globalValue;
public static int GlobalValue
{
get
{
return _globalValue;
}
set
{
_globalValue = value;
}
}
public static String username = "user-name";
public static String accesskey = "access-key";
public static String gridURL = "@hub.lambdatest.com/wd/hub";
public static DesiredCapabilities capabilities = new DesiredCapabilities();
public static IWebDriver driver1;
}
public class ParallelLTTests : IDisposable
{
public ParallelLTTests()
{
GlobalVar.capabilities.SetCapability("user", GlobalVar.username);
GlobalVar.capabilities.SetCapability("accessKey", GlobalVar.accesskey);
}
public void Dispose()
{
// Closure handled in each test case
}
}
public class UnitTest_1 : IClassFixture<ParallelLTTests>
{
[Theory]
[InlineData("chrome", "72.0", "Windows 10")]
[InlineData("MicrosoftEdge", "18.0", "Windows 10")]
[InlineData("Firefox", "70.0", "Windows 10")]
[InlineData("Safari", "12.0", "macOS Mojave")]
public void LT_ToDo_Test(String browser, String version, String os)
{
String itemName = "Yey, Let's add it to list";
IWebDriver driver;
GlobalVar.capabilities.SetCapability("browserName", browser);
GlobalVar.capabilities.SetCapability("version", version);
GlobalVar.capabilities.SetCapability("platform", os);
GlobalVar.capabilities.SetCapability("build", "[xUnit - 1] LT ToDoApp using Xunit in Parallel on LambdaTest");
GlobalVar.capabilities.SetCapability("name", "[xUnit - 1] - LT ToDoApp using Xunit in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + GlobalVar.username + ":" + GlobalVar.accesskey + GlobalVar.gridURL), GlobalVar.capabilities, TimeSpan.FromSeconds(2000));
driver.Url = "https://lambdatest.github.io/sample-todo-app/";
Assert.Equal("Sample page - lambdatest.com", driver.Title);
// Click on First Check box
IWebElement firstCheckBox = driver.FindElement(By.Name("li1"));
firstCheckBox.Click();
// Click on Second Check box
IWebElement secondCheckBox = driver.FindElement(By.Name("li2"));
secondCheckBox.Click();
// Enter Item name
IWebElement textfield = driver.FindElement(By.Id("sampletodotext"));
textfield.SendKeys(itemName);
// Click on Add button
IWebElement addButton = driver.FindElement(By.Id("addbutton"));
addButton.Click();
// Verified Added Item name
IWebElement itemtext = driver.FindElement(By.XPath("/html/body/div/div/div/ul/li[6]/span"));
String getText = itemtext.Text;
Assert.True(itemName.Contains(getText));
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("LT_ToDo_Test Passed");
driver.Close();
driver.Quit();
}
}
public class UnitTest_2 : IClassFixture<ParallelLTTests>
{
[Theory]
[InlineData("chrome", "72.0", "Windows 10")]
[InlineData("MicrosoftEdge", "18.0", "Windows 10")]
[InlineData("Firefox", "70.0", "Windows 10")]
public void Google_Test_1(String browser, String version, String os)
{
IWebDriver driver;
GlobalVar.capabilities.SetCapability("browserName", browser);
GlobalVar.capabilities.SetCapability("version", version);
GlobalVar.capabilities.SetCapability("platform", os);
GlobalVar.capabilities.SetCapability("build", "[xUnit - 2] - Google search (1) using XUnit in Parallel on LambdaTest");
GlobalVar.capabilities.SetCapability("name", "[xUnit - 2] - Google search (1) using XUnit in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + GlobalVar.username + ":" + GlobalVar.accesskey + GlobalVar.gridURL), GlobalVar.capabilities, TimeSpan.FromSeconds(2000));
driver.Url = "https://www.google.com";
System.Threading.Thread.Sleep(2000);
IWebElement element = driver.FindElement(By.XPath("//*[@id='tsf']/div[2]/div[1]/div[1]/div/div[2]/input"));
element.SendKeys("LambdaTest");
/* Submit the Search */
element.Submit();
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("Google_Test Passed");
driver.Close();
driver.Quit();
}
}
public class UnitTest_3 : IClassFixture<ParallelLTTests>
{
[Theory]
[InlineData("chrome", "72.0", "Windows 10")]
[InlineData("MicrosoftEdge", "18.0", "Windows 10")]
[InlineData("Firefox", "70.0", "macOS High Sierra")]
[InlineData("Safari", "12.0", "macOS Mojave")]
public void Google_Test_2(String browser, String version, String os)
{
IWebDriver driver;
GlobalVar.capabilities.SetCapability("browserName", browser);
GlobalVar.capabilities.SetCapability("version", version);
GlobalVar.capabilities.SetCapability("platform", os);
GlobalVar.capabilities.SetCapability("build", "[xUnit - 2] - Google Search (2) using XUnit in Parallel on LambdaTest");
GlobalVar.capabilities.SetCapability("name", "[xUnit - 2] - Google Search (2) using XUnit in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + GlobalVar.username + ":" + GlobalVar.accesskey + GlobalVar.gridURL), GlobalVar.capabilities, TimeSpan.FromSeconds(2000));
driver.Url = "https://www.google.com";
System.Threading.Thread.Sleep(2000);
IWebElement element = driver.FindElement(By.XPath("//*[@id='tsf']/div[2]/div[1]/div[1]/div/div[2]/input"));
element.SendKeys("LambdaTest");
/* Submit the Search */
element.Submit();
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("Google_Test Passed");
driver.Close();
driver.Quit();
}
}
}
The setup/initialization code is a part of the constructor, whereas the implementation under IDisposable [i.e., public void Dispose()] can be used to perform the de-initialization. For simplification, de-initialization is moved to the individual test cases. Shown below is the output snapshot from LambdaTest and Visual Studio.
Implementation (using MSTest C# unit testing framework)
To demonstrate the usage of parallelism with the MSTest framework, we implement the test cases used for the xUnit framework. The browser and operating system combinations for testing are also the same.
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
/* For using Remote Selenium WebDriver */
using OpenQA.Selenium.Remote;
using System;
using System.Threading;
[assembly: Parallelize(Workers = 5, Scope = ExecutionScope.MethodLevel)]
namespace ParallelLTSelenium
{
[TestClass]
public class ParallelLTTests
{
IWebDriver driver;
String username = "user-name";
String accesskey = "access-key";
String gridURL = "@hub.lambdatest.com/wd/hub";
DesiredCapabilities capabilities;
[TestInitialize]
public void setupInit()
{
capabilities = new DesiredCapabilities();
capabilities.SetCapability("user", username);
capabilities.SetCapability("accessKey", accesskey);
}
[DataTestMethod]
[DataRow("chrome", "72.0", "Windows 10")]
[DataRow("MicrosoftEdge", "18.0", "Windows 10")]
[DataRow("Firefox", "70.0", "macOS High Sierra")]
[DataRow("Safari", "12.0", "macOS Mojave")]
[TestMethod]
public void LT_ToDo_Test(String browser, String version, String os)
{
String itemName = "Yey, Let's add it to list";
capabilities.SetCapability("browserName", browser);
capabilities.SetCapability("version", version);
capabilities.SetCapability("platform", os);
capabilities.SetCapability("build", "LT ToDoApp using MsTest in Parallel on LambdaTest");
capabilities.SetCapability("name", "LT ToDoApp using MsTest in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + username + ":" + accesskey + gridURL), capabilities, TimeSpan.FromSeconds(2000));
driver.Url = "https://lambdatest.github.io/sample-todo-app/";
Assert.AreEqual("Sample page - lambdatest.com", driver.Title);
// Click on First Check box
IWebElement firstCheckBox = driver.FindElement(By.Name("li1"));
firstCheckBox.Click();
// Click on Second Check box
IWebElement secondCheckBox = driver.FindElement(By.Name("li2"));
secondCheckBox.Click();
// Enter Item name
IWebElement textfield = driver.FindElement(By.Id("sampletodotext"));
textfield.SendKeys(itemName);
// Click on Add button
IWebElement addButton = driver.FindElement(By.Id("addbutton"));
addButton.Click();
// Verified Added Item name
IWebElement itemtext = driver.FindElement(By.XPath("/html/body/div/div/div/ul/li[6]/span"));
String getText = itemtext.Text;
Assert.IsTrue(itemName.Contains(getText));
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("LT_ToDo_Test Passed");
}
[DataTestMethod]
[DataRow("chrome", "72.0", "Windows 10")]
[DataRow("MicrosoftEdge", "18.0", "Windows 10")]
[DataRow("Firefox", "70.0", "macOS High Sierra")]
[TestMethod]
public void Google_Test_1(String browser, String version, String os)
{
capabilities.SetCapability("browserName", browser);
capabilities.SetCapability("version", version);
capabilities.SetCapability("platform", os);
capabilities.SetCapability("build", "Google search (1) using MsTest in Parallel on LambdaTest");
capabilities.SetCapability("name", "Google search (1) using MsTest in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + username + ":" + accesskey + gridURL), capabilities, TimeSpan.FromSeconds(2000));
//System.Threading.Thread.Sleep(2000);
driver.Url = "https://www.google.com";
IWebElement element = driver.FindElement(By.XPath("//*[@id='tsf']/div[2]/div[1]/div[1]/div/div[2]/input"));
element.SendKeys("LambdaTest");
/* Submit the Search */
element.Submit();
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("Google_Test Passed");
}
[DataTestMethod]
[DataRow("chrome", "72.0", "Windows 10")]
[DataRow("MicrosoftEdge", "18.0", "Windows 10")]
[DataRow("Firefox", "70.0", "macOS High Sierra")]
[DataRow("Safari", "12.0", "macOS Mojave")]
[TestMethod]
public void Google_Test_2(String browser, String version, String os)
{
capabilities.SetCapability("browserName", browser);
capabilities.SetCapability("version", version);
capabilities.SetCapability("platform", os);
capabilities.SetCapability("build", "Google Search (2) using MsTest in Parallel on LambdaTest");
capabilities.SetCapability("name", "Google Search (2) using MsTest in Parallel on LambdaTest");
driver = new RemoteWebDriver(new Uri("https://" + username + ":" + accesskey + gridURL), capabilities, TimeSpan.FromSeconds(2000));
//System.Threading.Thread.Sleep(2000);
driver.Url = "https://www.google.com";
IWebElement element = driver.FindElement(By.XPath("//*[@id='tsf']/div[2]/div[1]/div[1]/div/div[2]/input"));
element.SendKeys("LambdaTest");
/* Submit the Search */
element.Submit();
/* Perform wait to check the output */
System.Threading.Thread.Sleep(2000);
Console.WriteLine("Google_Test Passed");
}
[TestCleanup]
public void Cleanup()
{
if (driver != null)
driver.Quit();
}
}
}
Each test case is under a separate [TestMethod] attribute. Since we have implemented three tests (LT_ToDo_Test, Google_Test_1, and Google_Test_2), three concurrent sessions will be executed till the time the number of pending tests are less than three. The [DataRow] attribute is used to create parameterized tests.
Shown below is the output snapshot from the Automation tab on LambdaTest and Visual Studio:
Conclusion
Each of the .NET frameworks that we have discussed so far has evolved over a period of time. All these frameworks are well-suited for cross browser testing as they have support for Selenium. MSTest V2 is a giant leap in terms of usability when compared to MSTest V1. The xUnit framework is more sophisticated as it is more community-focused, and the creators had immense learning while building the NUnit framework.
To conclude, the NUnit vs. XUnit vs. MSTest comparison is more about which C# test framework is well-suited to perform your job and the test expertise that your team possesses. If I had an option to select a framework, I would go with xUnit since it is more extensible and has fewer attributes, making the code clean & easy to maintain.