Thursday, May 2, 2019

FAQs for Visual Studio load testing

https://docs.microsoft.com/en-us/azure/devops/test/load-test/reference-qa?view=azure-devops

Q: What are virtual users?

A: Virtual users create load by accessing your app or web site all at the same time during your test run. That way, you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.

Q: How many virtual users can I configure in my load test?

A: In the full version of Visual Studio Enterprise, the number of virtual users is unlimited. In Visual Studio Enterprise trial version, the virtual user count is limited to 250.

Q: Is there a difference between what I can analyze during a running test versus a completed test?

A: Yes, these are the differences:

Performance counters A smaller subset of the performance counter data is available while a test is running.

Views When the load test run has completed, the Summary View and Details View are available.

Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times to simulate the time spent by a user on a web page.

Q: Why should I use Cloud-based Load Testing?

A: If you don't want to set up machines for load testing, or you don't have available resources, you can use the Cloud-based Load Testing service. It sets up virtual machines in the cloud that will run your load test. Note that your web site must be publicly available on the internet for load testing using Azure DevOps to access it.

Q: Are there any limits when running the cloud-based load tests?

A: Yes. Based on where you're running the test, each test run duration limit is:

Visual Studio IDE: 48 hours
Azure DevOps Load test page:
- URL-based load tests: 48 hours
- JMeter load tests: 4 hours
Azure portal: 1 hour

Q: Can I have other test types, besides web performance tests, in a load test mix?

A: Yes, you can include unit tests and coded web tests, but not coded UI tests.

Q: Does it support any other Web Test file formats?

A: At present only Visual Studio Web Test format files are supported. We'd be pleased to hear from you if you need support for other file formats. Email us at vsoloadtest@microsoft.com.

Q: What are test agents? How do they relate to my test run?

A: Test agents are computing resources, like CPU, memory, and network, that generate load by simulating virtual users. Test agents use agent cores to create virtual users. Each core creates at least 1 virtual user.

For load test runs in Azure DevOps with the Visual Studio IDE, you can specify the number of cores to use. For example, if you get errors when you run your test, you might have to increase the number of cores.

Otherwise, your tests and the number of virtual users that you specify determine how many cores and agents are used.

Q: What is the maximum test duration and number of concurrent users?

A: The limitations for load testing in the Azure Portal depend on the web application service tier license type, as follows:

License type	Max duration (mins)	Max user load (VUser)
Free	1	40
Shared	30	1,000
Basic/Standard/Premium	60	20,000

Q: Where do I specify the number of cores for runs in Azure DevOps with the Visual Studio IDE?

A: You can do that here:

What do the values mean?

Cores	Agents
0	(Default) The number of cores is based on the number of virtual users that you specify for your test.
1	Your test run will use 1 agent.
2 - 10	Each agent will always use 2 cores.
11 - 40	Each agent will always use 4 cores.
41 - 200	Each agent will always use 8 cores.

The maximum number of cores for each test run is 200. If your test run needs more cores, you can run up to 10 load tests at the same time.

The minimum number of virtual users per agent core is 1. If your load test requires more cores, contact vsoloadtest@microsoft.com.

The number of agents also depends on your text mix (web performance test or unit test). If you have only web performance tests, we suggest using between 600 and 2,500 virtual users for every two cores. If you have unit tests, the agent count depends on what your unit tests do. This means you will have to test if you have enough agents by running a shorter duration load test run or use goal-based load testing.

Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?

A: If your test run uses 25 or more virtual users per core, then VUMs = (max virtual user load for your test run) * (test run duration in minutes).

If your test run uses fewer than 25 users per core, then VUMs = (number of cores) * (25 virtual users per core) * (test run duration in minutes).

The minimum values used to calculate VUMs are 25 virtual users and 1 minute. If your test run values are smaller than the minimum values, then those values are rounded up to meet the minimums. For example, if your test run specifies 20 virtual users for 30 seconds, then your test run will actually run with 25 virtual users for 1 minute = 25 VUMs, not 15 VUMs.

Also, test run duration is in minutes, not seconds. For example, if your test run duration is 5 minutes and 15 seconds, then that duration is rounded up to 6 minutes.

A minimum of 250 virtual user minutes, including the warm-up period, is deducted from your subscription for:

Completed runs, based on the full duration of the run
Aborted runs, based on the elapsed run duration

Tuesday, March 26, 2019

Instantiating a String object - uses escape character \ and quote @

You can instantiate a String object in the following ways:

By assigning a string literal to a String variable. This is the most commonly used method for creating a string. The following example uses assignment to create several strings. Note that in C#, because the backslash (\) is an escape character, literal backslashes in a string must be escaped or the entire string must be @-quoted.

string string1 = "This is a string created by assignment.";
Console.WriteLine(string1);
string string2a = "The path is C:\\PublicDocuments\\Report1.doc";
Console.WriteLine(string2a);
string string2b = @"The path is C:\PublicDocuments\Report1.doc";
Console.WriteLine(string2b);
// The example displays the following output:
//       This is a string created by assignment.
//       The path is C:\PublicDocuments\Report1.doc
//       The path is C:\PublicDocuments\Report1.doc

By calling a String class constructor. The following example instantiates strings by calling several class constructors. Note that some of the constructors include pointers to character arrays or signed byte arrays as parameters. Visual Basic does not support calls to these constructors. For detailed information about String constructors, see the String constructor summary.

char[] chars = { 'w', 'o', 'r', 'd' };
sbyte[] bytes = { 0x41, 0x42, 0x43, 0x44, 0x45, 0x00 };

// Create a string from a character array.
string string1 = new string(chars);
Console.WriteLine(string1);

// Create a string that consists of a character repeated 20 times.
string string2 = new string('c', 20);
Console.WriteLine(string2);

string stringFromBytes = null;
string stringFromChars = null;
unsafe
{
   fixed (sbyte* pbytes = bytes)
   {
      // Create a string from a pointer to a signed byte array.
      stringFromBytes = new string(pbytes);
   }
   fixed (char* pchars = chars)
   {
      // Create a string from a pointer to a character array.
      stringFromChars = new string(pchars);
   }
}
Console.WriteLine(stringFromBytes);
Console.WriteLine(stringFromChars);
// The example displays the following output:
//       word
//       cccccccccccccccccccc
//       ABCDE
//       word

Tuesday, February 26, 2019

Visual Studio Load Testing vusers and requirements

https://blogs.msdn.microsoft.com/testingspot/2017/02/23/guide-to-get-started-with-visual-studio-web-load-testing-and-automation/

Software Requirements. Versions and licenses

Visual Studio

The first thing you will need is Visual Studio. The Web Load & Performance Testing features of Visual studio are only available on versions: Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013.

Other type of tests such as Coded UI, Manual tests and Unit testing are also available on Visual Studio Test Professional 2015however these are all functional tests. We’ll focus on performance web tests for now, this is NOT available on VS Test Professional. You can find a feature comparison among VS versions here: Compare Visual Studio 2015 Offerings.

Virtual User Licenses

On premises. No virtual user licenses are needed. You can execute load tests with any number of virtual users (as far as your hardware resources allow you) as long as you have any of the following:

– Visual Studio Enterprise 2015 with MSDN

– Visual Studio Enterprise 2015 annual and monthly subscribers (check Visual Studio 2015 Licensing White Paper)

– Visual Studio Ultimate 2013 with MSDN (check Visual Studio 2013 Licensing White Paper)

* In Visual Studio Enterprise (or Ultimate) trial version, the virtual user count is limited to 250.

Cloud-Based testing. No extra software installation needed. You do need:

– A Visual Studio Online (VSO) account (get one here).

– That gets you 20,000 virtual user minutes every month to load test at no extra charge (check how virtual users minutes work and test duration limitations here: VSO Virtual users minutes).

Controller and Agents Software

On premises. If you are planning to load test an application in an on-premises environment, you will need a Controller and at least 1 agent machine (check next section for hardware requirements). You can find the installable software for these two components based on your Visual studio version here:

– Download Agents for Microsoft Visual Studio 2013 Update 5. This is the correct version of the Agents and Controllers software for load testing. Don’t use the Test Agents for VS 2015 for load testing. It is meant for continuous tests on build scenarios, it doesn’t have a test controller or a configuration tool.

Cloud-Based Load testing. No additional software installation needed.

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Visual Studio is used to record webtests, kick off Load Tests and visualize results. It is not recommended to install it on the Controller machine or Test Agents. The system requirements for the machine to host either Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013 are:

Windows 7 SP1 or above, or
Windows Server 2008 R2 SP1 or above.
A minimum of 1.6GHz, 1.0 GB of RAM (1.5 if working on a VM).
10 GB of available HD space (20 GB for VS 2013).
Check details here:
- Visual Studio Enterprise 2015 with Update 1
- Visual Studio Ultimate 2013 with Update 5

Controller and Agents Hardware Requirements

On premises. If you are planning a load test project in a on premises environment, you will need a controller and at least 1 test agent, to set this up you will need the following:

OS/Framework requirements
- Controller: Windows 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2
- Test Agents: Windows XP SP3 / 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2.
- Both: .NET Framework 4.5
- More information here: System requirements for Controller and Agents

You can also execute local test runs (without controller or test agents) directly from your Visual Studio host, this is done normally for debugging purposes since you are limited by resources.

Hardware sizing

The table below shows the recommended hardware requirements. You can use this table to plan how many test agents and what size of controller suit your load testing needs:

Configuration	Component	CPU	HD	Memory (RAM)
< 500 virtual users	Test agent	2.6 GHz	10 GB	2 GB
< 1000 virtual users	Test agent	Dual processor 2.6 GHz	10 GB	2 GB
N x 1000 virtual users	Test agent	Scale out to N agents each with Dual 2.6 Ghz	10 GB	2 GB
< 30 computers in the test environment. This includes agents and servers under test.	Test Controller	2.6 GHz	Read load test repository section
N x 30 computers in the test environment. This includes agents and servers under test.	Test Controller	N 2.6 GHz processors	Read load test repository section

Tips:

A rule of thumb that I use is that a Test Agent in a VM can hold from 500 to 1000 webtest virtual users on a typical load test. If you have many extraction plugins or external dependencies(CSV files, database connections) that number will decrease since it takes more processing to do that.
Test Agents save the load test results in a temp folder before sending everything to the Controller. You may want to monitor the free space in the HD and periodically clean the temporary folders. It's usually in this path: \\\c$\Users\\AppData\Local\VSEQT\QTAgent\\AGENT01\
Keep in mind other Server resources aside from CPU and RAM Memory. It's always a good practice to add the Test Agents to your Scenario for monitoring.
- In one of my test engagements I assumed that 3 BIG Test Agent VMs would be equivalent to 9 SMALL Test Agent VMs because the specs CPU and RAM specs matched up. However during the load test I found that this caused a bottleneck on the Test Agents because my virtual users were in queue waiting for the Agents to free threads.

Load Test Repository

Load tests results may be stored in the Load Test Results Repository, which is a SQL database. The Results Repository database is created by setup for controllers (or created automatically on the first local run of a load test), a the database will be created automatically if the load test schema is not present.

SQL Server 2012 Express LocalDB, which is installed with Visual Studio is the default database server for load tests. If SQL Server Express is detected with an existing load test database, Visual Studio Enterprise will try to connect to it and use it.

However, for heavier database needs you should consider upgrading to a full SQL Server to provide further scaling potential. Also, SQL Express is limited to using a maximum of 4 GB of disk space. If you will run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product if available. More on Load Test results repository.

Tip: In my experience, a typical single 1 hour load test can take from 100MB to 500MB of storage depending on the number of webtests, performance counters, VMs you are monitoring, etc. Plan storage needs accordingly and take in consideration that this information is temporarily stored in the Test Agents. When the test ends the results are collected in the Controller's RAM memory before finally being placed in the SQL Repository, so also plan RAM Memory needs accordingly.

Alternative: Testing using VSO or TFS instead of Controller

For test scenarios (still on premises) using Visual Studio Online (VSO) or Team Foundation Server (TFS) 2015, you won’t need a test controller because Agents for Microsoft Visual Studio 2015 handle orchestration by communicating with VSO or TFS 2015. For example, you’re running automated tests with your build and release workflows in VSO or TFS 2015.

Cloud-Based Load testing

You can use cloud-based load testing to avoid having to use your own resources and machines. In this case you don’t need a Controller or Test Agents. The cloud-service provides virtual machines in the cloud that generate the load needed to test your target application/website.

All you need is a Visual Studio Online (VSO) account. Check ‘Getting started with Load Tests’ further down in this document.

Friday, January 25, 2019

Run Tests through VS Command line and configure scheduling...

Running Web and Load tests from the Command-Line

slumley MSFTDecember 22, 20083

This blog post will show you how to run a web test or load test from the command-line instead of from within Visual Studio. Most of the time you will want to run you web or load test from within visual studio, but you can run them from a command-line. Maybe you have a set of 5 web tests that you want to run once a day. You could configure a scheduled task to run a batch file which contained ran the 5 web tests. The command line executable that is used to run tests is called mstest.exe and is installed when you install either VS Team Suite or VS Team Test SKU. Here is a MSDN help link for mstest: http://msdn.microsoft.com/en-us/library/ms182486.aspx

The easiest way to get started with mstest is to start a visual studio command prompt. Go to Start Menu -> All programs -> Microsoft Visual Studio v9.0 -> Visual Studio Tools -> Visual Studio 9.0 Command Prompt. This will launch a command window which has mstest.exe in its path. So type mstest /? This will show each of the available options. Let’s run through a few options of how to run tests.

1) Running a single WebTest

First let’s run a simple web test. To do this, we just need to specify the /TestContainer argument. A .webtest file or .loadtest file is considered a test container in the same way that a dll which contains unit tests is a test container. So if we have a web test called WebTest1.webtest, you would use the following command to run the web test:

mstest /TestContainer:WebTest1.webtest

This will produce output similar to the following:

Loading WebTest1.webtest...

Starting execution...

Results               Top Level Tests
-------               ---------------
Passed                webtest1.webtest

1/1 test(s) Passed

Summary
-------
Test Run Completed.

Passed 1
---------
Total 1

Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\Projects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_38_14.trx

2) Running Multiple WebTests

Now let’s execute 2 web tests from one call to mstest. You can specify multiple test container arguments on one command-line. So say we wanted to execute WebTest1.webtest and WebTest2.webtest. You would use the following command-line:

mstest /TestContainer:WebTest1.webtest /TestContainer:WebTest2.webtest

This will produce output similar to the following:

Loading WebTest1.webtest...

Loading WebTest2.webtest...

Starting execution...

Results Top Level Tests

------- ---------------

Passed webtest1.webtest

Passed webtest2.webtest

2/2 test(s) Passed

Summary

-------

Test Run Completed.

Passed 2

---------

Total 2

Results file: C:\Documents and Settings\slumley\My Documents\Visual Studio 10\P

rojects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_4

3_04.trx

3) Specifying deployment items

One of the big differences with running from the command line is that you are not within a visual studio project. When you run a web test from within Visual Studio, we will try to determine what needs to be deployed with the web test in order for it to run. For example, maybe you have written a custom validation rule or a extraction rule. When you run the test from within visual studio, we will look at the references of the test project and try to figure out what needs to be deployed.

When you are running from the command-line, you will need to be very explicit about what you deploy. So if you have a dll that needs to be deployed for a test to run, you will need to inform mstest. The way you do this is by specifying /runconfig parameter. One of the sections of a run config is deployment items. Use visual studio to create a run config. Then to run a test with a runconfig, you would have a command-line similar to the following:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

4) Executing a test on a controller/agent setup

When in VS, to run a web or load test on a controller/agent setup you need to use a test run configuration. You would open the run config, specify to run the test remotely and specify a controller name. It works the same way from the command-line. You will need to create a run config that has the correct settings. Then you run it the same way as you do when specifying deployment items:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

5) Running a coded web test

Running a coded web test is similar to running a unit test. You will set the /testcontainer argument to the dll which contains the coded test.

mstest /TestContainer:TestProject1.dll

One thing you will notice when running a test this way, is that mstest will execute all of the tests within the dll.

6) Executing a specific coded test in a dll

So if you want to execute just one test within a dll, you would need to use the /test argument. For example, to run WebTest1Coded, you would use the following command line:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded

If you wanted to execute 2 tests, you can specify multiple /Test arguments. For example:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded /Test:WebTest2Coded

7) Executing a load test

This is the same as executing a web test except you need to specify the .loadtest file

mstest /TestContainer:LoadTest1.loadtest

When the test completes, you can either open the trx file to get to the load test results or you can use the manage load test results dialog form within VS. Here is a blog on that: http://blogs.msdn.com/slumley/pages/managing-load-test-results.aspx

8) Specifying the results file name

You will notice that the results file(trx file) gets a unique name which contains user, machine and a timestamp. If you want to specify the name of the results file and where it is generated, you can use the /resultsfile parameter. For example, to create a results file called MyResults.trx and have it saved to c:\results, I would do the following:

mstest /TestContainer:WebTest1.webtest /resultsfile:c:\results\MyResults.trx

Hopefully this post shows you how to make use of the mstest tool.

Tuesday, January 13, 2015

Test List Editor of VSTS 2010 is deprecated in VSTS 2012

In Visual Studio 2005/2008/2010, we can drag the selected tests from the Test List Editor into the test list. This is a good feature, but it is deprecated in VSTS 2012. The new feature is called Test Explorer.

If you have 1,500+ unit tests in Test List Editor in VSTS 2010, it is not easy to upgrade those tests in VSTS 2012 because it is deprecated.

Do you think Test Explorer is user friendly compared to Test List Editor?

Performance Test Smartly with JMH/Apache Jmeter

Thursday, May 2, 2019

FAQs for Visual Studio load testing

https://docs.microsoft.com/en-us/azure/devops/test/load-test/reference-qa?view=azure-devops

Q: What are virtual users?

A: Virtual users create load by accessing your app or web site all at the same time during your test run. That way, you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.

Q: How many virtual users can I configure in my load test?

Q: Is there a difference between what I can analyze during a running test versus a completed test?

A: Yes, these are the differences:

Performance counters A smaller subset of the performance counter data is available while a test is running.

Views When the load test run has completed, the Summary View and Details View are available.

Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times to simulate the time spent by a user on a web page.

Q: Why should I use Cloud-based Load Testing?

Q: Are there any limits when running the cloud-based load tests?

Q: Can I have other test types, besides web performance tests, in a load test mix?

A: Yes, you can include unit tests and coded web tests, but not coded UI tests.

Q: Does it support any other Web Test file formats?

A: At present only Visual Studio Web Test format files are supported. We'd be pleased to hear from you if you need support for other file formats. Email us at vsoloadtest@microsoft.com.

Q: What are test agents? How do they relate to my test run?

Q: What is the maximum test duration and number of concurrent users?

Q: Where do I specify the number of cores for runs in Azure DevOps with the Visual Studio IDE?

Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?

Tuesday, March 26, 2019

Instantiating a String object - uses escape character \ and quote @

Tuesday, February 26, 2019

Visual Studio Load Testing vusers and requirements

Software Requirements. Versions and licenses

Visual Studio

Virtual User Licenses

Controller and Agents Software

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Controller and Agents Hardware Requirements

Friday, January 25, 2019

Run Tests through VS Command line and configure scheduling...

Running Web and Load tests from the Command-Line

Tuesday, January 13, 2015

Test List Editor of VSTS 2010 is deprecated in VSTS 2012

Unix/Linux Commands

Report Abuse

Thursday, May 2, 2019

Q: What are virtual users?

A: Virtual users create load by accessing your app or web site all at the same time during your test run. That way, you can test performance under more realistic or projected conditions. Virtual users are simulated by test agents.

Q: How many virtual users can I configure in my load test?

Q: Is there a difference between what I can analyze during a running test versus a completed test?

A: Yes, these are the differences: Performance counters A smaller subset of the performance counter data is available while a test is running. Views When the load test run has completed, the Summary View and Details View are available.

Q: Can virtual users simulate pausing between test steps?

A: Yes, you can specify think times to simulate the time spent by a user on a web page.

Q: Why should I use Cloud-based Load Testing?

Q: Are there any limits when running the cloud-based load tests?

Q: Can I have other test types, besides web performance tests, in a load test mix?

A: Yes, you can include unit tests and coded web tests, but not coded UI tests.

Q: Does it support any other Web Test file formats?

A: At present only Visual Studio Web Test format files are supported. We'd be pleased to hear from you if you need support for other file formats. Email us at vsoloadtest@microsoft.com.

Q: What are test agents? How do they relate to my test run?

Q: What is the maximum test duration and number of concurrent users?

Q: Where do I specify the number of cores for runs in Azure DevOps with the Visual Studio IDE?

Q: What are virtual user minutes (VUMs)? How many minutes will my load test use?

Tuesday, March 26, 2019

Tuesday, February 26, 2019

Software Requirements. Versions and licenses

Visual Studio

Virtual User Licenses

Controller and Agents Software

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Controller and Agents Hardware Requirements

Friday, January 25, 2019

Tuesday, January 13, 2015

A: Yes, these are the differences:

Performance counters A smaller subset of the performance counter data is available while a test is running.

Views When the load test run has completed, the Summary View and Details View are available.