Visual Studio Load Testing vusers and requirements

https://blogs.msdn.microsoft.com/testingspot/2017/02/23/guide-to-get-started-with-visual-studio-web-load-testing-and-automation/

Software Requirements. Versions and licenses

Visual Studio

The first thing you will need is Visual Studio. The Web Load & Performance Testing features of Visual studio are only available on versions: Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013.

Other type of tests such as Coded UI, Manual tests and Unit testing are also available on Visual Studio Test Professional 2015however these are all functional tests. We’ll focus on performance web tests for now, this is NOT available on VS Test Professional. You can find a feature comparison among VS versions here: Compare Visual Studio 2015 Offerings.

Virtual User Licenses

On premises. No virtual user licenses are needed. You can execute load tests with any number of virtual users (as far as your hardware resources allow you) as long as you have any of the following:

– Visual Studio Enterprise 2015 with MSDN

– Visual Studio Enterprise 2015 annual and monthly subscribers (check Visual Studio 2015 Licensing White Paper)

– Visual Studio Ultimate 2013 with MSDN (check Visual Studio 2013 Licensing White Paper)

* In Visual Studio Enterprise (or Ultimate) trial version, the virtual user count is limited to 250.

Cloud-Based testing. No extra software installation needed. You do need:

– A Visual Studio Online (VSO) account (get one here).

– That gets you 20,000 virtual user minutes every month to load test at no extra charge (check how virtual users minutes work and test duration limitations here: VSO Virtual users minutes).

Controller and Agents Software

On premises. If you are planning to load test an application in an on-premises environment, you will need a Controller and at least 1 agent machine (check next section for hardware requirements). You can find the installable software for these two components based on your Visual studio version here:

– Download Agents for Microsoft Visual Studio 2013 Update 5. This is the correct version of the Agents and Controllers software for load testing. Don’t use the Test Agents for VS 2015 for load testing. It is meant for continuous tests on build scenarios, it doesn’t have a test controller or a configuration tool.

 Cloud-Based Load testing. No additional software installation needed.

2. Infrastructure/Hardware requirements

Visual Studio (IDE) System Requirements

Visual Studio is used to record webtests, kick off Load Tests and visualize results. It is not recommended to install it on the Controller machine or Test Agents. The system requirements for the machine to host either Visual Studio Enterprise 2015 or Visual Studio Ultimate 2013 are:

• Windows 7 SP1 or above, or
• Windows Server 2008 R2 SP1 or above.
• A minimum of 1.6GHz, 1.0 GB of RAM (1.5 if working on a VM).
• 10 GB of available HD space (20 GB for VS 2013).
• Check details here:
  • Visual Studio Enterprise 2015 with Update 1
  • Visual Studio Ultimate 2013 with Update 5

Controller and Agents Hardware Requirements

On premises. If you are planning a load test project in a on premises environment, you will need a controller and at least 1 test agent, to set this up you will need the following:

• OS/Framework requirements
  • Controller: Windows 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2
  • Test Agents: Windows XP SP3 / 7 SP1 / 8 / 8.1 or Windows Server 2008 R2 SP1 / 2012 / 2012 R2.
  • Both: .NET Framework 4.5
  • More information here: System requirements for Controller and Agents

You can also execute local test runs (without controller or test agents) directly from your Visual Studio host, this is done normally for debugging purposes since you are limited by resources.

• Hardware sizing

The table below shows the recommended hardware requirements. You can use this table to plan how many test agents and what size of controller suit your load testing needs:

Configuration	Component	CPU	HD	Memory (RAM)
< 500 virtual users	Test agent	2.6 GHz	10 GB	2 GB
< 1000 virtual users	Test agent	Dual processor 2.6 GHz	10 GB	2 GB
N x 1000 virtual users	Test agent	Scale out to N agents each with Dual 2.6 Ghz	10 GB	2 GB
< 30 computers in the test environment. This includes agents and servers under test.	Test Controller	2.6 GHz	Read load test repository section
N x 30 computers in the test environment. This includes agents and servers under test.	Test Controller	N 2.6 GHz processors	Read load test repository section

Tips:

• A rule of thumb that I use is that a Test Agent in a VM can hold from 500 to 1000 webtest virtual users on a typical load test. If you have many extraction plugins or external dependencies(CSV files, database connections) that number will decrease since it takes more processing to do that.
• Test Agents save the load test results in a temp folder before sending everything to the Controller. You may want to monitor the free space in the HD and periodically clean the temporary folders. It's usually in this path: \\\c$\Users\\AppData\Local\VSEQT\QTAgent\\AGENT01\
• Keep in mind other Server resources aside from CPU and RAM Memory. It's always a good practice to add the Test Agents to your Scenario for monitoring.
  • In one of my test engagements I assumed that 3 BIG Test Agent VMs would be equivalent to 9 SMALL Test Agent VMs because the specs CPU and RAM specs matched up. However during the load test I found that this caused a bottleneck on the Test Agents because my virtual users were in queue waiting for the Agents to free threads.

• Load Test Repository

Load tests results may be stored in the Load Test Results Repository, which is a SQL database. The Results Repository database is created by setup for controllers (or created automatically on the first local run of a load test), a the database will be created automatically if the load test schema is not present.

SQL Server 2012 Express LocalDB, which is installed with Visual Studio is the default database server for load tests. If SQL Server Express is detected with an existing load test database, Visual Studio Enterprise will try to connect to it and use it.

However, for heavier database needs you should consider upgrading to a full SQL Server to provide further scaling potential. Also, SQL Express is limited to using a maximum of 4 GB of disk space. If you will run many load tests over a long period of time, you should consider configuring the load test results store to use an instance of the full SQL Server product if available. More on Load Test results repository.

Tip: In my experience, a typical single 1 hour load test can take from 100MB to 500MB of storage depending on the number of webtests, performance counters, VMs you are monitoring, etc. Plan storage needs accordingly and take in consideration that this information is temporarily stored in the Test Agents. When the test ends the results are collected in the Controller's RAM memory before finally being placed in the SQL Repository, so also plan RAM Memory needs accordingly.

• Alternative: Testing using VSO or TFS instead of Controller

For test scenarios (still on premises) using Visual Studio Online (VSO) or Team Foundation Server (TFS) 2015, you won’t need a test controller because Agents for Microsoft Visual Studio 2015 handle orchestration by communicating with VSO or TFS 2015. For example, you’re running automated tests with your build and release workflows in VSO or TFS 2015.

Cloud-Based Load testing

You can use cloud-based load testing to avoid having to use your own resources and machines. In this case you don’t need a Controller or Test Agents. The cloud-service provides virtual machines in the cloud that generate the load needed to test your target application/website.

All you need is a Visual Studio Online (VSO) account. Check ‘Getting started with Load Tests’ further down in this document.

Run Tests through VS Command line and configure scheduling...

Running Web and Load tests from the Command-Line

★★★★★

★★★★

★★★

★★

★

slumley MSFTDecember 22, 20083

• 0
• 0

---

This blog post will show you how to run a web test or load test from the command-line instead of from within Visual Studio.  Most of the time you will want to run you web or load test from within visual studio, but you can run them from a command-line.  Maybe you have a set of 5 web tests that you want to run once a day.  You could configure a scheduled task to run a batch file which contained ran the 5 web tests.  The command line executable that is used to run tests is called mstest.exe and is installed when you install either VS Team Suite or VS Team Test SKU.  Here is a MSDN help link for mstest:  http://msdn.microsoft.com/en-us/library/ms182486.aspx

The easiest way to get started with mstest is to start a visual studio command prompt.  Go to Start Menu ->  All programs -> Microsoft Visual Studio v9.0 -> Visual Studio Tools -> Visual Studio 9.0 Command Prompt.   This will launch a command window which has mstest.exe in its path.  So type mstest /?  This will show each of the available options.  Let’s run through a few options of how to run tests.

1)      Running a single WebTest

First let’s run a simple web test.  To do this, we just need to specify the /TestContainer argument.  A .webtest file or .loadtest file is considered a test container in the same way that a dll which contains unit tests is a test container.  So if we have a web test called WebTest1.webtest, you would use the following command to run the web test:

mstest /TestContainer:WebTest1.webtest

This will produce output similar to the following:

Loading WebTest1.webtest...

Starting execution...

Results               Top Level Tests
-------               ---------------
Passed                webtest1.webtest

1/1 test(s) Passed

Summary
-------
Test Run Completed.

  Passed  1
 ---------
  Total   1

Results file:  C:\Documents and Settings\slumley\My Documents\Visual Studio 10\Projects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_38_14.trx

2)      Running Multiple WebTests

Now let’s execute 2 web tests from one call to mstest.  You can specify multiple test container arguments on one command-line.  So say we wanted to execute WebTest1.webtest and WebTest2.webtest.  You would use the following command-line:

mstest /TestContainer:WebTest1.webtest /TestContainer:WebTest2.webtest

This will produce output similar to the following:

Loading WebTest1.webtest...

Loading WebTest2.webtest...

Starting execution...

Results               Top Level Tests

-------               ---------------

Passed                webtest1.webtest

Passed                webtest2.webtest

2/2 test(s) Passed

Summary

-------

Test Run Completed.

  Passed  2

  ---------

  Total   2

Results file:  C:\Documents and Settings\slumley\My Documents\Visual Studio 10\P

rojects\TestProject3\TestProject3\TestResults\slumley_SLUMLEYTCM 2008-12-22 13_4

3_04.trx

3)      Specifying deployment items

One of the big differences with running from the command line is that you are not within a visual studio project.  When you run a web test from within Visual Studio, we will try to determine what needs to be deployed with the web test in order for it to run.  For example, maybe you have written a custom validation rule or a extraction rule.  When you run the test from within visual studio, we will look at the references of the test project and try to figure out what needs to be deployed. 

When you are running from the command-line, you will need to be very explicit about what you deploy.  So if you have a dll that needs to be deployed for a test to run, you will need to inform mstest.  The way you do this is by specifying /runconfig parameter.  One of the sections of a run config is deployment items.  Use visual studio to create a run config.  Then to run a test with a runconfig, you would have a command-line similar to the following:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

4)      Executing  a test on a controller/agent setup

When in VS, to run a web or load test on a controller/agent setup you need to use a test run configuration.  You would open the run config, specify to run the test remotely and specify a controller name.  It works the same way from the command-line.  You will need to create a run config that has the correct settings.  Then you run it the same way as you do when specifying deployment items:

Mstest /TestContainer:WebTest1.webtest /RunConfig:NewRunConfig.testrunconfig

5)      Running a coded web test

Running a coded web test is similar to running a unit test.  You will set the /testcontainer argument to the dll which contains the coded test.

mstest /TestContainer:TestProject1.dll

One thing you will notice when running a test this way, is that mstest will execute all of the tests within the dll.

6)      Executing a specific coded test in a dll

So if you want to execute just one test within a dll, you would need to use the /test argument.  For example, to run WebTest1Coded, you would use the following command line:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded

If you wanted to execute 2 tests, you can specify multiple /Test arguments.  For example:

mstest /TestContainer:TestProject1.dll /Test:WebTest1Coded /Test:WebTest2Coded

7)      Executing a load test

This is the same as executing a web test except you need to specify the .loadtest file

mstest /TestContainer:LoadTest1.loadtest

When the test completes, you can either open the trx file to get to the load test results or you can use the manage load test results dialog form within VS.  Here is a blog on that: http://blogs.msdn.com/slumley/pages/managing-load-test-results.aspx

8)      Specifying the results file name

You will notice that the results file(trx file) gets a unique name which contains user, machine and a timestamp.  If you want to specify the name of the results file and where it is generated, you can use the /resultsfile parameter.  For example, to create a results file called MyResults.trx and have it saved to c:\results, I would do the following:

mstest /TestContainer:WebTest1.webtest /resultsfile:c:\results\MyResults.trx

Hopefully this post shows you how to make use of the mstest tool.

Test List Editor of VSTS 2010 is deprecated in VSTS 2012

In Visual Studio 2005/2008/2010, we can drag the selected tests from the Test List Editor into the test list. This is a good feature, but it is deprecated in VSTS 2012. The new feature is called Test Explorer.

If you have 1,500+ unit tests in Test List Editor  in VSTS 2010, it is not easy to upgrade those tests in VSTS 2012 because it is deprecated. 

Do you think Test Explorer is user friendly compared to Test List Editor? 

Why Visual studio 2013 doesn't have easy way to run multiple selected webtest/coded web test?

I work in a company which has 100 of client sites and I have coded web tests for all those sites as well as I have couple of coded UI test and load test to verify those sites login and up and running or not  by using visual studio 2010 ultimate, I can easily run all those tests or partial number of tests easily from the Test view or from the Test list editor of VS 2010. Every month during our IT maintenance I run all those web and coded UI tests.

Now I have installed Visual Studio 2013 ultimate, and don't see this incredible feature(test view/test list editor).

I see under load test there are options like run all tests in solution or selected test- from here I can not run easily all those tests(web and coded ui tests) together.  It is very hard to execute all those test from there.

Would you please consider this very important feature to be added in the visual studio 2013 or in the future releases of VS as it helps a lot for the automated tester. I think VS 2010 is much useful for me than VS2012 or VS2013 in this regard.

Thanks and Regards,

Abdur

===

Hi Jaeed,

One workaround is that you could run them in command line:

mstest /TestContainer: WebTest1.webtest /TestContainer: WebTest2.webtest

You could submit this feature request: http://visualstudio.uservoice.com/forums/121579-visual-studio. The Visual Studio product team is listening to user voice there. You can send your idea there and people can vote. If you submit this suggestion, I hope you could post that link here, so other community members who are interested in this issue will help you vote it. If it gets more votes, the product team expert would think about adding it as a new feature in the future product. Thanks for your understanding.

Best Regards,

How to add addtional browsers in Visual Studio 2010

Luckily we can add additional browsers by creating new .browser files in Program Files\Microsoft Visual Studio 10.0\Common7\IDE\Templates\LoadTest\Browsers (Program Files (x86) on 64-bit systems).  The files are simple XML documents that include a name and a few header items such as user agent so there isn’t really anything scary in there.  For example, the definition for IE8 is as follows:

<Browser Name="Internet Explorer 8.0" MaxConnections="6">

  <Headers>

    <Header Name="User-Agent" Value="Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1)" />

    <Header Name="Accept" Value="*/*" />

    <Header Name="Accept-Language" Value="{{$IEAcceptLanguage}}" />

    <Header Name="Accept-Encoding" Value="GZIP" />

  </Headers>

</Browser>

 

Same way you can add other browser like below:

 

Add Internet Explorer 9:

 

<Browser Name="Internet Explorer 9.0" MaxConnections="6">

  <Headers>

    <Header Name="User-Agent" Value="Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)" />

    <Header Name="Accept" Value="*/*" />

    <Header Name="Accept-Language" Value="{{$IEAcceptLanguage}}" />

    <Header Name="Accept-Encoding" Value="GZIP" />

  </Headers>

</Browser>

 

Add Internet Explorer 10:

 

<Browser Name="Internet Explorer 10.0" MaxConnections="6">

  <Headers>

    <Header Name="User-Agent" Value="Mozilla/6.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/5.0)" />

    <Header Name="Accept" Value="*/*" />

    <Header Name="Accept-Language" Value="{{$IEAcceptLanguage}}" />

    <Header Name="Accept-Encoding" Value="GZIP" />

  </Headers>

</Browser>

 

 

Add Crome 11:

 

<Browser Name="Chrome 11" MaxConnections="6">

  <Headers>   

    <Header Name="User-Agent" Value="Mozilla/5.0 (Windows NT 6.1) AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.68 Safari/534.24"/>

    <Header Name="Accept" Value="*/*" />

    <Header Name="Accept-Language" Value="{{$IEAcceptLanguage}}" />

    <Header Name="Accept-Encoding" Value="gzip" />

    <Header Name="Accept-Charset" Value="ISO-8859-1,*,utf-8" />

  </Headers>

</Browser>