Category / Continuous Integration

by Dima Korobskiy

How slow can you go?

System performance testing is often done as a high-ceremony performed for major releases only.

  • Project on its own, designing:
    • Requirement Analysis and Gathering
    • PoC/Tool selection
    • Performance Test Plan
    • Performance Test Modeling
    • Test Execution
    • Test Analysis
  • Done in a Test environment designed similar to Production
    • Highly controlled environment
  • Aiming to produce high loads and hard results: numbers, pass/fail
  • Done by performance testing professionals

In today’s Agile and DevOps era teams are craving for rapid feedback and that certainly includes the impact of changes on system performance from one build to another. Performance testing can be done with reduced ceremony, continuously:

  • As a part of a build pipeline
  • In a lower-end Test environment
  • Aiming to produce build trends because results could be more volatile
  • Done by a DevOps team

Continuous Performance Testing is not difficult to set up on most projects. All you need is the following three ingredients, and each of them could be an open-source tool:

  1. Load Generator Tool
  2. Continuous Integration Server
  3. Chart Generator

The overall architecture (the big picture) of a solution relies on the usual Continuous Integration approach:

 

Load Generator Tools

Load generators can load a system-under-test with (HTTP) requests simulating the activity of virtual users. They support load logic (# of simulated clients, # of loops, spikes, fixed time testing, etc.) as well as simple scripting and assertion logic. Load generation could also be distributed to multiple machines.

Open source choices include:

  • Apache JMeter: a mature performance testing IDE. Tests are developed primarily via composition and configuration of JMeter components.
  • Gatling: Scala code-based tests, which can be generated recorder UI gene5
  • Locust: Python code-based tests

You can find a more comprehensive comparison here: Open Source Load Testing Tools: Which One Should You Use?

Commercial tools include LoadRunner, Telerik Test Studio and New Relic Synthetics. Notably, Microsoft performance testing offerings in Visual Studio Enterprise and Azure DevOps were deprecated in 2011 and Microsoft recommends a migration path to JMeter.

Finally, load generators can be hosted and used as a service. Offerings in this space include BlazeMeter and WebPagetest (a cross of performance testing and web page loading profiler).

Continuous Integration (CI) Servers

Jenkins is a popular open-source choice offering a wide variety of plug-ins, including chart generators. Any other CI server is also an option.

Chart Generators

The critical part of Continuous Performance Testing is providing feedback to the DevOps team, ideally, on every build. This could be done in the form of generated charts showing performance trends. This could be done by feeding raw output data (e.g. CSVs, XMLs) to dashboards. An easier solution to implement though is to use CI server built-in or third-party chart generation capabilities. There are two good options that can be used if you’re running Jenkins.

Jenkins Performance Plug-In

Jenkins Performance Plug-In can be used to chart runtime fo executed tets. It understands JMeter, JUnit and Taurus formats. Taurus is a wrapper around 20+ other load generators.

Usage Tips:

  • Use Advanced > Display Performance Report Per Test Case = on. This would chart one line per a performance test case (e.g. a JMeter Sampler) average data point.
  • Chart titles = data file names. You can omit .xml suffix and give files verbose names with spaces to use them as chart titles. Patterns containing spaces would be matched. For example, the build/jmeter-report/ * pattern would match any data file name that starts from space.
  • Performance Plug-in would publish reports only for data files found in the latest build. Data files previously found in builds, but not present in the latest build are ignored. If you want to start a fresh baseline, you can do this by simply renaming data file(s).
  • If there is a single data file, its charts are published on the main job page. If there are several reports, charts could be found on the job’s Performance Trend page, ordered by titles.
  • You have to have at least two builds in order to see trend charts. If you have a test, which was temporarily disabled or removed then re-enabled, you would need to have it in at least two builds in order to see a corresponding trend line.
  • Deleted builds are automatically removed from reports.
  • It is recommended to split reports containing more than 10-20 tests so that they are easier to read.

 

Jenkins Plot Plug-In

Jenkins Plot Plug-In can be used to chart arbitrary generated data. It’s useful for fixed-time performance tests generating, for example, Transactions Per Second (TPS) metrics.

Advertisement
by Dima Korobskiy

Automate your Feature and Acceptance Tests in Four Easy Steps

Any significant web application at some point faces the question of how to approach testing of user-facing functionality. Developers could be (hopefully) happily churning out their JUnit-s, NUnit-s and Test::Unit-s, practicing TDD and Continuous Integration, but what does testing your classes, methods, and functions have to do with what your customers actually need from your system?

The answer is: not much, really. Testing web application from the end-user perspective involves dealing with test cases written in a business language, and secondly, firing multiple browsers and hunting down why this particular feature works perfectly fine in a browser Foo and does not work at all in a browser Baz. The right tools for automating this process are Behavior-Driven Development tools and browser “drivers”. These tools come from different development worlds and communities: Ruby, .NET, Java.

Fortunately, you can use many of these tools to test any web application regardless of your application platform whether it’s JavaEE, .NET, ColdFusion, PHP or Ruby on Rails. The tools came the long way since early days of Fit/Fitnesse and you can get from zero feature tests to automation in few easy steps using any desktop OS: Windows, Mac OS X or Linux. I promise.

What might be the most difficult part of this process is bewildering number of choices to make. A lot of choice is a Good Thing and all the tools mentioned in this post are free, open-source tools, but how it all fits together could be quite confusing. Fear not. Let’s walk through the steps.

Step 1. Select and Install a BDD Tool

Behavior-Driven Development is taking automated unit testing a level higher. The emphasis is on a language and readability of tests (or rather scenarios, broken down to steps in BDD parlance). One of the most active communities championing BDD tools is Ruby world. The framework that generated a lot of buzz in the last couple of years is Cucumber. It is a flexible Ruby framework that uses a wonderful domain-specific language for writing your scenarios: Gherkin. Gherkin is pretty much structured English or one of 37 other supported spoken languages. Cucumber natively uses Ruby for glue code (step definitions) that ties your features, scenarios and steps to browser drivers (or to native target applications).

Now, if you don’t know Ruby you might be tempted to go with one of Cucumber ports, for example cucumber-jvm (Java) or SpecFlow (.NET). You would get to write glue code in your programming language of choice, but the setup if often more complex, ports could be less mature or lagging behind flagship Ruby tools. It is hard to come up with other benefits of ports for testing web applications. Why not use this opportunity and learn a fun dynamic language with a lot of interesting ideas? It is used as the first language to teach kids, after all, and you don’t have to learn Rails and a lot of other Ruby frameworks, just the language itself.

My choice: Cucumber

Other choices: Fit/Fitnesse, SpecFlow, RSpec, easyb, etc.

What to do:

  • Windows: Install Ruby 1.8.x and add it to the PATH. Latest versions of Mac OS X come with Ruby 1.8.x. Don’t use Ruby 1.9.x just yet (see below).
  • Set RUBYOPT=-rubygems in your environment.
  • Go to your command line and install the Cucumber Ruby gem. 
    >gem install cucumber

    Note: Any gem installation might fail with the following:
    ERROR: http://rubygems.org/ does not appear to be a repository
    This problem is transient. Try again in 5 minutes. Another option is to download needed .gem manually and install it:

    >cd {download directory}
>gem install -lV cucumber

    The local installation (-l) will install Gem dependencies only if they are available in the Gem caches or in the local directory. You might need to download dependencies manually as well.

  • Windows: (optional, but highly recommended in order to display colored output) Install ANSICON.
  • Install RSpec for writing assertions in your glue code. 
    >gem install rspec
Pages: 1 2 3 4