Welcome!

Government Cloud Authors: Elizabeth White, Yeshim Deniz, Liz McMillan, Gopala Krishna Behara, Raju Myadam

Related Topics: @DevOpsSummit, Containers Expo Blog, @CloudExpo

@DevOpsSummit: Blog Post

Virtual Users, Parallel Testing & More By @Neotys | @DevOpsSummit [#DevOps]

Q&A with Performance Expert Brad Stoner

This week's post is taken from a collection of stories and a Q&A session based around the experiences of Brad Stoner, an expert in testing, performance and Virtual Users. Brad is a senior performance engineer at Neotys and was the manager of the Load and Performance Team at H&R Block. In his tenure there, he managed five people and successfully improved application performance and quality for the company. In a given year, it wasn't uncommon for Brad's team to be tasked with more than fifty projects around the performance testing process. This experience led Brad to found his own venture in Sandbreak Digital Solutions, a consulting company with a focus in web application performance testing, web page optimization, front end optimization, capacity testing, infrastructure validation and cloud testing. With twelve years of experience in IT, Brad has held multiple roles that range from systems engineering to operations management and we can certainly learn a lot from him.

The Neighbor's Cookies
In this webcast, Brad recounts multiple examples from his experience in which his classic performance testing methods came up short in some fashion. He explains what he did to correct course in each situation and how he learned from it. In a story called "Keeping An Eye On Your Neighbors," Brad explains how he was able to narrow down a client's problem to neighboring servers with too many connections, leading to spikes in CPU. In another story called "Outside The Firewall," he learned that testing from both inside and outside the firewall (not in isolation) is important as some defects can only be found at full load.

In another example called "All About The Cookies," Brad explains how a client had called him with a catastrophe in production. Everything had been tested and should have been completely fine. There were no large media or marketing occurrences at that time, so this performance spike was an odd thing to see. Brad looked at the servers and saw some were handling a much higher load than others - not the same as when the tests had occurred. His client was using cookie persistence to spread the load evenly during the test, but since load generators were sticking to web servers, the entire test they ran was bad. After looking at the data, Brad could see that large numbers of users were logging into one particular application at the same time and it was causing CPU to spike. As a result, users were getting kicked off right when they logged on! The ultimate culprit: incorrect cookie usage.

If you would like to hear more stories and lessons from Brad about application performance testing, here's the complete webcast recording.

Q&A With Brad Stoner

1. What is the impact of having performance testing tools installed on a virtual machine instead of a physical machine to generate load?

Brad: Honestly, to generate the load, I like to have a controller on a physical box. But, if I'm running a test from the cloud, it's all virtual anyways. It's hard to tell in reality if it's virtual or physical box so unless it's a perfect storm conditions (like in the "Neighbors" story), you're not going to notice a difference. I'm just as confident running a test from a physical box - both from my controller and my load generators - as I am from a VM. I'm not phased anymore. Five years ago had you asked me, I would have said, "I'm not doing anything on a VM". That's come a long way.

2. You told a story of a tester who ran a report during a long-running load test because he suspected it might cause problems and wanted to see if that was the case. If the tester knew this report would cause errors, why didn't he bring it up beforehand?

Brad: If you are familiar with a system and have an idea of where there might be a weakness, setting up and saying you are going to push that weakness causes the tester to look much closer at the system. If you do a test without really telling anyone, and it's in plain view, it's a lot easier to tell that this will be a significant problem by seeing what results. It's great because the team will perform these activities, a surprise problem occurs and I can pinpoint it right then and there. Then we repeat it to stabilize the system and begin to isolate the one-offs very quickly.

3. How do you calculate the number of virtual users to test with?

Brad: That's a good question. When looking at load testing profiles, we look at two things. The first is the amount of work being done on a system at a transactional rate. Second is the number of virtual users. If you mess with those two things, you can really have problems with your performance results. You need to identify how long the use case takes in real life, how many times a use case needs to be executed, do the simple math to get your total concurrency and get your number of virtual users online.

4. When generating load tests, is it not possible to have functional or integration testing run (for example, with Selenium testing)?

Brad: It's possible. If you have an internet facing app, you can do what you want. You can have regression testing, functional testing, manual or by hand. It's on the right path to get all of these activities to occur at the same time. In performance, if looking at the 80/20 rule, you're not treating a lot of pieces of the application. So, the more pieces you can get in there to better emulate what going on in production, the more defects you will find and the more success you are going to have when the application launches. You might have to use other tools, but what's important is how you're exercising the application, that's the main focus.

5. What do you think would be a good remedy for database connection or session timeouts?

Brad: Don't kill persistent connections (with firewall issues). It's done as a protective measure. Applications aren't that resilient. When performance testing, you're not focused on resilience. Negative testing is a great way to see what can happen from these activities. If you can do it, kill connections, bring down a server or database and pass the data to Operations or Development Operations and say if this is what you see, here's the problem. Use the data to define the incident quickly. Always look at database connection pooling. I will never write a test report again and say everything's good without looking at trends in connections, threads, open cursors and 10-12 other areas of an application that may come back to bite you if you tested longer. Look at response time and CPU, but really dig down to other things that interact with the user, to ensure that those don't have some kind of wobble or negative performance trend in them.

Key Takeaways
Hopefully you were able to pick up a few knowledge points and lessons from Brad's follies and experience in performance testing. After hearing quite a few examples of how things went wrong, we were able to learn how to make our tests better. Things such as connecting with agile and development teams more often, altering penetrations and performance testing among many other tidbits can completely change the outcome of your test and the end user experience. Hopefully these and the many other lessons in this webcast will guide you to better testing.

More Stories By Tim Hinds

Tim Hinds is the Product Marketing Manager for NeoLoad at Neotys. He has a background in Agile software development, Scrum, Kanban, Continuous Integration, Continuous Delivery, and Continuous Testing practices.

Previously, Tim was Product Marketing Manager at AccuRev, a company acquired by Micro Focus, where he worked with software configuration management, issue tracking, Agile project management, continuous integration, workflow automation, and distributed version control systems.

@ThingsExpo Stories
Headquartered in Plainsboro, NJ, Synametrics Technologies has provided IT professionals and computer systems developers since 1997. Based on the success of their initial product offerings (WinSQL and DeltaCopy), the company continues to create and hone innovative products that help its customers get more from their computer applications, databases and infrastructure. To date, over one million users around the world have chosen Synametrics solutions to help power their accelerated business or per...
We are seeing a major migration of enterprises applications to the cloud. As cloud and business use of real time applications accelerate, legacy networks are no longer able to architecturally support cloud adoption and deliver the performance and security required by highly distributed enterprises. These outdated solutions have become more costly and complicated to implement, install, manage, and maintain.SD-WAN offers unlimited capabilities for accessing the benefits of the cloud and Internet. ...
Bill Schmarzo, author of "Big Data: Understanding How Data Powers Big Business" and "Big Data MBA: Driving Business Strategies with Data Science," is responsible for setting the strategy and defining the Big Data service offerings and capabilities for EMC Global Services Big Data Practice. As the CTO for the Big Data Practice, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He's written several white papers, is an avid blogge...
DXWorldEXPO LLC, the producer of the world's most influential technology conferences and trade shows has announced the 22nd International CloudEXPO | DXWorldEXPO "Early Bird Registration" is now open. Register for Full Conference "Gold Pass" ▸ Here (Expo Hall ▸ Here)
Charles Araujo is an industry analyst, internationally recognized authority on the Digital Enterprise and author of The Quantum Age of IT: Why Everything You Know About IT is About to Change. As Principal Analyst with Intellyx, he writes, speaks and advises organizations on how to navigate through this time of disruption. He is also the founder of The Institute for Digital Transformation and a sought after keynote speaker. He has been a regular contributor to both InformationWeek and CIO Insight...
Join IBM November 1 at 21st Cloud Expo at the Santa Clara Convention Center in Santa Clara, CA, and learn how IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Cognitive analysis impacts today’s systems with unparalleled ability that were previously available only to manned, back-end operations. Thanks to cloud processing, IBM Watson can bring cognitive services and AI to intelligent, unmanned systems. Imagine a robot vacuum that becomes your personal assistant tha...
"MobiDev is a software development company and we do complex, custom software development for everybody from entrepreneurs to large enterprises," explained Alan Winters, U.S. Head of Business Development at MobiDev, in this SYS-CON.tv interview at 21st Cloud Expo, held Oct 31 – Nov 2, 2017, at the Santa Clara Convention Center in Santa Clara, CA.
I think DevOps is now a rambunctious teenager - it's starting to get a mind of its own, wanting to get its own things but it still needs some adult supervision," explained Thomas Hooker, VP of marketing at CollabNet, in this SYS-CON.tv interview at DevOps Summit at 20th Cloud Expo, held June 6-8, 2017, at the Javits Center in New York City, NY.
Recently, WebRTC has a lot of eyes from market. The use cases of WebRTC are expanding - video chat, online education, online health care etc. Not only for human-to-human communication, but also IoT use cases such as machine to human use cases can be seen recently. One of the typical use-case is remote camera monitoring. With WebRTC, people can have interoperability and flexibility for deploying monitoring service. However, the benefit of WebRTC for IoT is not only its convenience and interopera...
Cloud-enabled transformation has evolved from cost saving measure to business innovation strategy -- one that combines the cloud with cognitive capabilities to drive market disruption. Learn how you can achieve the insight and agility you need to gain a competitive advantage. Industry-acclaimed CTO and cloud expert, Shankar Kalyana presents. Only the most exceptional IBMers are appointed with the rare distinction of IBM Fellow, the highest technical honor in the company. Shankar has also receive...
It is of utmost importance for the future success of WebRTC to ensure that interoperability is operational between web browsers and any WebRTC-compliant client. To be guaranteed as operational and effective, interoperability must be tested extensively by establishing WebRTC data and media connections between different web browsers running on different devices and operating systems. In his session at WebRTC Summit at @ThingsExpo, Dr. Alex Gouaillard, CEO and Founder of CoSMo Software, presented ...
Business professionals no longer wonder if they'll migrate to the cloud; it's now a matter of when. The cloud environment has proved to be a major force in transitioning to an agile business model that enables quick decisions and fast implementation that solidify customer relationships. And when the cloud is combined with the power of cognitive computing, it drives innovation and transformation that achieves astounding competitive advantage.
WebRTC is great technology to build your own communication tools. It will be even more exciting experience it with advanced devices, such as a 360 Camera, 360 microphone, and a depth sensor camera. In his session at @ThingsExpo, Masashi Ganeko, a manager at INFOCOM Corporation, introduced two experimental projects from his team and what they learned from them. "Shotoku Tamago" uses the robot audition software HARK to track speakers in 360 video of a remote party. "Virtual Teleport" uses a multip...
Data is the fuel that drives the machine learning algorithmic engines and ultimately provides the business value. In his session at Cloud Expo, Ed Featherston, a director and senior enterprise architect at Collaborative Consulting, discussed the key considerations around quality, volume, timeliness, and pedigree that must be dealt with in order to properly fuel that engine.
IoT is rapidly becoming mainstream as more and more investments are made into the platforms and technology. As this movement continues to expand and gain momentum it creates a massive wall of noise that can be difficult to sift through. Unfortunately, this inevitably makes IoT less approachable for people to get started with and can hamper efforts to integrate this key technology into your own portfolio. There are so many connected products already in place today with many hundreds more on the h...
When shopping for a new data processing platform for IoT solutions, many development teams want to be able to test-drive options before making a choice. Yet when evaluating an IoT solution, it’s simply not feasible to do so at scale with physical devices. Building a sensor simulator is the next best choice; however, generating a realistic simulation at very high TPS with ease of configurability is a formidable challenge. When dealing with multiple application or transport protocols, you would be...
Detecting internal user threats in the Big Data eco-system is challenging and cumbersome. Many organizations monitor internal usage of the Big Data eco-system using a set of alerts. This is not a scalable process given the increase in the number of alerts with the accelerating growth in data volume and user base. Organizations are increasingly leveraging machine learning to monitor only those data elements that are sensitive and critical, autonomously establish monitoring policies, and to detect...
In his keynote at 18th Cloud Expo, Andrew Keys, Co-Founder of ConsenSys Enterprise, provided an overview of the evolution of the Internet and the Database and the future of their combination – the Blockchain. Andrew Keys is Co-Founder of ConsenSys Enterprise. He comes to ConsenSys Enterprise with capital markets, technology and entrepreneurial experience. Previously, he worked for UBS investment bank in equities analysis. Later, he was responsible for the creation and distribution of life settl...
In his session at @ThingsExpo, Dr. Robert Cohen, an economist and senior fellow at the Economic Strategy Institute, presented the findings of a series of six detailed case studies of how large corporations are implementing IoT. The session explored how IoT has improved their economic performance, had major impacts on business models and resulted in impressive ROIs. The companies covered span manufacturing and services firms. He also explored servicification, how manufacturing firms shift from se...
DevOpsSummit New York 2018, colocated with CloudEXPO | DXWorldEXPO New York 2018 will be held November 11-13, 2018, in New York City. Digital Transformation (DX) is a major focus with the introduction of DXWorldEXPO within the program. Successful transformation requires a laser focus on being data-driven and on using all the tools available that enable transformation if they plan to survive over the long term. A total of 88% of Fortune 500 companies from a generation ago are now out of bus...