Welcome!

Government Cloud Authors: Elizabeth White, Pat Romanski, Dana Gardner, Liz McMillan, Gopala Krishna Behara

Related Topics: Microservices Expo, Java IoT, Containers Expo Blog, Agile Computing, @CloudExpo

Microservices Expo: Article

Scrum Buts, “Meta” Thinking, and Agile Architecture

How would we ever expect to architect an enterprise if we didn’t take an Agile Architecture approach that dealt with change?

ZapThink has long bemoaned the Agile Manifesto paradox: that the point to the Manifesto was to be less dogmatic about software development, but today people are overly dogmatic about Agile, defeating its entire purpose. In fact, this paradox has found its way into what is perhaps the most popular of the Agile methodologies: Scrum. Not to worry, all you Scrum aficionados out there; we’re not going to teach you how to do Scrum in this ZapFlash. Instead, this article is more about how to think about a broad set of problems in a particular way, starting with Scrum Buts.

Enter the Scrum But
The notion of a Scrum But arose when it became clear that thousands of organizations were attempting to follow Scrum for their software development projects, but many of them were having problems with one or another of its tenets. As a result, they would say things like:

“We use Scrum, but Retrospectives are a waste of time, so we don't do them.”

Or:

“We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long.”

Where Retrospectives and Sprints are well-known Scrum best practices. Note that both of these statements follow the same “We use Scrum, but X so Y” pattern, hence the term Scrum But.

The question with Scrum Buts, of course, is what to do with them—or more specifically, how to think about them. There are two schools of thought:

  1. Scrum Buts are simply excuses not to do Scrum properly, and if you’re not doing it properly, then you’re not really doing it at all.
  2. Resolving Scrum Buts when they come up in order to achieve the desired result (software that meets the stakeholders’ needs) is actually a part of the Scrum methodology. As a result, Scrum Buts are expected and even welcomed.

From ZapThink’s perspective, option #1 is an example of taking a dogmatic approach, which is inherently non-Agile. Option #2 basically says that you can modify the rules if necessary (one of the four principles of the Agile Manifesto), even the rules of Scrum itself.

In other words, principle #2 is self-referential—which may be more Agile to be sure, but people have problems with self-reference. It brings up uncomfortable visions of the liar’s paradox: “everything I say is a lie,” or in more modern terms, the first rule of Fight Club (“do not talk about Fight Club.”) How can we make sense of the world or anything in it if we have to deal with self-reference paradoxes? If a rule of Scrum is “you can change the rules of Scrum,” then couldn’t anything and everything be Scrum? What use is that?

“Meta” Thinking
Fortunately, such problems of self-reference have a straightforward, if subtle solution. Instead of thinking of such statements as referring to themselves, think of them as actually two separate but related statements, where one relates to the other. We call this “meta” thinking.

In the case of Scrum Buts, we have the Scrum methodology and the Scrum meta-methodology. A meta-methodology is a methodology for creating methodologies. Remember, the Scrum methodology is a methodology for creating software. The Scrum meta-methodology is a methodology for creating or improving the Scrum methodology. So when someone says:

“When you say, ‘We use Scrum, but we can't build a piece of functionality in a month, so our Sprints are 6 weeks long,’ my recommendation is to try three 30-day Sprints before extending the length of the Sprint.”

That entire statement is a Scrum meta-methodology statement. Furthermore, without such statements, your methodology wouldn’t be Agile. The obvious conclusion is that all Agile methodologies are actually meta-methodologies. Otherwise they wouldn’t be Agile!

Avoiding the Hall of Mirrors
I’m sure one of you wise guys out there is thinking, what about methodologies for creating meta-methodologies? We’d call meta-meta-methodologies, of course. And what about methodologies for creating those, ad infinitum? We call this the hall of mirrors problem, because you only need to have two mirrors facing each other to get the infinite tunnel effect.

Simple answer: we don’t need a methodology for creating meta-methodologies. Instead, an informal approach will do. In general, we only want to go to the meta-meta step if there’s a bona fide reason to do so, as with Model-Driven Architecture, when they talk about meta-meta-models. But even the brainiacs at the OMG don’t spend much time thinking about meta-meta-meta-models—at least, I hope not!

We often get this question in our Licensed ZapThink Architect SOA course when we discuss meta-policies. A meta-policy, of course, is a policy that applies to other policies. Because the automation of policy-related processes is a core enabler of SOA governance, polices for how an organization performs such automation are quite important—and such policies are meta-policies. An example of a useful meta-policy: “we’ll configure our registry/repository’s design-time workflow to conform to our existing Service lifecycle.” Such a configuration means configuring the policies in the tool as a way of automating the corresponding workflow, and thus that statement is a meta-policy.

In this example, the hall of mirrors question focuses more on automation then on the policies themselves. Does it make sense to automate meta-policy related processes? The answer is basically no. It’s best to create, communicate, and enforce meta-policies manually, through human communication activities. Yes, you could think of that last statement as a meta-meta-policy, but thinking about it that way doesn’t really help you, so don’t bother.

More Meta-Thinking
Meta-models are well known now, and other than those plus meta-methodologies and meta-policies, where does meta thinking come up? Way back in 2005, ZapThink discussed the notion of meta-processes. A meta-process, of course, is a process for dealing with processes, for example, the process of composing Services to implement a business process. While we pointed out in that article that such meta-processes should be handled manually, we also predicted increasing automation of Service composition processes—a capability that is still well in the future (although if you have any examples, please let us know.)

Another meta that is central to ZapThink’s thinking is the meta-requirement. In particular, we’ve been talking about the meta-requirement of business agility as a fundamental driver of SOA, and by extension, Agile Architecture in general. When the business asks for an agile system, they are asking for a system that can respond to changing requirements—which is what makes such agility a meta-requirement. And remember, it’s because the business agility meta-requirement is an emergent property of the enterprise that requires us to characterize the enterprise itself as a complex system.

Finally, at the risk of belaboring the point, let’s talk about meta-architecture: what does it mean to architect an architecture? Yes, ZapThink has been spending a lot of our brain cycles on meta-architecture as well. We’ve been putting our stamp on how best to do SOA for several years now. Our students may be architecting their organizations and their component systems, but ZapThink has been architecting SOA itself. And now that we can stick a fork in that, it’s time to work on architecting Cloud architecture and more broadly, Agile Architecture.

The ZapThink Take
Meta thinking about something means more than just thinking about the thing, it means thinking about how the thing might change. Meta-processes are processes for changing processes. The reason we need meta-policies is because policies might change—otherwise we’d just hard-code them into our software and be done with them. The business agility meta-requirement is fundamentally a requirement for change.

Complex systems are themselves self-adapting. They are always in a state of change. How would we ever expect to architect a complex system like an enterprise if we didn’t take an Agile Architecture approach that worked at the meta level to deal with change? Perhaps the lack of a complex systems approach to traditional Enterprise Architecture—architectural approaches that presume a final to-be state—is why all such approaches are inherently flawed.

Our Scrum But example is an illuminating illustration of what we mean by an Agile Architectural approach. You could look at a list of Scrum Buts and say, this team is doomed to failure. They’ve taken the good bits to Scrum and stripped those out, and now they’re screwed. Alternatively, you could look at the same list and say, with a few bits of advice about how to deal with the Scrum Buts (in other words, the right meta-methodology), this team might be successful after all.

This admittedly oversimplified scenario has two outcomes: team crashes and burns, or team is successful in spite of their Scrum Buts. In complex systems theory, these outcomes are called attractors: given a set of circumstances, the end result will usually follow one of a set of patterns, in spite of the fact that different people are involved, each with their own skills and preferences. The system is subject to perturbations (the Scrum Buts, in our example), as well as constraints (the advice that makes up the meta-methodology), and the various identities of the people.

Without the appropriate advice, the attractor that is most likely to describe the outcome is the failure attractor. Clearly, the more Scrum Buts you have, and the more serious they are, the more likely your project will fail (although failure is still not certain). But with the proper meta-methodology, you can steer the project toward the success attractor, in spite of all the Scrum Buts and the various people on the team, who may be disgruntled, incompetent, overworked, or whatever.

Note that the success attractor is not a final state in a traditional sense. Rather, it allows for the fact that perturbations, constraints, and identities are always subject to change. Generalize our Scrum meta-methodology example to the level of the enterprise, and you can get a sense of what we mean by Agile Architecture. Can we design the complex system of the enterprise, a system consisting of human and technology subsystems, to move toward desirable attractors through the introduction of appropriate meta-policies, meta-processes, and meta-methodologies? That’s the million dollar meta-architecture question.

More Stories By Jason Bloomberg

Jason Bloomberg is a leading IT industry analyst, Forbes contributor, keynote speaker, and globally recognized expert on multiple disruptive trends in enterprise technology and digital transformation. He is ranked #5 on Onalytica’s list of top Digital Transformation influencers for 2018 and #15 on Jax’s list of top DevOps influencers for 2017, the only person to appear on both lists.

As founder and president of Agile Digital Transformation analyst firm Intellyx, he advises, writes, and speaks on a diverse set of topics, including digital transformation, artificial intelligence, cloud computing, devops, big data/analytics, cybersecurity, blockchain/bitcoin/cryptocurrency, no-code/low-code platforms and tools, organizational transformation, internet of things, enterprise architecture, SD-WAN/SDX, mainframes, hybrid IT, and legacy transformation, among other topics.

Mr. Bloomberg’s articles in Forbes are often viewed by more than 100,000 readers. During his career, he has published over 1,200 articles (over 200 for Forbes alone), spoken at over 400 conferences and webinars, and he has been quoted in the press and blogosphere over 2,000 times.

Mr. Bloomberg is the author or coauthor of four books: The Agile Architecture Revolution (Wiley, 2013), Service Orient or Be Doomed! How Service Orientation Will Change Your Business (Wiley, 2006), XML and Web Services Unleashed (SAMS Publishing, 2002), and Web Page Scripting Techniques (Hayden Books, 1996). His next book, Agile Digital Transformation, is due within the next year.

At SOA-focused industry analyst firm ZapThink from 2001 to 2013, Mr. Bloomberg created and delivered the Licensed ZapThink Architect (LZA) Service-Oriented Architecture (SOA) course and associated credential, certifying over 1,700 professionals worldwide. He is one of the original Managing Partners of ZapThink LLC, which was acquired by Dovel Technologies in 2011.

Prior to ZapThink, Mr. Bloomberg built a diverse background in eBusiness technology management and industry analysis, including serving as a senior analyst in IDC’s eBusiness Advisory group, as well as holding eBusiness management positions at USWeb/CKS (later marchFIRST) and WaveBend Solutions (now Hitachi Consulting), and several software and web development positions.

IoT & Smart Cities Stories
The challenges of aggregating data from consumer-oriented devices, such as wearable technologies and smart thermostats, are fairly well-understood. However, there are a new set of challenges for IoT devices that generate megabytes or gigabytes of data per second. Certainly, the infrastructure will have to change, as those volumes of data will likely overwhelm the available bandwidth for aggregating the data into a central repository. Ochandarena discusses a whole new way to think about your next...
DXWorldEXPO LLC announced today that Big Data Federation to Exhibit at the 22nd International CloudEXPO, colocated with DevOpsSUMMIT and DXWorldEXPO, November 12-13, 2018 in New York City. Big Data Federation, Inc. develops and applies artificial intelligence to predict financial and economic events that matter. The company uncovers patterns and precise drivers of performance and outcomes with the aid of machine-learning algorithms, big data, and fundamental analysis. Their products are deployed...
Dynatrace is an application performance management software company with products for the information technology departments and digital business owners of medium and large businesses. Building the Future of Monitoring with Artificial Intelligence. Today we can collect lots and lots of performance data. We build beautiful dashboards and even have fancy query languages to access and transform the data. Still performance data is a secret language only a couple of people understand. The more busine...
All in Mobile is a place where we continually maximize their impact by fostering understanding, empathy, insights, creativity and joy. They believe that a truly useful and desirable mobile app doesn't need the brightest idea or the most advanced technology. A great product begins with understanding people. It's easy to think that customers will love your app, but can you justify it? They make sure your final app is something that users truly want and need. The only way to do this is by ...
CloudEXPO | DevOpsSUMMIT | DXWorldEXPO are the world's most influential, independent events where Cloud Computing was coined and where technology buyers and vendors meet to experience and discuss the big picture of Digital Transformation and all of the strategies, tactics, and tools they need to realize their goals. Sponsors of DXWorldEXPO | CloudEXPO benefit from unmatched branding, profile building and lead generation opportunities.
Digital Transformation and Disruption, Amazon Style - What You Can Learn. Chris Kocher is a co-founder of Grey Heron, a management and strategic marketing consulting firm. He has 25+ years in both strategic and hands-on operating experience helping executives and investors build revenues and shareholder value. He has consulted with over 130 companies on innovating with new business models, product strategies and monetization. Chris has held management positions at HP and Symantec in addition to ...
Cell networks have the advantage of long-range communications, reaching an estimated 90% of the world. But cell networks such as 2G, 3G and LTE consume lots of power and were designed for connecting people. They are not optimized for low- or battery-powered devices or for IoT applications with infrequently transmitted data. Cell IoT modules that support narrow-band IoT and 4G cell networks will enable cell connectivity, device management, and app enablement for low-power wide-area network IoT. B...
The hierarchical architecture that distributes "compute" within the network specially at the edge can enable new services by harnessing emerging technologies. But Edge-Compute comes at increased cost that needs to be managed and potentially augmented by creative architecture solutions as there will always a catching-up with the capacity demands. Processing power in smartphones has enhanced YoY and there is increasingly spare compute capacity that can be potentially pooled. Uber has successfully ...
SYS-CON Events announced today that CrowdReviews.com has been named “Media Sponsor” of SYS-CON's 22nd International Cloud Expo, which will take place on June 5–7, 2018, at the Javits Center in New York City, NY. CrowdReviews.com is a transparent online platform for determining which products and services are the best based on the opinion of the crowd. The crowd consists of Internet users that have experienced products and services first-hand and have an interest in letting other potential buye...
When talking IoT we often focus on the devices, the sensors, the hardware itself. The new smart appliances, the new smart or self-driving cars (which are amalgamations of many ‘things'). When we are looking at the world of IoT, we should take a step back, look at the big picture. What value are these devices providing. IoT is not about the devices, its about the data consumed and generated. The devices are tools, mechanisms, conduits. This paper discusses the considerations when dealing with the...