Introduction to DevOps

• DevOps is a cultural transformation.
• DevOps is not a tool or a job title. It is a shared mindset.
• Embrace the DevOps Culture.
• #1 reason why DevOps fails is due to issues around organizational learning and cultural change.
• DevOps is not a tool or a job title. It is a shared mindset.

“Tools are not the solution to a cultural problem.”

– Gartner

“Team culture makes a large difference to a team’s ability to deliver software and meet or exceed their organizational goals.”

– Accelerate State of DevOps 2021

How to change a Culture

• Think Differently

  • Social Coding
  • Work in small batches
  • Minimum viable product

• Work Differently

  Team culture makes a large difference to a team’s ability to deliver software and meet or exceed their organizational goals.

• Organize Differently

• Measure Differently

  • Measure what matters
  • You get what you measure

Business Case For DevOps

Disruptive Business model:

• 52% of the Fortune 500 have disappeared since the year 2000.

• When disruption happen, businesses need to adopt according no matter what, and this adaptation should be agile and lean.

  

  Digitization + Business Model:

• Technology is the enabler of innovation, not the driver of innovation.

• The Businesses, who adapt to new tech, survive.

• The refusal to change according to the digital ages makes it susceptible to bankruptcy.

DevOps Adoption

Unlearn what you have Learned

• A different mindset
• Unlearn your current culture
• Often easier said than done

Consider this:

• fail fast and roll back quickly
• test in market instead of analyzing
• modular design which makes individual components replaceable

How are they doing this?

• What is their secret?
• They have embraced the DevOps culture.

Definition of DevOps

The term (development and operations) is an extension of agile development environments that aims to enhance the process of software delivery as a whole. — Patrick Debois, 2009

DevOps defined:

• Recognition that working in silos doesn’t work
• Development and operations engineers working together
• Following lean and agile principles
• Delivering software in a rapid and continuous manner

DevOps requires:

• A change in culture
• A new application design
• Leveraging automation
• Programmable platform

What DevOps not:

• Not simply combining development and operations
• Not a separate team
• Not a tool
• Not one size fits all
• Not just automation

Essential Characteristics of DevOps

What’s the Goal?

Agility is the goal:

• Smart experimentation
• Moving in market
• With maximum velocity and minimum risk
• Gaining quick, valuable insights

Agility: The Three pillars

1. DevOps:

• Cultural change
• Automated pipeline
• infrastructure as code
• immutable infrastructure

1. Microservices:

• Loose coupling/binding
• RESTful APIs
• Designed to resist failures
• Test by breaking/fail fast

1. Containers

• portability
• Developer centric
• Ecosystem enabler
• Fast startup

The Perfect Combination/Storm

• DevOps for speed and agility
• Microservices for small deployments
• Containers for ephemeral runtimes

Learning how to work differently

“DevOps starts with learning how to work differently. It embraces cross-functional teams with openness, transparency, and respect as pillars.” — Tony Stafford, Shadow Soft.

Application Evolution

DevOps has three dimensions:

Responsibility, transparency, feedback:

“Culture is the #1 success factor in DevOps. Building a culture of shared responsibility, transparency, and faster feedback is the foundation of every high-performing DevOps team.” — Atlassian

Culture, culture, culture:

While tools and methods are important; … it’s the culture that has the biggest impact.

How to change a Culture?

-Change thinking patterns of people

• working methodology as well as environment
• Organizational change.
• Change of the way people are measured.

Leading Up to DevOps

• Architects worked for months designing the system.
• Development worked for months on features.
• Testing opened defects and sent the code back to development.
• At some point, the code is released to operations.
• The operations team took forever to deploy.

Traditional Waterfall Method

Problems with Waterfall Approach

• No room for change
• No idea if it works till end
• Each step ends when the next begins
• Mistakes found in the later stages are more expensive to fix
• Long time between software releases
• Team work separately, unaware of their impact on each other
• Least familiar people with the code are deploying it into production

XP, Agile, and Beyond

Extreme Programming (XP)

• In 1996, Kent Beck introduced Extreme Programming
• Based on an interactive approach to software development
• Intended to improve software quality, responsiveness to changing customer requirements
• One of the first Agile methods

The Agile Manifesto

We have come to value:

• Individuals and interactions over processes and tools
• Working Software over comprehensive docs
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Agile Development

Agile alone is not good enough:

2 Speed IT:

• This is how Shadow IT started, as Ops team wasn’t meeting their needs.

Shadow IT

• Resources the business doesn’t know about
• People go around IT
• We need the solution to this problem, and DevOps is the answer

Brief History of DevOps

2007 Patrick Debois:

He recognized Dev and Ops worked ineffectively and not together.

2008 Agile Conference:

Andrew Clay Shafer – Agile Conference 2008 BoF (Birds of a Feather) meeting “Agile Infrastructure”

2009 Velocity 10+ deploys per day:

John Allspaw – Velocity 2009 “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr”

DevOpsDays – Patrick Debois started the first DevOpsDays conference Ghent, Belgium, October 2009

2010 Continuous Delivery:

Continuous Delivery – by Jez Humble and David Farley

2013 The Phoenix Project:

The Phoenix Project – by Gene Kim, Kevin Behr and George Spafford

2015 State of DevOps Report:

State of DevOps Reports – from DevOps Research and Assessment (DORA), founded by Dr. Nicole Forsgren, Gene Kim, and Jez Humble

2016 The DevOps Handbook:

The DevOps Handbook – by Gene Kim, Jez Humble, Patrick Debois, and John Willis

2019 10 Years of DevOpsDays:

DevOpsDays – 40 events in 21 countries are scheduled for 2019 (10 years later) Patrick Debois (lead 2009-15) Bridget Kromhout (lead 2015-2020)

Why is the history significant?

It reminds us that DevOps is:

• From the practitioners, by practitioners
• Not a product, specification, or job title
• An experience-based movement
• Decentralized and open to all

Thinking DevOps

Social Coding Principles

What is social coding?

• Open source practice

• All repos are public

• Everyone is encouraged to contribute

• Anarchy is controlled via Pull Requests

  Code reuse dilemma:

• Code has 80% of what you need, but 20% is missing

• How do you add 20% missing features?

  • Make a feature request and depend on another team?
  • Rebuild 100% of what you need (no dependencies)

Social Coding Solution:

• Discuss with the repo owner

• Agree to develop it

• Open an Issue and assign it to yourself

• Fork the code and make your changes

• Issue a Pull Request to review and merge back

  Pair Programming:

• Two programmers on one workstation

• The driver is typing

• The navigator is reviewing

• Every 20 minutes they switch roles

  Pair programming benefits:

• Higher code quality

• Defects found earlier

• Lower maintenance costs

• Skills transfer

• Two set of eyes on every line of codebase

Git Repository Guidelines

• Create a separate Git repository for every component

• Create a new branch for every Issue

• Use PRs to merge to master

• Every PR is an opportunity for a code review

  Git feature branch workflow:

Working in Small Batches

• Concept from Lean Manufacturing

• Faster feedback

• Supports experimentation

• Minimize waste

• Deliver faster

  Small batch example:

  You need to mail 1000 brochures:

• Step 1: Fold brochures

• Step 2: Insert brochures into envelopes

• Step 3: Seal the envelopes

• Step 4: Stamp the envelopes with postage

Batch of 50 brochures:

Single Piece Flow:

Measuring the size of batches

• Feature size supports frequent releases
• Features should be completed in a sprint
• Features are a step toward a goal, so keep them small

Minimum Viable Product (MVP)

• MVP is not “Phase 1” of a project
• MVP is an experiment to test your value hypothesis and learn
• MVP is focused on learning, not delivery
• At the end of each MVP, you decide whether to pivot or persevere

Minimum Viable Product Example:

Gaining an understanding

• MVP is a tool for learning
• The experiment may fail and that’s okay
• Failure leads to understanding
• What did you learn from it?
• What will you do differently?

Test Driven Development (TDD)

The importance of testing:

“If it’s worth building, it’s worth testing. If it’s not worth testing, why are you wasting your time working on it?” — Scott Ambler

What is test driven development?

• Test cases drive the design

• You write the tests first then you write the code to make the test pass

• This keeps you focused on the purpose of the code

• Code is of no use if your client can’t call it

  Why devs don’t test:

• I already know my code works

• I don’t write broken code

• I have no time

Basic TDD workflow

Why is TDD important for DevOps?

• It saves time when developing
• You can code faster and with more confidence
• It ensures the code is working as expected
• It ensures that future changes don’t break your code
• In order to create a DevOps CI/CD pipeline, all testing must be automated

Behavior Driven Development (BDD)

• Describes the behavior of the system from the outside
• Great for integration testing
• Uses a syntax both devs and stakeholders can easily understand

BDD vs. TDD:

• BDD ensures that you’re building the “right thing”
• TDD ensures that you are building the “thing right”

BDD workflow

• Explore the problem domain and describe the behavior

• Document the behavior using Gherkin syntax

• Use BDD tools to run those scenarios

• One document that’s both the specification and the tests

  Gherkin:

• An easy-to-read natural language syntax

• Given … When… Then…

• Understandable by everyone

  Gherkin Syntax:

• Given (some context)

• When (some event happens)

• Then (some testable outcome)

• And (more context, events, or outcomes)

Retail BDD example

Gherkin for acceptance criteria:

• Add acceptance criteria to every user story
• Use Gherkin to do that
• Indisputable definition of “done”

Expected benefits of BDD

• Improves communication
• More precise guidance
• Provides a common syntax
• Self-documenting
• Higher code quality
• Acceptance criteria for user stories

Cloud Native Microservices

Think differently about application design:

Think cloud native:

• The Twelve-Factor App
• A collection of stateless microservices
• Each service maintains its own database
• Resilience through horizontal scaling
• Failing instances are killed and respawned
• Continuous Delivery of services

Think microservices:

“The microservices architectural style is an approach to developing a single application as a suit of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery.” — Martin Fowler and James Lewis

Monolith vs. Microservices

Designing for Failure

Failure happens:

• Embrace failures – they will happen!
• How to avoid –> How to identify and what to do about it
• Operational concern –> developer concern
• Plan to be throttled
• Plan to retry (with exponential back off)
• Degrade gracefully
• Cache when appropriate

Retry pattern:

Circuit Breaker pattern:

Bulkhead pattern:

Chaos engineering:

• Also known as monkey testing
• You deliberately kill services
• Netflix created The Simian Army tools
• You cannot know how something will respond to a failure until it actually fails

Working DevOps

Taylorism and Working in Silos

Working DevOps:

• Culture of teaming and collaboration

• Agile development as a shared discipline

• Automate relentlessly

• Push smaller releases faster

  Taylorism:

• Adoption of command and control management

• Organizations divided into functional silos

• Decision-making is separated from work

Impact of Taylorism on IT:

Software development is bespoke:

• Software development is NOT like assembling automobiles

• Most of the parts don’t exist, yet

• Software development is craft work

• Taylorism is not appropriate for craft work

  Abandon Command and Control:

• Command and control is not Agile

• Stop working in silos

• Let your people amaze you

Software Engineering vs. Civil Engineering

Software engineering is organic:

• Software stack is constantly updated

• New features are being added

• System behavior changes over time

• Yet we treat software engineering like a civil engineering project

  The project model is flawed:

• The project model doesn’t work for software development

• Treat software development like product development

• Encourage ownership and understanding

• Software engineering is not civil engineering

• Maintain stable, lasting teams

Required DevOps Behaviors

Diametrically opposed views:

• Enterprises see “new” as complex and time-consuming
• DevOps delivers a continual series of small changes
• These cannot survive traditional overheads

A clash of work culture:

The no-win scenario:

• Development wants innovation
• Operations wants stability

Operations view of development:

• Development teams throw dead cats over the wall

• Manually implemented changes

• Lack of back-out plans

• Lack of testing

• Environments that don’t look like production

  Development view of operations:

• All-or-nothing changes

• Change windows in the dead of night

• Implemented by people furthest away from the application

• Ops just cuts and pastes from “runbooks”

  No-win scenario:

• If the website works, the developers get the praise!

• If the website is down, operations gets the blame!

  Required DevOps behaviors:

Infrastructure as Code

• Described an executable textual format

• Configure using that description

• Configuration Management Systems to make this possible (Ansible, puppet etc.)

• Never perform configurations manually

• Use version control

  Ephemeral immutable infrastructure:

• Server drift is a major source of failure

• Servers are cattle not pets

• Infrastructure is transient

• Build through parallel infrastructure

  Immutable delivery via containers:

• Applications are packaged in containers

• Same container that runs in production can be run locally

• Dependencies are contained

• No variance limits side effects

• Rolling updates with immediate roll-back

  Immutable way of working:

• You never make changes to a running container

• You make changes to the image

• Then redeploy a new container

• Keep images up-to-date

Continuous Integration (CI)

CI vs. CD:

• CI/CD is not one thing
• Continuous Integration (CI):
  • Continuously building, testing, and merging to master
• Continuous Delivery (CD):

  • Continuously deploying to a production-like environment

    Traditional Development:

• Devs work in long-lived development branches
• Branches are periodically merged into a release
• Builds are run periodically
• Devs continue to add to the development branch

Continuous Integration

• Devs integrate code often
• Devs work in short-lived feature branches
• Each check-in is verified by an automated build

Changes are kept small:

• Working in small batches

• Committing regularly

• Using pull requests

• Committing all changes daily

  CI automation:

• Build and test every pull request

• Use CI tools that monitor version control

• Test should run after each build

• Never merge a PR with failing tests

  Benefits of CI:

• Faster reaction times to changes

• Reduced code integration risk

• Higher code quality

• The code in version control works

• Master branch is always deployable

Continuous Delivery

“Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.” — Martin Fowler

Release to production at any time:

• The master branch should always be ready to deploy
• You need a way to know if something will “break the build”
• Deliver every change to a production-like environment

CI/CD pipeline:

• Automated gates that create a pipeline of checks:
  • Unit testing
  • Code quality checks
  • Integration testing
  • Security testing
  • Vulnerability scanning
  • Package signing

A CI/CD pipeline needs:

• A code repository
• A build server
• An integration server
• An artifact repository
• Automatic configuration and deployment

Continuous integration and delivery:

Five key principles:

1. Build quality in
2. Work in small batches
3. Computers perform repetitive tasks, people solve problems
4. Relentlessly pursue continuous improvement
5. Everyone is responsible

CI/CD + Continuous deployment:

How DevOps manages risk:

• Deployment is king
• Deployment is decoupled from activation
• Deployment is not “one size fits all”

Organizing for DevOps

Organizational Impact of DevOps

How does organization affect DevOps?

Is the culture of your organization agile?

• Small teams
• Dedicated teams
• Cross-functional teams
• Self-organizing teams

Conway’s Law:

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.” — Melvin Conway, Datamation, 1968

Traditional organization around technology:

Organized around business domains:

Align teams with the business:

• Each team has its own mission aligned with the business
• Teams have end-to-end responsibility for what they build
• Teams should have a long-term mission, usually around a single business domain

There is No DevOps Team

DevOps is often misunderstood:

• Dev and Ops are not separate things
• You aren’t a DevOps if you’re not a Dev, and other way is also true

Perspectives on DevOps:

DevOps is not a team:

“The DevOps movement addresses the dysfunction that results from organizations composed of functional silos. Thus, creating another functional silo that sits between Dev and Ops is clearly a poor (and ironic) way to try, and solve these problems.” — Jez Humble, The DevOps Handbook

Working in silos doesn’t work:

A DevOps team means we’re DevOps, right?

DevOps is not a job title:

• A culture transformation on an organizational scale
• Development and operations engineers working together
• Following lean and agile principles
• Cross-functional teams with openness, transparency, and trust as pillars

Everyone is Responsible for Success

Bad behavior:

“Bad behavior arises when you abstract people from the consequences of their actions.” — Jez Humble, Continuous Delivery

Functional silos breed bad behavior:

Actions have consequences:

• Make people aware of the consequences of their actions
• Make people responsible for the consequences of their actions

DevOps organizational objective:

Shared consciousness

…with

distributed (local) control

Measuring DevOps

Rewarding for “A” while hoping for “B”

On the folly of rewarding for A, while hoping for B

“Whether dealing with monkeys, rats, or human beings, it is hardly controversial to state that most organisms seek information concerning what activities are rewarded, and then seek to do (or at least pretend to do) those things, often to the virtual exclusion of activities not rewarded. The extent to which this occurs, of course, will depend on the perceived attractiveness of the rewards offered, but neither operant nor expectancy theorists would quarrel with the essence of this notion.” — Steven Kerr, The Ohio State University

Measure what matters:

Social metrics:

• Who is leveraging the code you are building?
• Whose code are you leveraging?

DevOps metrics:

• A baseline provides a concrete number for comparison as you implement your DevOps changes
• Metric goals allow you to reason about these numbers and judge the success of your progress

DevOps changes the objective:

• Old school is focused on mean time to failure (MTTF)
• DevOps is focused on mean time to recovery (MTTR)

Vanity metrics vs. actionable metrics

Vanity metrics:

• We had 10,000 daily hits to our website!

• Now what? (What does a hit represent?)

• What actions drove those visitors to you?

• Which actions to take next?

  Actionable metrics:

Actionable metric examples:

• Reduce time to market
• Increase overall availability
• Reduce time to deploy
• Defects detected before production
• More efficient use of infrastructure
• Quicker performance feedback

Top four actionable metrics:

1. Mean lead time
2. Release frequency
3. Change failure rate
4. Mean time to recovery (MTTR)

How to Measure Your Culture

Culture measurements:

You can rate statements developed by Dr. Nicole Forsgren to measure your team’s culture, including statements about information, failures, collaboration, and new ideas.

Strongly agree or disagree?

• On my team, information is actively sought
• On my team, failures are learning opportunities and messengers of them are not punished
• On my team, responsibilities are shared
• On my team, cross-functional collaboration is encouraged and rewarded
• On my team, failure causes inquiry
• On my team, new ideas are welcomed

Comparison of DevOps to Site Reliability Engineering

What is SRE?

“…what happens when a software engineer is tasked with what used to be called operations.” —Ben Treynor Sloss

Goal: Automate yourself out of a job.

Tenets of SRE:

• Hire only software engineers

• Site reliability engineers work on reducing toil through automation

• SRE teams are separate from development teams

• Stability is controlled through error budgets

• Developers rotate through operations

  Team differences:

• SRE maintains separate development and operations silos with one staffing pool

• DevOps breaks down the silos into one team with one business objective

Maintaining stability:

Commonality:

• Both seek to make both Dev and Ops work visible to each other

• Both require a blameless culture

• The objective of both is to deploy software faster with stability

  DevOps + SRE:

• SRE maintains the infrastructure

• DevOps uses infrastructure to maintain their applications