Introduction to DevOps

• DevOps is a cultural transformation.
• DevOps is not a tool or a job title. It is a shared mindset.
• Embrace the DevOps Culture.
• #1 reason why DevOps fails is due to issues around organizational learning and cultural change.
• DevOps is not a tool or a job title. It is a shared mindset.

“Tools are not the solution to a cultural problem.”

– Gartner

“Team culture makes a large difference to a team’s ability to deliver software and meet or exceed their organizational goals.”

– Accelerate State of DevOps 2021

How to change a Culture

• Think Differently

  • Social Coding
  • Work in small batches
  • Minimum viable product

• Work Differently

  Team culture makes a large difference to a team’s ability to deliver software and meet or exceed their organizational goals.

• Organize Differently

• Measure Differently

  • Measure what matters
  • You get what you measure

Business Case For DevOps

Disruptive Business model:

• 52% of the Fortune 500 have disappeared since the year 2000.

• When disruption happen, businesses need to adopt according no matter what, and this adaptation should be agile and lean.

  

  Digitization + Business Model:

• Technology is the enabler of innovation, not the driver of innovation.

• The Businesses, who adapt to new tech, survive.

• The refusal to change according to the digital ages makes it susceptible to bankruptcy.

DevOps Adoption

Unlearn what you have Learned

• A different mindset
• Unlearn your current culture
• Often easier said than done

Consider this:

• fail fast and roll back quickly
• test in market instead of analyzing
• modular design which makes individual components replaceable

How are they doing this?

• What is their secret?
• They have embraced the DevOps culture.

Definition of DevOps

The term (development and operations) is an extension of agile development environments that aims to enhance the process of software delivery as a whole. — Patrick Debois, 2009

DevOps defined:

• Recognition that working in silos doesn’t work
• Development and operations engineers working together
• Following lean and agile principles
• Delivering software in a rapid and continuous manner

DevOps requires:

• A change in culture
• A new application design
• Leveraging automation
• Programmable platform

What DevOps not:

• Not simply combining development and operations
• Not a separate team
• Not a tool
• Not one size fits all
• Not just automation

Essential Characteristics of DevOps

What’s the Goal?

Agility is the goal:

• Smart experimentation
• Moving in market
• With maximum velocity and minimum risk
• Gaining quick, valuable insights

Agility: The Three pillars

1. DevOps:

• Cultural change
• Automated pipeline
• infrastructure as code
• immutable infrastructure

1. Microservices:

• Loose coupling/binding
• RESTful APIs
• Designed to resist failures
• Test by breaking/fail fast

1. Containers

• portability
• Developer centric
• Ecosystem enabler
• Fast startup

The Perfect Combination/Storm

• DevOps for speed and agility
• Microservices for small deployments
• Containers for ephemeral runtimes

Learning how to work differently

“DevOps starts with learning how to work differently. It embraces cross-functional teams with openness, transparency, and respect as pillars.” — Tony Stafford, Shadow Soft.

Application Evolution

DevOps has three dimensions:

Responsibility, transparency, feedback:

“Culture is the #1 success factor in DevOps. Building a culture of shared responsibility, transparency, and faster feedback is the foundation of every high-performing DevOps team.” — Atlassian

Culture, culture, culture:

While tools and methods are important; … it’s the culture that has the biggest impact.

How to change a Culture?

-Change thinking patterns of people

• working methodology as well as environment
• Organizational change.
• Change of the way people are measured.

Leading Up to DevOps

• Architects worked for months designing the system.
• Development worked for months on features.
• Testing opened defects and sent the code back to development.
• At some point, the code is released to operations.
• The operations team took forever to deploy.

Traditional Waterfall Method

Problems with Waterfall Approach

• No room for change
• No idea if it works till end
• Each step ends when the next begins
• Mistakes found in the later stages are more expensive to fix
• Long time between software releases
• Team work separately, unaware of their impact on each other
• Least familiar people with the code are deploying it into production

XP, Agile, and Beyond

Extreme Programming (XP)

• In 1996, Kent Beck introduced Extreme Programming
• Based on an interactive approach to software development
• Intended to improve software quality, responsiveness to changing customer requirements
• One of the first Agile methods

The Agile Manifesto

We have come to value:

• Individuals and interactions over processes and tools
• Working Software over comprehensive docs
• Customer collaboration over contract negotiation
• Responding to change over following a plan

That is, while there is value in the items on the right, we value the items on the left more.

Agile Development

Agile alone is not good enough:

2 Speed IT:

• This is how Shadow IT started, as Ops team wasn’t meeting their needs.

Shadow IT

• Resources the business doesn’t know about
• People go around IT
• We need the solution to this problem, and DevOps is the answer

Brief History of DevOps

2007 Patrick Debois:

He recognized Dev and Ops worked ineffectively and not together.

2008 Agile Conference:

Andrew Clay Shafer – Agile Conference 2008 BoF (Birds of a Feather) meeting “Agile Infrastructure”

2009 Velocity 10+ deploys per day:

John Allspaw – Velocity 2009 “10+ Deploys Per Day: Dev and Ops Cooperation at Flickr”

DevOpsDays – Patrick Debois started the first DevOpsDays conference Ghent, Belgium, October 2009

2010 Continuous Delivery:

Continuous Delivery – by Jez Humble and David Farley

2013 The Phoenix Project:

The Phoenix Project – by Gene Kim, Kevin Behr and George Spafford

2015 State of DevOps Report:

State of DevOps Reports – from DevOps Research and Assessment (DORA), founded by Dr. Nicole Forsgren, Gene Kim, and Jez Humble

2016 The DevOps Handbook:

The DevOps Handbook – by Gene Kim, Jez Humble, Patrick Debois, and John Willis

2019 10 Years of DevOpsDays:

DevOpsDays – 40 events in 21 countries are scheduled for 2019 (10 years later) Patrick Debois (lead 2009-15) Bridget Kromhout (lead 2015-2020)

Why is the history significant?

It reminds us that DevOps is:

• From the practitioners, by practitioners
• Not a product, specification, or job title
• An experience-based movement
• Decentralized and open to all

Thinking DevOps

Social Coding Principles

What is social coding?

• Open source practice

• All repos are public

• Everyone is encouraged to contribute

• Anarchy is controlled via Pull Requests

  Code reuse dilemma:

• Code has 80% of what you need, but 20% is missing

• How do you add 20% missing features?

  • Make a feature request and depend on another team?
  • Rebuild 100% of what you need (no dependencies)

Social Coding Solution:

• Discuss with the repo owner

• Agree to develop it

• Open an Issue and assign it to yourself

• Fork the code and make your changes

• Issue a Pull Request to review and merge back

  Pair Programming:

• Two programmers on one workstation

• The driver is typing

• The navigator is reviewing

• Every 20 minutes they switch roles

  Pair programming benefits:

• Higher code quality

• Defects found earlier

• Lower maintenance costs

• Skills transfer

• Two set of eyes on every line of codebase

Git Repository Guidelines

• Create a separate Git repository for every component

• Create a new branch for every Issue

• Use PRs to merge to master

• Every PR is an opportunity for a code review

  Git feature branch workflow:

Working in Small Batches

• Concept from Lean Manufacturing

• Faster feedback

• Supports experimentation

• Minimize waste

• Deliver faster

  Small batch example:

  You need to mail 1000 brochures:

• Step 1: Fold brochures

• Step 2: Insert brochures into envelopes

• Step 3: Seal the envelopes

• Step 4: Stamp the envelopes with postage

Batch of 50 brochures:

Single Piece Flow:

Measuring the size of batches

• Feature size supports frequent releases
• Features should be completed in a sprint
• Features are a step toward a goal, so keep them small

Minimum Viable Product (MVP)

• MVP is not “Phase 1” of a project
• MVP is an experiment to test your value hypothesis and learn
• MVP is focused on learning, not delivery
• At the end of each MVP, you decide whether to pivot or persevere

Minimum Viable Product Example:

Gaining an understanding

• MVP is a tool for learning
• The experiment may fail and that’s okay
• Failure leads to understanding
• What did you learn from it?
• What will you do differently?

Test Driven Development (TDD)

The importance of testing:

“If it’s worth building, it’s worth testing. If it’s not worth testing, why are you wasting your time working on it?” — Scott Ambler

What is test driven development?

• Test cases drive the design

• You write the tests first then you write the code to make the test pass

• This keeps you focused on the purpose of the code

• Code is of no use if your client can’t call it

  Why devs don’t test:

• I already know my code works

• I don’t write broken code

• I have no time

Basic TDD workflow

Why is TDD important for DevOps?

• It saves time when developing
• You can code faster and with more confidence
• It ensures the code is working as expected
• It ensures that future changes don’t break your code
• In order to create a DevOps CI/CD pipeline, all testing must be automated

Behavior Driven Development (BDD)

• Describes the behavior of the system from the outside
• Great for integration testing
• Uses a syntax both devs and stakeholders can easily understand

BDD vs. TDD:

• BDD ensures that you’re building the “right thing”
• TDD ensures that you are building the “thing right”

BDD workflow

• Explore the problem domain and describe the behavior

• Document the behavior using Gherkin syntax

• Use BDD tools to run those scenarios

• One document that’s both the specification and the tests

  Gherkin:

• An easy-to-read natural language syntax

• Given … When… Then…

• Understandable by everyone

  Gherkin Syntax:

• Given (some context)

• When (some event happens)

• Then (some testable outcome)

• And (more context, events, or outcomes)

Retail BDD example

Gherkin for acceptance criteria:

• Add acceptance criteria to every user story
• Use Gherkin to do that
• Indisputable definition of “done”

Expected benefits of BDD

• Improves communication
• More precise guidance
• Provides a common syntax
• Self-documenting
• Higher code quality
• Acceptance criteria for user stories

Cloud Native Microservices

Think differently about application design:

Think cloud native:

• The Twelve-Factor App
• A collection of stateless microservices
• Each service maintains its own database
• Resilience through horizontal scaling
• Failing instances are killed and respawned
• Continuous Delivery of services

Think microservices:

“The microservices architectural style is an approach to developing a single application as a suit of small services, each running in its own process and communicating with lightweight mechanisms, often an HTTP resource API. These services are built around business capabilities and independently deployable by fully automated deployment machinery.” — Martin Fowler and James Lewis

Monolith vs. Microservices

Designing for Failure

Failure happens:

• Embrace failures – they will happen!
• How to avoid –> How to identify and what to do about it
• Operational concern –> developer concern
• Plan to be throttled
• Plan to retry (with exponential back off)
• Degrade gracefully
• Cache when appropriate

Retry pattern:

Circuit Breaker pattern:

Bulkhead pattern:

Chaos engineering:

• Also known as monkey testing
• You deliberately kill services
• Netflix created The Simian Army tools
• You cannot know how something will respond to a failure until it actually fails

Working DevOps

Taylorism and Working in Silos

Working DevOps:

• Culture of teaming and collaboration

• Agile development as a shared discipline

• Automate relentlessly

• Push smaller releases faster

  Taylorism:

• Adoption of command and control management

• Organizations divided into functional silos

• Decision-making is separated from work

Impact of Taylorism on IT:

Software development is bespoke:

• Software development is NOT like assembling automobiles

• Most of the parts don’t exist, yet

• Software development is craft work

• Taylorism is not appropriate for craft work

  Abandon Command and Control:

• Command and control is not Agile

• Stop working in silos

• Let your people amaze you

Software Engineering vs. Civil Engineering

Software engineering is organic:

• Software stack is constantly updated

• New features are being added

• System behavior changes over time

• Yet we treat software engineering like a civil engineering project

  The project model is flawed:

• The project model doesn’t work for software development

• Treat software development like product development

• Encourage ownership and understanding

• Software engineering is not civil engineering

• Maintain stable, lasting teams

Required DevOps Behaviors

Diametrically opposed views:

• Enterprises see “new” as complex and time-consuming
• DevOps delivers a continual series of small changes
• These cannot survive traditional overheads

A clash of work culture:

The no-win scenario:

• Development wants innovation
• Operations wants stability

Operations view of development:

• Development teams throw dead cats over the wall

• Manually implemented changes

• Lack of back-out plans

• Lack of testing

• Environments that don’t look like production

  Development view of operations:

• All-or-nothing changes

• Change windows in the dead of night

• Implemented by people furthest away from the application

• Ops just cuts and pastes from “runbooks”

  No-win scenario:

• If the website works, the developers get the praise!

• If the website is down, operations gets the blame!

  Required DevOps behaviors:

Infrastructure as Code

• Described an executable textual format

• Configure using that description

• Configuration Management Systems to make this possible (Ansible, puppet etc.)

• Never perform configurations manually

• Use version control

  Ephemeral immutable infrastructure:

• Server drift is a major source of failure

• Servers are cattle not pets

• Infrastructure is transient

• Build through parallel infrastructure

  Immutable delivery via containers:

• Applications are packaged in containers

• Same container that runs in production can be run locally

• Dependencies are contained

• No variance limits side effects

• Rolling updates with immediate roll-back

  Immutable way of working:

• You never make changes to a running container

• You make changes to the image

• Then redeploy a new container

• Keep images up-to-date

Continuous Integration (CI)

CI vs. CD:

• CI/CD is not one thing
• Continuous Integration (CI):
  • Continuously building, testing, and merging to master
• Continuous Delivery (CD):

  • Continuously deploying to a production-like environment

    Traditional Development:

• Devs work in long-lived development branches
• Branches are periodically merged into a release
• Builds are run periodically
• Devs continue to add to the development branch

Continuous Integration

• Devs integrate code often
• Devs work in short-lived feature branches
• Each check-in is verified by an automated build

Changes are kept small:

• Working in small batches

• Committing regularly

• Using pull requests

• Committing all changes daily

  CI automation:

• Build and test every pull request

• Use CI tools that monitor version control

• Test should run after each build

• Never merge a PR with failing tests

  Benefits of CI:

• Faster reaction times to changes

• Reduced code integration risk

• Higher code quality

• The code in version control works

• Master branch is always deployable

Continuous Delivery

“Continuous Delivery is a software development discipline where you build software in such a way that the software can be released to production at any time.” — Martin Fowler

Release to production at any time:

• The master branch should always be ready to deploy
• You need a way to know if something will “break the build”
• Deliver every change to a production-like environment

CI/CD pipeline:

• Automated gates that create a pipeline of checks:
  • Unit testing
  • Code quality checks
  • Integration testing
  • Security testing
  • Vulnerability scanning
  • Package signing

A CI/CD pipeline needs:

• A code repository
• A build server
• An integration server
• An artifact repository
• Automatic configuration and deployment

Continuous integration and delivery:

Five key principles:

1. Build quality in
2. Work in small batches
3. Computers perform repetitive tasks, people solve problems
4. Relentlessly pursue continuous improvement
5. Everyone is responsible

CI/CD + Continuous deployment:

How DevOps manages risk:

• Deployment is king
• Deployment is decoupled from activation
• Deployment is not “one size fits all”

Organizing for DevOps

Organizational Impact of DevOps

How does organization affect DevOps?

Is the culture of your organization agile?

• Small teams
• Dedicated teams
• Cross-functional teams
• Self-organizing teams

Conway’s Law:

“Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.” — Melvin Conway, Datamation, 1968

Traditional organization around technology:

Organized around business domains:

Align teams with the business:

• Each team has its own mission aligned with the business
• Teams have end-to-end responsibility for what they build
• Teams should have a long-term mission, usually around a single business domain

There is No DevOps Team

DevOps is often misunderstood:

• Dev and Ops are not separate things
• You aren’t a DevOps if you’re not a Dev, and other way is also true

Perspectives on DevOps:

DevOps is not a team:

“The DevOps movement addresses the dysfunction that results from organizations composed of functional silos. Thus, creating another functional silo that sits between Dev and Ops is clearly a poor (and ironic) way to try, and solve these problems.” — Jez Humble, The DevOps Handbook

Working in silos doesn’t work:

A DevOps team means we’re DevOps, right?

DevOps is not a job title:

• A culture transformation on an organizational scale
• Development and operations engineers working together
• Following lean and agile principles
• Cross-functional teams with openness, transparency, and trust as pillars

Everyone is Responsible for Success

Bad behavior:

“Bad behavior arises when you abstract people from the consequences of their actions.” — Jez Humble, Continuous Delivery

Functional silos breed bad behavior:

Actions have consequences:

• Make people aware of the consequences of their actions
• Make people responsible for the consequences of their actions

DevOps organizational objective:

Shared consciousness

…with

distributed (local) control

Measuring DevOps

Rewarding for “A” while hoping for “B”

On the folly of rewarding for A, while hoping for B

“Whether dealing with monkeys, rats, or human beings, it is hardly controversial to state that most organisms seek information concerning what activities are rewarded, and then seek to do (or at least pretend to do) those things, often to the virtual exclusion of activities not rewarded. The extent to which this occurs, of course, will depend on the perceived attractiveness of the rewards offered, but neither operant nor expectancy theorists would quarrel with the essence of this notion.” — Steven Kerr, The Ohio State University

Measure what matters:

Social metrics:

• Who is leveraging the code you are building?
• Whose code are you leveraging?

DevOps metrics:

• A baseline provides a concrete number for comparison as you implement your DevOps changes
• Metric goals allow you to reason about these numbers and judge the success of your progress

DevOps changes the objective:

• Old school is focused on mean time to failure (MTTF)
• DevOps is focused on mean time to recovery (MTTR)

Vanity metrics vs. actionable metrics

Vanity metrics:

• We had 10,000 daily hits to our website!

• Now what? (What does a hit represent?)

• What actions drove those visitors to you?

• Which actions to take next?

  Actionable metrics:

Actionable metric examples:

• Reduce time to market
• Increase overall availability
• Reduce time to deploy
• Defects detected before production
• More efficient use of infrastructure
• Quicker performance feedback

Top four actionable metrics:

1. Mean lead time
2. Release frequency
3. Change failure rate
4. Mean time to recovery (MTTR)

How to Measure Your Culture

Culture measurements:

You can rate statements developed by Dr. Nicole Forsgren to measure your team’s culture, including statements about information, failures, collaboration, and new ideas.

Strongly agree or disagree?

• On my team, information is actively sought
• On my team, failures are learning opportunities and messengers of them are not punished
• On my team, responsibilities are shared
• On my team, cross-functional collaboration is encouraged and rewarded
• On my team, failure causes inquiry
• On my team, new ideas are welcomed

Comparison of DevOps to Site Reliability Engineering

What is SRE?

“…what happens when a software engineer is tasked with what used to be called operations.” —Ben Treynor Sloss

Goal: Automate yourself out of a job.

Tenets of SRE:

• Hire only software engineers

• Site reliability engineers work on reducing toil through automation

• SRE teams are separate from development teams

• Stability is controlled through error budgets

• Developers rotate through operations

  Team differences:

• SRE maintains separate development and operations silos with one staffing pool

• DevOps breaks down the silos into one team with one business objective

Maintaining stability:

Commonality:

• Both seek to make both Dev and Ops work visible to each other

• Both require a blameless culture

• The objective of both is to deploy software faster with stability

  DevOps + SRE:

• SRE maintains the infrastructure

• DevOps uses infrastructure to maintain their applications

Overview of Cloud Computing

Cloud Computing Models

Components of Cloud Computing

Emerging Trends and Practices

Cloud Security, Monitoring, Case Studies, Jobs

Introduction to Agile and Scrum

• > 70% of organizations have incorporated some Agile approaches. — Project Management Institute
• 28% more successful using agile than traditional projects. — Price Waterhouse Coopers
• 47% agile transformations’ failure rate. — Forbes
  • #1 reason is inexperience with implementing and integrating the Agile methodology. — VersionOne
• Agile is a mindset that requires culture change.
• It’s hard to learn Agile from just reading a book.
• Recognizing when something is wrong is just as important as knowing how to do something right.

Introduction to Agile Philosophy: Agile Principles

What is Agile?

Agile is an iterative approach to project management that helps teams be responsive and deliver value to their customers faster

Agile defining characteristics:

Agile emphasizes:

• Adaptive planning
• Evolutionary development
• Early delivery
• Continual improvement
• Responsiveness to change

Agile Manifesto:

We have come to value:

Individuals and interactions over processes and tools.

Working software over comprehensive documentation.

Customer collaboration over contract negotiation.

Responding to change over following a plan.

That is, while there is value in the items on the right, we value the items on the left more.

Agile Software development:

• An iterative approach to software development consistent with Agile Manifesto
• Emphasizes flexibility, interactivity, and a high level of transparency
• Uses small, co-located, cross-functional, self-organizing teams

Key takeaway:

Build what is needed, not what was planned.

Methodologies Overview

Traditional Waterfall Development:

Problems with waterfall approach:

• No provisions for changing requirements
• No idea if it works until the end
• Each step ends when the next begins
• Mistakes found in the later stages are more expensive to fix
• There is usually a long time between software releases
• Teams work separately, unaware of their impact on each other
• The people who know the least about the code are deploying it into production

Extreme Programming (XP)

• In 1996 Kent Beck introduced XP
• Based on an interactive approach to software development
• Intended to improve software quality and responsiveness to changing customer requirements
• One of the first Agile method

Extreme Programming values:

• Simplicity
• Communication
• Feedback
• Respect
• Courage

Kanban

What is Kanban?

Kanban | ‘kanban | noun

(also Kanban system) a Japanese manufacturing system in which the supply of components is regulated through the use of an instruction card sent along the production line.

• An instruction card used in a Kanban system.

Origin

1970s: Japanese, Literally mean ‘billboard, sign’

Core principles of Kanban:

• Visualize the workflow
• Limit work in progress (WIP)
• Manage and enhance the flow
• Make process policies explicit
• Continuously improve

Working Agile

• Working in small batches
• Minimum Viable Product (MVP)
• Behavior Driven Development (BDD)
• Test Driven Development (TDD) (Gherkin Syntax — Developed by Cucumber Company)
• Pair programming

Introduction to Scrum Methodology

Agile and Scrum:

• Agile is a philosophy for doing work (not prescriptive)
• Scrum is a methodology for doing work (add process)

Scrum Overview

Scrum:

• Is a management framework for incremental product development

• Prescribes small, cross-functional, self-organizing teams

• Provides a structure of roles, meeting, rules, and artifacts

• Uses fixed-length iterations called sprints

• Has a goal to build a potentially shippable product increment with every iteration

• Easy to Understand – Difficult to master

  Sprint:

• A sprint is one iteration through the design, code, test, and deploy cycle

• Every sprint should have a goal

• 2 weeks in duration

Steps in the Scrum process:

Agile development is iterative:

The 3 Roles of Scrum

Scrum roles:

• Product owner
• Scrum master
• Scrum team

1. Product owner:

• Represents the stakeholder interests
• Articulates the product vision
• Is the final arbiter of requirements’ questions
• Constantly re-prioritizes the product backlog, adjusting any expectations
• Accepts or rejects each product increment
• Decides whether to ship
• Decides whether to continue development

1. Scrum master:

• If your team is experienced, you might skip this role, but if you have a team new to Scrum, you require an experienced Scrum master.
  • Facilitates the Scrum process
  • Coaches the team
  • Creates an environment to allow the team to be self-organizing
  • Shields the team from external interference to keep it “in the zone”
  • Helps resolve impediments
  • Enforces sprint timeboxes
  • Captures empirical data to adjust forecasts
  • Has no management authority over the team

2. Scrum Team:

• A cross-functional team consisting of
  • Developers
  • Testers
  • Business analysts
  • Domain experts
  • Others
• Self-organizing
  • There are no externally assigned roles
• Self-managing
  • They self-assign their own work
• Membership: consists of 7 ± 2 collaborative members
• Co-located: most successful when located in one team room, particularly for the first few Sprints
• Dedicated: Most successful with long-term, full-time membership
• Negotiates commitments with the product owner – one sprint at a time
• Has autonomy regarding how to reach commitments

Artifacts, Events, and Benefits

Scrum Artifacts:

• Product backlog

• Sprint backlog

• Done increment

  Scrum events:

• Sprint planning meeting

• Daily Scrum meeting (a.k.a. daily stand-up)

• Sprint

• Sprint review

• Sprint retrospective

  Benefits of Scrum:

• Higher productivity

• Better product quality

• Reduced time to market

• Increased stakeholders satisfaction

• Better team dynamics

• Happier employees

  Scrum vs. Kanban:

Organizing for Success: Organizational impact of Agile

Organize for success:

• Proper organization is critical to success

• Existing teams may need to be reorganized

  Conway’s Law:

  “Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.”

  — Melvin Conway, Datamation, 1968

  Examples of Conway’s Law:

  If you ask an organization with four teams to write a compiler … you will get a 4-pass compiler!

  How teams should be aligned?

• Teams are loosely coupled, tightly aligned

• Each team has its own mission aligned with the business (like a “mini startup”)

• Teams have end-to-end responsibility for what they build

• The long-term mission is usually around a single business domain

  Autonomy is important:

• It’s motivating – and motivated people build better stuff

• It’s fast – decisions happen locally in the team

• It minimizes handoffs and waiting, so teams don’t get bogged down

The Agile dilemma!

The entire organization must be Agile:

Agile + DevOps = Alignment

Mistaking Iterative Development for Agile

The biggest pitfall for companies is that, they think they’re agile, but actually they’re just doing an iterative work.

Agile is not…

• Agile isn’t a new version of a waterfall, software development life cycle (SDLC), where you do legacy development in sprints
• Agile isn’t just the developers working in each sprint, it involves a cross-functional team
• The Agile Manifesto doesn’t include the term “Agile project management” (and so there are no “project managers” in Agile)

Agile Planning

Planning to be Agile: Destination Unknown

Deadlines:

“I love deadlines… I like the whooshing sound they make as they fly by”. – Douglas Adams

• How do you avoid this?

  Plan iteratively:

• Don’t decide everything at the point when you know the least

• Plan for what you know

• Adjust as you know more

• Your estimates will be more accurate

Agile Roles and the Need for Training

Formulas for failure:

• Product manager becomes product owner
• Project manager becomes scrum master
• Developers (alone) become scrum team

Product Manager vs. Product Owner:

Project Manager vs. Scrum Master:

Development Team vs. Scrum Team:

“Until and unless business leaders accept the idea that they are no longer managing projects with fixed functions, timeframes, and costs, as they did with waterfall, they will struggle to use agile as it was designed to be used.”

— Bob Kantor, Founder Kantor Consulting Group, Inc.

The roles have changed:

• You cannot put people in new roles without the proper training and mindset
• This mindset must come down from upper management

Kanban and Agile Planning Tools

Agile planning tools:

• Tools will not make you Agile

• Tools can support your Agile process

• Many Agile planning tools

• ZenHub is one of them

  ZenHub:

• Plug-in to GitHub

• Provides a kanban board and project management reporting

• Customizable and integrated with GitHub

  Why use ZenHub?

• Helps you manage where you are in a project based on GitHub Issues

• Provides an easy way to let management know how you are doing

• Maintains up-to-date status due to integration with GitHub

• Allows developers to only use one tool – GitHub

What is Kanban Board?

Real World Example:

Default ZenHub pipelines:

User Stories: Creating Good User Stories

What is a user story?

A user story represents a small piece of business value that a team can deliver in an iteration.

Story contents:

Stories should contain:

• A brief description of the need and business value

• Any assumptions or details

• The definition of “done”

  Story description:

• User stories document a persona requesting a function to achieve a goal:

  As a <some role>

  I need <some function>

  So that <some benefit>

  Assumptions and details:

  It’s important to document what you know;

• List any assumptions

• Document any details that may help the developer

  Acceptance criteria:

• It is critical to document the definition of “done”

• I like to use the Gherkin syntax

  Given <some precondition>

  When <some event happens>

  Then <some outcome>

  Sample Story:

Bill Wake’s INVEST:

• Independent

• Negotiable

• Valuable

• Estimable

• Small

• Testable

  Epic:

• A big idea

• A user story that is bigger than a single sprint

• A user story that is too big to estimate on its own

  When to use an epic?

• When a story is too large in scope it is considered an epic

• Backlog items tend to start as epics when they are lower priority and less defined

• For sprint planning, epics should be broken down into smaller stories

Effectively using Story Points

What are story points?

Story point;

• A metric used to estimate the difficulty of implementing a given user story
• An abstract measure of overall effort

What does a story point measure?

Relative T-Shirt sizes

• Story points acknowledge that humans are bad at estimating time-to-completion

• Instead, story points use relative T-Shirt sizes (S, M, L, XL)

• Most tools use Fibonacci numbers (1, 2, 3, 5, 8, 13, 21)

  Agree on what “medium” means:

• Since story points are relative, it’s important to agree on what “medium” is

• Then, evaluate from there

• Is it the same, larger, or smaller than medium

  Story size:

• A story should be small enough to be coded and tested within a single sprint iteration – ideally, just a few days

• Large stories should be broken down into smaller ones

  Story point antipattern

• Equating a story point to wall-clock time

• Humans are bad at estimating wall-clock time

• Don’t do it!

Building the Product Backlog

Steps in the Scrum process:

Product Backlog:

• A product backlog contains all the unimplemented stories not yet in a sprint
• Stories are ranked in order of importance and/or business value
• Stories are more detailed at the top, less detailed at the bottom

Sample requirements:

• What: A service for counting things
• Must allow multiple counters
• Counters must persist across restarts of service
• Counters can be reset

ZenHub Kanban board:

Creating new stories

Story Template:

As a <some role>

I need <some function>

So that <some benefit>

Need a service for counting things:

As a User

I need a service that has a counter

So that I can keep track of how many times something was done

Creating the next story:

Must allow multiple counters:

As a User

I need to have multiple counters

So that I can keep track of several counts at once

Creating the next story:

Persist counters across restarts:

As a Service Provider

I need the service to persist the last known count

So that users don’t lose track of their counts after the service is restarted

Creating the last story:

Counters can be reset:

As a System Administrator

I need the ability to reset the counter

So that I can redo counting from the start

Stories in the backlog:

Prioritize the product backlog:

The Planning Process

Backlog Refinement: Getting Started

Backlog refinement:

• Keep the product backlog ranked by priority so that the important stories are always on the top

• Break large stories down into smaller ones

• Make sure that stories near the top of the backlog are groomed and complete

  Backlog refinement meeting:

  Who should attend?

• Product owner

• Scrum master

• Development team (optional)

  • Lead developer/architect

    What is the goal?

• Groom the backlog by ranking the stories in order of importance

• Make sure the story contains enough information for a developer to start working on it

Backlog refinement workflow:

New issue triage:

• Start with new issue triage

• Goal: At the end of backlog refinement, the New Issues column is empty

  Take stories from new issues and…

• Move them into the product backlog if they will be worked on soon

• Move them into the icebox if they are a good idea but not now

• Reject them if they are not where you want to go

Backlog refinement workflow:

• Product owner sorts the product backlog in order of importance
• The team may provide estimates and other technical information
• Large vague items are split and clarified
• The goal is to make the stories “sprint read”

Complete the story template:

As a <some role>

I need <some function>

So that <some benefit>

Assumptions and Details:

<anything you already know>

Acceptance Criteria:

Given <some precondition>

When <some event>

Then <some measurable outcome>

Need a service that has a counter:

Must Persist counter across restarts:

Deploy service to the cloud:

Ability to reset the counter:

Backlog Refinement: Finishing Up

Label:

• Help visualize the work

Labels in GitHub

Need a service that has a counter:

Must persist counter across restarts:

Deploy service to the cloud:

Ability to reset the counter:

Technical debt

• Technical debt is anything you need to do that doesn’t involve creating a new feature

• Technical debt builds up when you take shortcuts, but may also occur naturally

  Examples of technical debt:

• Code refactoring

• Setup and maintenance of environments

• Changing technology like databases

• Updating vulnerable libraries

Backlog refinement Tips

• You should refine the backlog every sprint to ensure the priorities are correct
• Have at least two sprints’ worth of stories groomed
• The more time you spend refining the backlog, the easier sprint planning will be

Sprint Planning

• The purpose of sprint planning is to define what can be delivered in the sprint and how that work will be achieved
• This is accomplished by producing a sprint backlog

Sprint planning meeting

Who should attend?

• Product owner

• Scrum master

• Development team

  Sprint planning goals:

• Each sprint should have a clearly defined business goal

• The product owner describes the goal and product backlog items supporting it

• It’s important for the whole team to understand why they are building the increment

  Mechanics of sprint planning

  The development team;

• Takes stories from the top of the product backlog and assigns them to the sprint backlog

• Assigns story points and labels

• Ensures each story contains enough information for a developer to start working on it

• Stops adding stories when the team’s velocity is reached

  Team velocity:

• The number of story points a team can complete in a single sprint

• This will change over time as the team gets better at estimating and better at executing

• The velocity is unique to the team because the story point assignment is unique to the team

  Create a sprint milestone:

• Create a sprint milestone to start the sprint

• The milestone title should be short

• The description should document the milestone goal

• The duration should be 2 weeks

Create a milestone:

Executing the Plan

Workflow for Daily Plan Execution

Steps in the Scrum Process:

The Sprint:

• A sprint is one iteration through the design, code, test, deploy cycle

• It is usually 2 weeks in duration

• Every sprint should have a goal

  Daily Execution:

• Take the next highest priority item from the sprint backlog

• Assign it to yourself

• Move it in process

• No one should have more than one story assigned to them unless they are blocked
• When you are finished, move the story to Review/QA and open a PR
• When the PR is merged, move the story to the Done column

The Daily Stand-Up

• Occurs every day at the same time and place

• Sometimes called the “daily Scrum”

• Each team member briefly reports on their work

• Called a “stand-up” during the meeting to keep it short

• Timeboxed to 15 mins

• Not a project status meeting – all status should be tabled for later discussion

  Daily stand-up meeting:

  Who should attend?

• Scrum master

• Development team

• Product owner (optional)

  Daily stand-up question:

  Each team member answers three questions:

  1. What did I accomplish the previous day?
  2. What will I work on today?
  3. What blockers or impediments are in my way?

  Impediments and blockers:

• Impediments identified by the team should be unblocked by the scrum master

• Developers that are blocked should work on the next story

  Tabled topics:

• Topics raised during the daily stand-up should be held until the meeting has ended

• Anyone interested in those topics can stay to discuss

Completing the Sprint

Using Burndown Charts

Milestones and burndowns:

• Milestones can be created for anything in your project

  • sprint, beta drop, demo, release…

• Burndown charts can be used to measure your progress against a milestone

  Burndown chart:

• The measurement of story points completed vs. story points remaining for a sprint

• Over time the story points remaining should go down, hence the name: burndown

Burndown chart examples:

The Sprint Review

• Live demonstration of implemented stories

• Product owner determines if stories are done based on acceptance criteria

• Done stories are closed

  Sprint Review meeting:

  Who should attend?

• Product owner

• Scrum master

• Development team

• Stakeholders

• Customers (Optional)

  Sprint review:

• Feedback gets converted into new product backlog stories

• This is where iterative development allows the creation of products that couldn’t have been specified up-front in a plan-driven approach

  Rejected Stories:

• What about stories that are not considered done?

• Add a label to indicate this and close them

• Write a new story with new acceptance criteria

• This will keep the velocity more accurate

The Sprint Retrospective

• A meeting to reflect on the sprint

• Measures the health of the process

• The development team must feel comfortable to speak freely

  Who attend the meeting:

• Scrum master

• Development team

  A time for reflection:

  Three questions are answered:

  1. What went well? (keep doing)
  2. What didn’t go well? (stop doing)
  3. What should we change for the next sprint?

  The goal is improvement:

• This is critical for maintaining a healthy team

• The scrum master must ensure that changes are made as a result of the feedback

• The goal is to improve for the next sprint

Measuring Success

Using Measurements Effectively

Measurements and metrics:

• You can’t improve what you can’t measure

• High performing teams use metrics to continually improve

• They take baselines and set goals and measure against them

• Beware of vanity metrics

• Look for the actionable metrics

  Baselines and Goals:

  Baseline:

• It currently requires 5 size team members, 10 hours to deploy a new release of your product

• This costs you $X for every release

  Goals:

• Reduce deployment time from 10 hours to 2 hours

• Increase percentage of defects detected in testing from 25% to 50%

Top 4 actionable metrics:

1. Mean Lead Time

• How long does it take from the idea to production?

1. Release Frequency

• How often can you deliver changes?

1. Change Failure Rate

• How typically do changes fail?

1. Meantime to Recovery (MTTR)

How quickly can you recover from failure?

Example metrics:

• Reduce time-to-market for new features
• Increase overall availability of the product
• Reduce the time it takes to deploy a software release
• Increase the percentage of defects detected in testing before production release
• Provide performance and user feedback to the team in a more timely manner

Getting Ready for the Next Sprint

End of sprint activities:

• Move stories from done to closed

• Close the current milestone

• Create a new sprint milestone

• Adjust unfinished work

  Handling untouched stories:

• Stories not worked on can be moved to the top of the product backlog

• Resist the urge to move them to the next sprint

• Remember to unassign them from the sprint milestone

  Handling unfinished stories:

• Don’t move unfinished stories into the next sprint!

• Give the developers credit for the work they did

• This will keep your velocity more accurate

• Adjust the description and story points of the unfinished story, label it unfinished, and move it to done

• Write a new story for the remaining work

• Assign remaining story points and move it to the next sprint

  Ready for the next sprint:

• All stories assigned to the current sprint are closed

• All unfinished stories are reassigned

• The sprint milestone is closed

• A new sprint milestone is created

Agile Anti-Patterns and Health Check

Agile Anti-Patterns:

• No real product owner/Multiple product owners

• Teams are too large

• Teams are not dedicated

• Teams are too geographically distributed

• Teams are siloed

• Teams are not self-managing

  YOU WILL FAIL!

  …and you should not wonder why.

Scrum health check:

• The accountabilities of product owner, development team(s) and Scrum master are identified and enacted
• Work is organized in consecutive sprints of 2–4 weeks or fewer
• There is a sprint backlog with a visualization of remaining work for the sprint
• At sprint planning a forecast, a sprint backlog, and a sprint goal are created
• The result of the daily Scrum is work being re-planned for the next day
• No later than by the end of the sprint, a Done increment is created
• Stakeholders offer feedback as a result of inspecting the increment at the sprint review
• Product backlog is updated as a result of the sprint review
• Product owner, development team(s) and Scrum master align on the work process for their next sprint at the sprint retrospective

SDLC - The Software Development Lifecycle

What is Software Engineering?

• Application of scientific principles for design and creation of software
• Systematic approach to software development
  • Design
  • Build
  • Test

Software Crisis:

• Began in mid-1960s

• Resulted from software development that ran over budget and behind schedule with unmanageable and buggy code

• By the time old software solutions were developed, the newer technologies got the traction, and code refactoring would become necessary

• Solutions didn’t scale well to large projects

  Engineering principles:

• Resolved as standardized methods in software engineering and computer-aided software engineering (CASE) tools were developed

Software engineer vs. software developer:

SE responsibilities:

• Design, build and maintain software systems

• Write and test code

• Consult with stakeholders, third party vendors’ security specialists, and other team members

  Software Development Life Cycle (SDLC):

• Scientific approach to software development

• Guides the software development process

• Identifies discrete steps needed to develop software

Introduction to SDLC

• Systematic process to develop high-quality software

• Aims to produce software that meets requirements

• Defined phases with their own processes and deliverables

• Cycle of planning, design, and development

• Minimizes development risks and costs

  History of the SDLC:

• Conceived of in the mid-1960s

• A deliberate approach needed to manage complex projects

• Initially used the waterfall method

• Adapted to use iterative methods

  Advantages of the SDLC:

• Improves efficiency and reduce risks

• Team members know what they should be working on and when

• Facilitates communication among stakeholders

• Team members know when development can move to the next phase

• Respond to changing requirements

• Solve problems early in the process

• Reduces overlapping responsibilities

Phases of the SDLC

• Organizations may have different names for each stage
• Some organizations have more or fewer stages

Phase 1: Planning

Requirements;

• Gathered
• Analyzed
• Documented
• Prioritized

Prototyping:

• Small-scale replica to clarify requirements

• Tests design ideas

• Can be developed at various stages of the SDLC

  Software Requirements Specification:

• Requirements are documented in the SRS

• All stakeholders must agree

Phase 2: Design

Phase 3: Development

Development starts when the design document is finalized and sent to the developers to write code for it.

Phase 4: Testing

• Code is tested to ensure stability, security, and that it meets requirements from the SRS

• Bugs reported, tracked, fixed, and retested

  Some common tests:

• Unit testing

• Integration testing

• System testing

• Acceptance testing

Phase 5: Deployment

Phase 6: Maintenance

Building Quality Software

Common software engineering processes:

• Requirements gathering
• Design
• Coding for quality
• Testing
• Release
• Documenting

Requirement Gathering

• The SRS encompasses the process of collecting and documenting the set of requirements that the software needs to adhere to

• It may include a set of use cases that describe the business needs and user flows that the software must implement

  Software requirements can be classified into four broad categories:

  1. Functionality
  2. External & user interface
  3. System features
  4. Non-functional

Design

• Transforming requirements into code
• Breaking down requirements into sets of related components
• Communicating business rules and application logic

Coding for quality

It refers to characteristics of the code, including attributes:

• Maintainability

• Readability

• Testability

• Security

  Quality code must fulfill the intended requirements of the software without defects

• Clean and consistent

• Easy to read and maintain

• Well documented

• Efficient

  Coding for quality entails a following set of coding practices during development;

• Following coding standards

• Using linters to detect errors

• Commenting in the code itself to make it easy to understand and modify

Testing

The process of verifying that the software matches established requirements and is free of bugs

• Identify errors, gaps, or missing requirements

• Ensures reliability, security, performance, and efficiency

• Software testing can often be automated or done manually

  The types of testing are;

• Unit testing

• Integration testing

• System testing

• User acceptance testing (UAT) or Beta testing

Releases

Documenting

System documentation:

README files, inline comments, architecture and design documents, verification information, and maintenance guides

User documentation:

User guides, instructional videos, manuals, online and inline help

Requirements

Steps to gathering requirements:

• Identifying stakeholders
• Establishing goals and objectives
• Eliciting requirements from the stakeholders
• Documenting the requirements
• Analyzing and confirming the requirements
• Prioritizing

1) Identifying stakeholders

Key personnel:

• Decision-makers
• End-users
• System administrators
• Engineering Marketing
• Sales
• Customer support

2) Establishing goals and objectives

Goals: broad, long-term achievable outcomes

Objectives: actionable, measurable actions that achieve the goal

3) Eliciting, documenting, confirming

Elicit:

• Surveys

• Questionnaires

• Interviews

  Document:

• Align with goals and objectives

• Easily understood

  Confirm:

• Consistency

• Clarity

• Completeness

4) Prioritizing

• Must-have
• Highly desired
• Nice to have

Requirements documentation:

• Software requirements specification (SRS)
• User requirements specification (URS)
• System requirements specification (SysRS)

1) Software requirements specification (SRS)

• Captures functionalities the software should perform
• Establishes benchmarks / service-levels for performance
• Purpose and scope
  • Purpose
    • Who has access to the SRS
    • How it should be used
  • Scope - Software benefits
• Constraints, assumptions, dependencies
  • Constraints: how the software must operate under given conditions
  • Assumptions: required OS or hardware
  • Dependencies: on other software products
• Requirements
  • Functional: functions of the software
  • External interface: users and interactions with other hardware or software
  • System features: functions of the system
  • Non-functional: performance, safety, security, quality

2) User requirements specification (URS)

• Describe business need and end-user expectations
• User stories:
  • Who is the user?
  • What is the function that need to be performed?
  • Why does the user want this functionality?
• Confirmed during user acceptance testing
• Often combined into the SRS

3) System Requirements Specification (SysRS)

• Outlines requirements of the system
• Broader than an SRS
• Contains;
  • System capabilities
  • Interface and user characteristics
  • Policy
  • Regulation
  • Personnel
  • Performance
  • Security
  • System acceptance criteria
  • Hardware expectations

The Software Building Process and Associated Roles

Software Development Methodologies

Common development methodologies:

A process is needed to clarify communication and facilitates information sharing among team members.

Some of these methodologies are:

• Waterfall

• V-shape model

• Agile

Sequential vs. iterative:

Waterfall pros and cons

V-shape model pros and cons

Agile pros and cons

Software Versions

• Software versions are identified by version numbers, indicate:

  • When the software was released
  • When it was updated
  • If any minor changes or fixes were made to the software

• Software developers use versioning to keep track of new software, updates, and patches

  Version numbers:

• Version numbers can be short or long, with 2, ,3, or 4 set

• Each number set is divided by a period

• An application with a 1.0 version number indicates the first release

• Software with many releases and updates will have a larger number

• Some use dates for versioning, such as Ubuntu Linux version 18.04.2 released in 2018 April, with a change shown in the third number set

  What do version numbers mean?

  Some version numbers follow the semantic numbering system, and have 4 parts separated by a period

• the first number indicates major changes to the software, such as a new release

• The second number indicates that minor changes were made to a piece of software

• The third number in the version number indicates patches or minor bug fixes

• The fourth number indicates build numbers, build dates, and less significant changes

Version compatibility:

• Older versions may not work as well in newer versions
• Compatibility with old and new versions of software is a common problem
• Troubleshoot compatibility issues by viewing the software version
• Update software to a newer version that is compatible
• Backwards-compatible software functions properly with older versions of files, programs, and systems

Software Testing

• Integrate quality checks throughout SDLC
• Purpose
  • Ensure software meets requirements
  • Error-free software

Test cases:

Three types of testing:

1. Functional testing:

The purpose of functional is to check:

• Usability
• Accessibility

2. Non-functional testing

Its attributes are:

• Performance
• Security
• Scalability
• Availability

Non-functional testing questions:

• How does the application behave under stress?
• What happens when many users log in at the same time?
• Are instructions consistent with behavior?
• How does the application behave under different OSs?
• How does the application handle disaster recovery?
• How secure is the application?

3. Regression Testing

• Confirms changes don’t break the application
• Occurs after fixes such as change in requirements or when defects are fixed

Choosing test cases for regression testing:

Testing levels

Unit → Integration → System → Acceptance

1. Unit testing

• Test a module code
• Occurs during the build phase of the SDLC
• Eliminate errors before integration with other modules

2. Integration testing

• Identify errors introduced when two or more modules are combined
• Type of black-box test
• Occurs after modules are combined into larger application

Purpose of integration testing:

3. System testing

• Compliance with SRS
• Validate the system
• Functional and non-functional
• Staging environment

4. Acceptance testing

Software Documentation

• Written assets

• Video assets

• Graphical assets

  Product vs. process documentation:

  Product Documentation	Process Documentation
  Relates to product functionality	Describes how to complete a task

Types of product documentation

1. Requirements documentation

Intended for the development team including developers, architects, and QA. Describes expected features and functionality.

It includes:

• SRS
• SysRS
• User acceptance specification

2. Design documentation

Written by architects and development team to explain how the software will be built to meet the requirements.

• Consists of both conceptual and technical documents

3. Technical documentation

Written in the code to help developers read the code:

• Comments embedded in code and working papers that explain how the code works, documents that record ideas and thoughts during implementation

4. Quality Assurance documentation

Pertains to the testing team’s strategy progress, and metrics:

• Test plans, test data, test scenarios, test cases, test strategies, and traceability matrices

5. User documentation

Intended for end-users to explain to operate software or help install and troubleshoot system:

• FAQs, installation and help guides, tutorials, and user manuals

Standard operating procedures

• Accompanies process documentation
• Step-by-step instructions on how to accomplish common yet complex tasks
• Ex: organization specific instructions for check in code to a repository
• Types of SOPs
  • Flowcharts
  • Hierarchical
  • Step-by-step

Updating documentation

• Must be kept up-to-date
• Documentation should be reviewed and updated periodically

Roles in Software Engineering Projects

• Project manager / Scrum master

• Stakeholders

• System / software architect

• UX Designer

• Developer

• Tester / QA engineer

• Site reliability / Ops engineer

• Product manager / Product owner

• Technical writer / Information developer

Introduction to Software Development

Overview of Web and Cloud Development

Cloud Applications

• Built to work seamlessly with a Cloud-based back-end infrastructure

• Cloud-based data storage and data processing, and other Cloud services, making them scalable and resilient

  Building websites and cloud applications:

  The environment is divided into two primary areas:

1. Front-End

• Deals with everything that happens at the client-side
• Specializes in front-end coding, using HTML, CSS, JavaScript and related frameworks, libraries, and tools

1. Back-End

• Deals with the server before the code and data are sent to the client
• Handles the logic and functionality and the authentication processes that keep data secure
• Back-end developers may also work with relational or NoSQL databases

Full-stack developers have skills, knowledge, and experience in both front-end and back-end environments.

Developers Tools:

• Code editor
• IDE

Learning Front-End Development

• HTML is used to create the structure and CSS is used to design it and make it appealing

• CSS is also used to create websites that have cross browser compatibility such as PC, mobiles devices etc.

• JS adds interactivity

  A front-end development language is Syntactically Awesome Style Sheets (SASS)

• An extension of CSS that is compatible with all versions of CSS.

• SASS enables you to use things like variables, nested rules, inline imports to keep things organized.

• SASS allows you to create style sheets faster and more easily.

  Learner Style Sheets (LESS)

• LESS enhances CSS, adding more styles and functions.

• It is backwards compatible with CSS.

• Less.js is a JS tool that converts the LESS styles to CSS styles.

  Websites are designed as reactive and responsive

• Reactive or adaptive websites display the version of the website designed for a specific screen size.

• A website can provide more information if opened on a PC than when opened on a mobile device.

• Responsive design of a website means that it will automatically resize to the device.

• If you open up a products’ website on your mobile device, it will adapt itself to the small size of the screen and still show you all the features.

JavaScript’s frameworksorks:

Angular Framework:

• an open-source framework maintained by google

• Allows websites to render HTML pages quickly and efficiently

• Tools for routing and form validation

  React.js:

• Developed and maintained by Meta

• It is a JS library that builds and renders components for a web page

• Routing is not a part of this framework and will need to be added using a third-party tool

  Vue.js:

• maintained by the community and its main focus is the view layer which includes UI, buttons, visual components

• Flexible, scalable, and integrates well with other frameworks

• Very adaptable – it can be a library, or it can be the framework

  The task of a front-end developer evolves continuously.

• The technologies are upgraded constantly, and so the front-end developers need to keep upgrading the websites that they create.

• The websites that they create should work in multiple browsers, multiple operating systems, and multiple devices.

The importance of Back-End Development

• Creates and manages resources needed to respond to client requests

• Enables server infrastructure to process request, supply data and provide other services securely

  What does the back-end developer do?

• Process the data you enter while browsing, such as:

  • Login information
  • Product searches
  • Payment information

• Write and maintain the parts of the application that process the inputs

Back-End Developer skills:

Examples of tasks and associated skills that back-end developers need:

APIs, routing, and endpoints:

• APIs, routes, and endpoints process requests from the Front-End
  • API is a code that works with data
  • Routes is a path to a website or page
  • Endpoint can be an API or route
• Back-end developers create routes to direct requests to correct service
• APIs provide a way for Cloud Apps to access resources from the back-end

Back-end languages and frameworks:

Some popular back-end languages are:

• JavaScript
  • Node.js
  • Express
• Python
  • Django
  • Flask

Working with databases:

Languages and tools for working with databases:

• Structured Query Language (SQL)
• Object-Relational Mapping (ORM)

Introducing Application Development Tools

A cloud application developer’s workbench includes:

• Version Control
• Libraries
  • Collection of reusable code
  • Multiple code libs can be integrated into a project
  • Call from your code when required
  • Used to solve a specific problem or add a specific feature

Frameworks:

• Provide a standard way to build and deploy applications

• Act as a skeleton you extend by adding your own code

• Dictate the architecture of the app

• Call your code

• Allow you less control than libs

  Inversion of Control:

• Libs allow you to call functions as when required

• Frameworks define the workflow that you must follow

• Inversion of control makes the framework extensible

  

  More tools:

• CI/CD

• Build tools

  • Transform source code into binaries for installation
  • Important in environments with many interconnected projects and multiple developers
  • Automate tasks like
    • Downloading dependencies
    • Compiling source code into binary code
    • Packaging that binary code
    • Running Tests
    • Deployment to production systems

Examples of Build Tools:

Packages:

• Packages make apps easy to install
• Packages contain
  • App files
  • Instructions for installation
  • Metadata

Package managers:

• Make working with packages easier
• Coordinate with file archives to extract package archives
• Verify checksums and digital certificates to ensure the integrity and authenticity of the package
• Locate, download, install, or update existing software from a software repository
• Manage dependencies to ensure a package is installed with all packages it requires

Package Managers by platform:

Cloud application package managers:

Introduction to Software Stacks

What is a software stack?

• Combination of technologies

• Used for creating applications and solutions

• Stacked in a hierarchy to support the application from user to computer hardware

• Typically include;

  • Front-end technologies
  • Back-end technologies

  Parts of the software stack:

  

  Examples of software stack:

• Python-Django

• Ruby on Rails

• ASP .NET

• LAMP

• MEAN

• MEVN

• MERN

LAMP Stack:

MEAN and relasted stacks:

Comparison of MEAN, MEVN, and LAMP:

• MEAN
  • All parts use JS – one language to learn
  • Lost of documentation and reusable code
  • Not suited to large-scale applications or relational data
• MEVN
  • Similar to MEAN
  • Less reusable libs
• LAMP
  • Lots of reusable code and support
  • Only on Linux
  • Not suited in non-relational data
  • Uses different languages

Programming Languages and Organization

Interpreted and Compiled Programming Languages

Interpreted programming:

Interpreted programming examples:

Compiled programming:

• Programs that you run on your computer
• Packaged or compiled into one file
• Usually larger programs
• Used to help solve more challenging problems, like interpreting source code

Compiled programming examples:

• C, C++, and C# are used in many OSs, like MS Windows, Apple macOS and Linux
• Java works well across platforms, like the Android OS

Compiled programming:

Comparing Compiled and Interpreted Programming Languages

Choosing a programming language:

Developers determine what programming language is best to use depending on:

• What they are most experienced with and trust
• What is best for their users
• What is the most efficient to use

Programming languages:

Interpreted vs. compiled

Query and Assembly Programming Languages

Programming language levels:

High-level programming languages:

• More sophisticated

• Use common English

• SQL, Pascal, Python

  Low-level programming languages:

• Use simple symbols to represent machine code

• ARM, MIPS, X86

Query languages:

• A query is a request for information from a database
• The database searches its tables for information requested and returns results
• Important that both the user application making the query and the database handling the query are speaking the same language
• Writing a query means using predefined and understandable instructions to make the request to a database
• Achieved using programmatic code (query language/database query language)
• Most prevalent database query language is SQL
• Other query languages available:
  • AQL, CQL, Datalog, and DMX

SQL vs. NoSQL:

• NoSQL (Not Only SQL)
• Key difference is data structures
• SQL databases:
  • Relational
  • Use structured, predefined schemas
• NoSQL databases:
  • Non-relational
  • Dynamic schemas for unstructured data

How does a query language work?

Query language is predominantly used to:

• Request data from a database
• Create, read, update, and delete data in a database (CRUD)
• Database consists of structured tables with multiple row and columns of data

When a user performs a query, the database:

1. retrieve data from the table
2. Arranges data into some sort of order
3. Returns and presents query results

Query statements:

• Database queries are either:

  • Select commands
  • Action commands (CREATE, INSERT, UPDATE)

• More common to use the term “statement”

• Select queries request data from a database

• Action queries manipulate data in a database

  Common query statements:

query statement examples:

Assembly Languages

• Less sophisticated than query languages, structured programming languages, and OOP languages

• Uses simple symbols to represent 0s and 1s

• Closely tied to CPU architecture

• Each CPU type has its own assembly language

  Assembly language syntax:

• Simple readable format

• Entered one line at a time

• One statement per line

  {lable} mnemonic {operand list} ;{comment}

  mov TOTAL, 212 ;Transfer the value in the memory variable TOTAL

  Assemblers:

• Assembly languages are translated using an assembler instead of a compiler or interpreter

• One statement translates into just one machine code instruction

• Opposite to high-level languages where one statement can be translated into multiple machine code instructions

  Translate using mnemonics:

• Input (INP), Output (OUT), Load (LDA), Store (STA), Add (ADD)

  Statements consist of:

• Opcodes that tell the CPU what to do with data

• Operands that tell the CPU where to find the data

Understanding Code Organization Methods

Pseudocode vs. flowcharts:

Flowcharts:

• Graphical or pictorial representation of an algorithm
• Symbols, shapes, and arrows in different colors to demo a process or program
• Analyze different methods of solving a problem or completing a process
• Standard symbols to highlight elements and relationships

Flowchart software:

Pseudocode:

Pseudocode Advantages:

• Simply explains each line of code
• Focuses more on logic
• Code development stage is easier
• Word/phrases represent lines of computer operations
• Simplifies translation to code
• Code in different computer languages
• Easier review by development groups
• Translates quickly and easily to any computer language
• More concise, easier to modify
• Easier than developing a flowchart
• Usually less than one page

Programming Logic and Concepts

Branching and Looping Programming Logic

Introduction to programming logic:

Boolean expressions and variables:

Branching programming logic:

Branching statements allow program execution flow:

• if
• if-then-else
• Switch
• GoTo

Looping programming logic:

While loop: Condition is evaluated before processing, if true, then loop is executed

For loop: Initial value performed once, condition tests and compares, if false is returned, loop is stopped

Do-while loop: Condition always executed after the body of a loop

Introduction to Programming Concepts

What are identifiers?

• Software developers use identifiers to reference program components
  • Stored values
  • Methods
  • Interfaces
  • Classes
• Identifiers store two types of data values:
  • Constants
  • Variables

What are containers?

• Special type of identifiers to reference multiple program elements

  • No need to create a variable for every element
  • Faster and more efficient
  • Ex:
    • To store six numerical integers – create six variables
    • To store 1,000+ integers – use a container

• Arrays and vectors

  Arrays:

• Simplest type of container

• Fixed number of elements stored in sequential order, starting at zero

• Declare an array

  • Specify data type (int, bool, str)
  • Specify max number of elements it can contain

• Syntax

  • Data type > array name > array size []

  int my_array[50]

Vectors:

• Dynamic size
• Automatically resize as elements are added or removed
  • a.k.a. ‘dynamic arrays’
• Take up more memory space
• Take longer to access as not stored in sequential memory
• Syntax
  • Container type/data type in <>/name of array

vector <int> my_vector;

What are functions?

• Consequence of modular programming software development methodology

  • Multiple modular components

• Structured, stand-alone, reusable code that performs a single specific action

• Some languages refer to them as subroutines, procedures, methods, or modules

  Two types:

• Standard library functions – built-in functions

• User-defined functions – you write yourself

What are objects?

• Objects are key to understanding object-oriented programming (OOP)
• OOP is a programming methodology focused on objects rather than functions
• Objects contain data in the form of properties (attributes) and code in the form of procedures (methods)
• OOP packages methods with data structures
  • Objects operate on their own data structure

Objects in programming

• Consist of states (properties) and behaviors (methods)
• Store properties in fields (variables)
• Expose their behaviors through methods (functions)

Software Architecture Design and Patterns

Introduction to Software Architecture

Software architecture and design:

• Design and documentation take place during the design phase of the SDLC

• Software architecture is the organization of the system

• Serves as a blueprint for developers

• Comprised of fundamentals structures and behaviors

  Early design decisions:

• How components interact

• Operating environment

• Design principles

• Costly to change once implemented

• Addresses non-functional aspects

  Why software architecture is important:

• Communication

• Earliest design decisions

• Flexibility

• Increases lifespan

  Software architecture and tech stacks:

• Guides technology stack choice

• Tech stacks must address non-functional capabilities

• Tech stacks include:

  • Software
  • Programming languages
  • Libs
  • Frameworks
  • Architects must weigh advantages and disadvantages of tech stack choices

Artifacts

• Software design document (SDD)
• Architectural diagrams
• Unified modeling language (UML) diagrams

1. Software Design Document (SDD)

• Collection of tech specs regarding design implementation
• Design considerations:
  • Assumptions
  • Dependencies
  • Constraints
  • Requirements
  • Objectives
  • Methodologies

2. Architectural diagrams
   It displays:

• Components
• Interactions
• Constraints
• Confines
• Architectural patterns

2. UML diagrams

• Visually communicate structures and behaviors
• Not constrained by a programming language

Deployment considerations

• Architecture drives production environment choices
• Production environment is the infrastructure that runs and delivers the software
  • Servers
  • Load balancers
  • Databases

Software Design and Modeling

Software Design:

Software design is a process to document:

• Structural components

• Behavioral attributes

  Models express software design using:

• Diagrams and flowcharts

• Unified Modeling Language (UML)

  Characteristics of structured design:

• Structural elements: modules & submodules

• Cohesive

• Loosely coupled

  Structure diagram example:

Behavioral models:

• Describe what a system does but doesn’t explain how it does it
• Communicate the behavior of the system
• Many types of behavioral UML diagrams
  • State transition
  • Interaction

Unified Modeling Language (UML):

• Visual representations to communicate architecture, design, and implementation
• Two types: structural and behavioral
• Programming language agnostic

Advantages of Unified Modeling Language (UML):

State transition diagram example:

Interaction diagram:

Object-Oriented Analysis and Design

Object-Oriented Languages:

• A patient could be an object
• An object contains data, and an object can perform actions

Classes and objects:

Object-Oriented analysis and design:

• Used for a system that can be modeled by interacting objects
• OOAD allows developers to work on different aspects of the same application at the same time
• Visual UML diagrams can be made to show both static structure and dynamic behavior of a system

Class diagram:

Software Architecture Patterns and Deployment Topologies

Approaches to Application Architecture

What is component?

• An individual unit of encapsulated functionality
• Serves as a part of an application in conjunction with other components

Component characteristics:

1. Reusable: reused in different applications
2. Replaceable: easily replaced with another component
3. Independent: doesn’t have dependencies on other components
4. Extensible: add behavior without changing other components
5. Encapsulated: doesn’t expose its specific implementation
6. Non-context specific: operates in different environments

Components examples:

Component-based architecture:

• Decomposes design into logical components
• Higher level abstraction than objects
• Defines, composes, and implements loosely coupled independent components, so they work together to create an application

Services

• Designed to be deployed independently and reused by multiple systems
• Solution to a business need
• Has one unique, always running instance with whom multiple clients communicate

Examples of Services:

• A service is a component that can be deployed independently
• Checking a customer’s credit
• Calculating a monthly loan payment
• Processing a mortgage application

Service-oriented architecture:

• Loosely coupled services that communicate over a network
• Supports building distributed systems that deliver services to other applications through the communication protocol

Distributed systems

• Multiple services located on different machines
• Services coordinate interactions via a communication protocol such as HTTP
• Appears to the end-user as a single coherent system

Distributed system characteristics:

• Shares resources
• Fault-tolerant
• Multiple activities run concurrently
• Scalable
• Runs on a variety of computers
• Programmed in a variety of languages

Nodes:

• Any devices on a network that can recognize, process, and transmit data to other nodes on the network
• Distributed systems have multiple interconnected nodes running services

Distributed system architectures:

Architectural Patterns in Software

Types of architectural patterns:

2-tier

3-tier

Peer-to-peer (P2P)

Event-driven

Microservices

Examples:

Combining patterns

Application Deployment Environments

Application environments:

Include:

• Application code/executables
• Software stack (libs, apps, middleware, OS)
• Networking infrastructure
• Hardware (compute, memory and storage)

Pre-production environments:

Production environment

• Entire solution stack ++

• Intended for all users

• Take load into consideration

• Other non-functional requirements

  • Security
  • Reliability
  • Scalability

• More complex than pre-production environments

  On-premises deployment:

• System and infrastructure reside in-house

• Offers greater control of the application

• Organization is responsible for everything

• Usually more expensive than compared to cloud deployment

Cloud deployment types:

Production Deployment Components

Production deployment infrastructure:

Web and application servers:

Proxy server:

• An intermediate server that handles requests between two tiers

• Can be used for load balancing, system optimization, caching, as a firewall, obscuring the source of a request, encrypting messages, scanning for malware, and more

• Can improve efficiency, privacy, and security

  Databases and database servers:

• Databases are a collection of related data stored on a computer that can be accessed in various ways

• DBMS (Database Management System) controls a database by connecting it to users or other programs

• Database servers control the flow and storage of data

Job Opportunities and Skill sets in Software Engineering

What does a Software Engineer Do?

Software engineering:

• Engineering

• Mathematics

• Computing

  Types of Software:

• Desktop and web applications

• Mobile Applications

• Games

• Operating Systems

• Network controllers

  Types of technologies:

• Programming languages

• Development environments

• Frameworks

• Libs, databases, and servers

  Categories of software engineer:

• Back-end engineers or systems developers

• Front-end engineers or application developers

  Software engineering teams:

• Off-the-shelf software

• Bespoke software

• Internal software

  And within the teams they might work on:

• Data integration

• Business logic

• User interfaces

  Software engineering tasks:

• Designing new software systems

• Writing and testing code

• Evaluating and testing software

• Optimizing software programs

• Maintaining and updating software systems

• Documenting code

• Presenting new systems to users and customers

• Integrating and deploying software

Responsibilities:

Skills Required in Software Engineering

What are hard skills?

Commonly required hard skills in software engineering:

• Programming languages
• Version control
• Cloud computing
• Testing and debugging
• Monitoring
• Troubleshooting
• Agile development
• Database architecture

What are soft skills?

• Hard to define, quantify, or certify
• Easily transferable

Hard skills for software engineers

Analysis and design:

• Analyze users’ needs

• Design solutions

  Development:

• Computer programming

• Coding

• Languages:

  • Java
  • Python
  • C#
  • Ruby

• Frameworks

Test:

• Testing

  • Meets functional specification
  • Easy to use

• Debugging

  Deployment:

• Shell scripting

• Containers

• CI/CD

• Monitoring

• Troubleshooting

Soft Skills for Software Engineers

Teamwork:

• Different teams

  • Project-based
  • Role-based

• Squads

• Pair programming

• Take advantage of strengths

• Learn from each other

  Communication:

• Peers

• Managers

• Clients

• Users

  Time management:

• Time-sensitive projects

  • Meet deadlines
  • Avoid delays

• Teams across time-zones

  Problem-solving:

• Design an appropriate solution

• Write an effective code

• Locate and resolve bugs

• Manage issues

  Adaptability:

• Client changes

• Management request

• User needs

  Open to feedback:

• Peer review

• Mentor

• Stakeholders

Careers in Software Engineering

Job Outlook for Software Engineers

Employment options:

• Employed roles:
  • Apprenticeship/internship
  • Part-time
  • Full-time
• Self-employed/independent:
  • Contracting/consulting
  • Freelancing
• Volunteer on open source projects

Career Paths in Software Engineering

Technical

• Coding and problem-solving

  Management

• Leadership and soft skills

Career progression:

• Junior or Associate Software Engineer
  • Develop small chunks of software
  • Supported by a team leader or mentor
  • Gain new skills and experience
• Software Engineer
  • Break tasks down into sub-tasks
  • Learn new languages
  • Understand the software development lifecycles
  • Mentor junior software engineers
• Senior Software Engineer
  • Work across a project
  • Mentor software engineers and review code
  • Solve problems efficiently
• Staff Software Engineer
  • Part of the technical team
  • Develop, maintain, and extend software
  • Ensure software meets expectations
  • Ensure software uses resources efficiently
• Technical Lead
  • Manage a team of developers and engineers
  • Responsible for development lifecycle
  • Report to stakeholders
• Principal Engineer/Technical Architect
  • Responsible for architecture and design
  • Create processes and procedures
• Engineering Manager
  • Support team
  • Encourage career progression
• Director of Engineering
  • Strategic and technical role
  • Determine project priority
  • Identify hiring needs
  • Define goals
  • Define new projects
  • Specify requirements
• Chief Technology Officer (CTO)
  • Oversee research and development
  • Monitor corporate technology
  • Evaluate new technology and products

Other career directions

• Prefer interacting with clients:
  • Technical sales
  • Customer support
• Prefer working with numbers and data:
  • Data engineering
  • Data science
  • Database administration
  • Database development
• Prefer finding and fixing bugs:
  • Software testing

Software Engineering Job Titles

Job Titles:

• Front-end engineer

• Back-end engineer

• Full-stack engineer

• DevOps engineer

• Software Quality Assurance Engineer

• Software Integration Engineer

• Software Security Engineer

• Mobile App Developer

• Games Developer

Code of Ethics

Origins of the code of ethics:

• Developed by the Joint Task Force on Software Engineering Ethics and Professional Practices
  • Institute of Electrical and Electronics Engineers Computer Society (IEEE-CS)
  • Association for Computing Machinery (ACM)
  • Championed the need to hold software engineers accountable

About the code of ethics:

• Pertains to the analysis, design, development, testing, and maintenance software cycle
• Dedicated to serving the public interest

The 8 principles

1. Public

2. Client/Employer

3. Product

4. Judgement

5. Management

6. Profession

7. Colleagues

8. Self

Supplemental guide to behavior

• Use in conjunction with conscientious decision-making and common sense
• Knowing where to apply principles is at the discretion and wisdom of the individual

Introduction to Linux

Introducing Linux and Unix

What is Unix?

• Unix is a family of operating systems
• Popular Unix-based OSs include:
  • Oracle Solaris (and Open Solaris)
  • FreeBSD
  • HP-UX
  • IBM AIX
  • Apple macOS

UNIX beginnings:

Linux’s beginnings:

Linux, use cases today:

• Android
• Supercomputers
• Data Centers and Cloud Servers
• PCs

Overview of Linux Architecture

Kernel

• Lowest-level software in system
• Starts on boot and remains in memory
• Bridge between apps and hardware
• Key jobs:
  • Memory management
  • Process management
  • Device drivers
  • System calls and security

Linux filesystem:

Linux Terminal Overview

Communicating with Linux System:

Introduction to Linux Commands

Overview of Common Linux Shell Commands

What is a shell?

• User interface for running commands

• Interactive language

• Scripting language

  Shell command applications:

• Getting information

  • whoami – username
  • id – user ID and group ID
  • uname – operating system name
  • ps – running processes
  • top – resource usage
  • df – mounted file systems
  • man – reference manual
  • date – today’s date

• Navigating and working with files and directories

• Printing file and string contents

• Compression and archiving

  • tar – archive a set of files
  • zip – compress a set of files
  • unzip – extract files from a compressed zip archive

• Performing network operations

  • hostname – print hostname
  • ping – send packets to URL and print response
  • ifconfig – display or configure system network interfaces
  • curl – display contents of file at a URL
  • wget – download file from a URL

• Monitoring performance and status

Customizing View of File Content

• sort — Sort lines in a file, -r will do the same in the reverse
• uniq — Filter out repeated lines
• grep (“global regular expression print”) — Return lines in file matching pattern
• grep -i — makes grep search case-insensitive
• cut — Extracts a section from each line
• paste — Merge lines from different files

Introduction to Shell Scripting

Shell Scripting Basics

What is a script?

• Script: list of commands interpreted by a scripting language

• Commands can be entered interactively or listed in a text file

• Scripting languages are interpreted at runtime

• Scripting is slower to run, but faster to develop

  What is a script used for?

• Widely used to automate processes

• ETL jobs, file backups and archiving, system admin

• Used for application integration, plug-in development, web apps, and many other tasks

  Shell scripts and the ‘shebang’

• Shell script – executable text file with an interpreter directive

• Aka ‘shebang’ directive

  #!interpreter [optional-arg]

• ‘interpreter’ – path to an executable program

• ‘optional-arg’ – single argument string

Example – ‘shebang’ directives

Shell script directive:

#!/bin/sh
#!/bin/bash

Python script directive:

#!/usr/bin/env python3

Filters, Pipes, and Variables

Pipes and filters:

Filters are shell commands, which:

• Take input from standard input

• Send output to standard output

• Transform input data into output data

• Examples are wc, cat, more, head, sort, …

• Filters can be chained together

  Pipe command – |

• For chaining filter commands

  commmand1 | command2

• Output of command 1 is input of command 2

• Pipe stands for pipeline

Shell variables:

• Scope limited to shell

• Set – list all shell variables

  Defining shell variables:

  var_name=value

• No spaces around =

  unset var_name

• deletes var_name

Environment Variables:

• Extended scope

  export var_name

• env — list all environment variables

Useful Features of the Bash Shell

Metacharacters

• # — precedes a comment
• ; — command separator
• * — filename expansion wildcard
• ? — single character wildcard in filename expansion

Quoting

• \ — escape special character interpretation
• "" — interpret literally, but evaluate meta-characters
• '' — interpret literally

I/O redirection

Input/Output, or I/O redirection, refers to a set of features used for redirecting

• > — Redirect output to file
• >> — Append output to a file
• 2> — Redirect standard error to a file
• 2>> — Append standard error to a file
• < — Redirect file contents to standard input

Command substitution

• Replace command with its output

  $(command) or `command`

• Store output of pwd command in here:

Command line arguments

• Program arguments specified on the command line

• A way to pass arguments to a shell script

  ./MyBashScript.sh arg1 arg2
  

Batch vs. concurrent modes

Bath mode:

• Commands run sequentially

  command1; command2

  Concurrent mode:

• Commands run in parallel

  command1 & command2

Scheduling Jobs using Cron

Job scheduling

• Schedule jobs to run automatically at certain times

  • Load script at midnight every night
  • Backup script to run every Sunday at 2 AM

• Cron allows you to automate such tasks

  What are cron, crond, and crontab?

• Cron is a service that runs jobs

• Crond interprets ‘crontab files’ and submits jobs to cron

• A crontab is a table of jobs and schedule data

• Crontab command invokes text editor to edit a crontab file

Scheduling cron jobs with crontab

Viewing and Removing cron jobs

Git and GitHub Fundamentals

Overview of Git/GitHub

Working without Version Control:

Working with Version Control:

Git

• Free and open source software
• Distributed VCS
• Accessible anywhere in the world
• One of the most common version control systems available
• Can also version control images, documents, etc.

Short Glossary of Terms

SSH protocol – A method for secure remote login from one computer to another.

Repository – The folders of your projects that are set up for version control.

Fork – The process you use to request that someone reviews and approves your changes before they become final.

Working directory – A directory on your file system, including its files and subdirectories, that is associated with a git repository.

Introduction to GitHub

Background of Git:

• Large software projects need a way to track and control source code updates
• Linux needs automated source-version control
• Key characteristics include:
  • Strong support for non-linear development
  • Distributed development
  • Compatibility with existing systems and protocols
  • Efficient handling of large projects
  • Cryptographic authentication of history
  • Pluggable merge strategies

Git Repository Model:

• What is special about the Git Repository model?

  • Distributed VCS
  • Tracks source code
  • Coordinates among programmers
    • Tracks changes
    • Supports non-linear workflows

• Created in 2005 by Linus Torvalds

  What is Git?

• Git is a distributed VCS

  • Tracks changes to content
  • Provides a central point for collaboration

• Git allows for centralized administration

  • Teams have controlled access scope
  • The main branch should always correspond to deployable code

• IBM Cloud is built around open-source tools including Git repositories

GitHub

• GitHub is an online hosting service for Git repositories
  • Hosted by a subsidiary of Microsoft
  • Offers free, professional, and enterprise accounts
  • Till 2022, it had over 100M repos
  • What is a Repository?
    • A data structure for storing documents including application source code
    • A repository can track and maintain version-control

GitLab

GitLab is:

• A DevOps platform, delivered as a single application

• Provides access to Git repositories

• Provides source code management

  GitLab enables developers to:

• Collaborate

• Work from a local copy

• Branch and merge code

• Streamline testing and delivery with CI/CD

Using Git Commands and Managing GitHub Projects

GitHub Workflows with Branches and Git Commands

GitHub Branches

What are branches?

• Branches store all files in GitHub
• The master branch stores the deployable code
• Create a new branch for planned changes

Merging Branches:

• Start with a common base
• The code is branched while new features are developed
• Both branches are undergoing changes
• When the two streams of work are ready to merge, each branch’s code is identified as a tip and the two tips are merged into a third, combined branch

What is a Pull Request?

• A PR makes the proposed (committed) changes available for others to review and use

• A pull can follow any commits, even if code is unfinished

• PRs can target specific users

• GitHub automatically makes a PR if you make a change on a branch you don’t own

• Log files record the approval of the merge

  Merging into the Master/Main Branch

• The master branch should be the only deployed code

• Developers can change source files in a branch, but the changes are not released until

  • They are committed
  • A Pull command is issued
  • The code is reviewed and approved
  • The approved code is merged back into the master code

Cloning and Forking GitHub Projects

• Powerful tools include forking and cloning a repository

• Cloning creates a copy of a repository on your local machine

• Cloned copies can sync between locations

• Forking modifies or extends a project

  Remote Repositories:

• Remote repos are stored elsewhere

• Push, pull, and fetch data to share work

• Origin refers to your fork

• Upstream refers to the original work

Forking a Project

• Forking
  • Takes a copy of a GitHub repository to use it as the base for a new project
  • Submit changes back to the original repository
• Independently make changes to a project
  • Submit a PR to the original project owner
  • Owner decides whether to accept updates
• Keep a copy of the license file
  • Often a legal requirement
  • Good practice

Syncing a Fork of a Project:

To keep a fork in sync with the original work from a local clone:

• Create a local clone of the project

• Configure Git to sync the fork

  • Open terminal and change to the directory containing the clone
  • To access the remote repository, type git remote -v
  • Type git remote add upstream <clone repo url>
  • To see the changes, type git remote -v

  Commands for Managing Forks:

  To grab upstream branches

git fetch upstream

To merge changes into the master branch

git merge upstream/master

Managing GitHub Projects

GitHub Developer:

A Developer communicates with others using these commands:

• git-clone from the upstream to prime the local repository
• git-pull and git-fetch from “origin” to keep-up-to-date with the upstream
• git-push to shared repository, if you adopt CVS style shared repository workflow
• git-format-patch to prepare email submission
• git-send-email to send your email submission without corruption by your MUA (Mail User Agent)
• git-request-pull to create a summary of changes for your upstream to pull

GitHub Integrator:

An integrator

• Receives changes made by others

• Reviews and responds to PRs

• Publishes the result for others to use

  Integrators use the following commands:

• git-am to apply patches emailed in from your contributors

• git-pull to merge from your trusted lieutenants

• git-format-patch to prepare and send suggested alternatives to contributors

• git-revert to undo botched commits

• git-push to publish the bleeding edge

GitHub Repository Administrator

A Repository Administrator sets up and maintains access to the repository by developers

• git-daemon to allow anonymous download from repository
• git-shell can be used as a restricted login shell for shared central repository users
• git-http-backend provides a server-side implementation of Git-over-HTTP (Smart HTTP) allowing both fetch and push services
• gitweb provides a web front-end to Git repositories, which can be set-up using the git-instaweb script

Python Programming Fundamentals

Logic Operators: OR

Logic Operators: AND

Python Data Structures

Tuples

• Tuples are an ordered sequence

• Here is a Tuple “Ratings”

• Tuples are written as comma-separated elements within parentheses

• Tuples concatenation is possible

• Tuple slicing is also possible

• Tuples are immutable

• If one want to manipulate tuples, they have to create a new tuple with the desired values

• Tuples nesting (tuple containing another tuple) is also possible

  Ratings = (10, 9, 6, 5, 10, 8, 9, 6, 2)

  

  

  

  

  

  

Lists

• Lists are also ordered in sequence

• Here is a List “L”

  L = ["Michael Jackson", 10.1, 1982]

• A List is represented with square brackets

• List is mutable

• List can nest other lists and tuples

• We can combine lists

• List can be extended with extend() method

• append() adds only one element to the List, if we append L.append([1,2,3,4]), the List “L” will be:

  L = ["Michael Jackson", 10.1, 1982,[1,2,3,4]]

• The method split() can convert the string into the List

  "Hello, World!".split()

• The split() can be used with a delimiter we would like to split on as an argument

  "A,B,C,D".split(",")

• Multiple names referring to the same object is known as aliasing

• We can clone the list, where both lists will be of their independent copies
• So changing List “A”, will not change List “B”

Dictionaries

• Dictionaries are denoted with curly Brackets {}
• The keys have to be immutable and unique
• The values can be immutable, mutable and duplicates
• Each key and value pair is separated by a comma

Sets

• Sets are a type of collection
  • This means that like lists and tuples you can input different python types
• Unlike lists and tuples they are unordered
  • This means sets don’t record element position
• Sets only have unique elements
  • This means there is only one of a particular element in a set

Sets: Creating a Set

• You can convert a list into set

  List = ['foo']
  set(List)

• To add elements to the set, set.add('foo')

• To remove an element, set.remove(‘foo’)

• To check if an element is present in the set:

  'foo' in set
  True/False

Sets: Mathematical Expression

• To find the intersection of the sets elements present in the both sets), set1 & set2 or set1.intersection(set2)

• Union of the sets, contain elements of both the sets combined, set1.union(set2)

• To find the difference of sets:

  #set1 difference from set2
  set1.difference(set2)
  #set2 difference from set 1
  set2.difference(set1)

• To find is a set is a subset/superset (have all the elements of other set), `set1.issubset/issuperset(set2)