GAO-20-213, AGILE SOFTWARE DEVELOPMENT: DHS Has Made Significant Progress in Implementing Leading Practices, but Needs to take Additional Actions

AGILE SOFTWARE

DEVELOPMENT

DHS Has Made

Significant Progress in

Implementing Leading

Practices, but Needs

to Take Additional

Actions

Report to Congressional Requesters

June 2020

GAO-20-213

United States Government Accountability Office

______________________________________ United States Government Accountability Office

June 2020

AGILE SOFTWARE DEVELOPMENT

DHS Has Made

Significant Progress in Implementing

Leading Practices, but

Needs to Take Additional

Action

What GAO Found

The Department of Homeland Security (DHS) has taken steps to implement

selected leading practices in its transition from waterfall, an approach that

historically delivered useable software years after program initiation, to Agile

software development, which is focused on incremental and rapid delivery of

working software in small segments. As shown below, this quick, iterative

approach is to deliver results faster and collect user feedback continuously.

Comparison of Agile and Waterfall Methods for Developing Software

DHS has fully addressed one of three leading practice areas for organization

change management and partially addressed the other two. Collectively, these

practices advise an organization to plan for, implement, and measure the impact

when undertaking a significant change. The department has fully defined plans

for transitioning to Agile development. DHS has partially addressed

implementation—the department completed 134 activities but deferred roughly

34 percent of planned activities to a later date. These deferred activities are in

progress or have not been started. With respect to the third practice, DHS

clarified expected outcomes for the transition, such as reduced risk of large,

expensive IT failures. However, these outcomes are not tied to target measures.

Without these, DHS will not know if the transition is achieving its desired results.

DHS has also addressed four of the nine leading practices for adopting Agile

software development. For example, the department has modified its acquisition

policies to support Agile development methods. However, it needs to take

additional steps to, among other things, ensure all staff are appropriately trained

and establish expectations for tracking software code quality. By fully addressing

leading practices, DHS can reduce the risk of continued problems in developing

and acquiring current, as well as, future IT systems.

Why GAO Did This Study

Many of DHS’s major IT acquisition

programs have taken longer than

expected to develop or failed to deliver

the desired value. In April 2016, to help

improve the department’s IT acquisition

and management, DHS identified Agile

software development as the preferred

approach for all of its IT programs and

projects.

GAO was asked to examine DHS’s

adoption of Agile software development.

The objective of this review was to

assess the extent to which DHS has

addressed selected leading practices for

its transition to the use of Agile software

development.

GAO identified leading practices for

planning, implementing, and measuring

organizational change that apply to

DHS’s transition to Agile through its

review of guidance published by the

Project Management Institute and GAO.

GAO also reviewed work it performed to

develop leading practices for Agile

software development adoption. GAO

analyzed DHS documentation, such as

policies, guidance, plans, and working

group artifacts and assessed them

against the selected leading practices.

GAO also reviewed the implementation

of selected practices within individual IT

projects. Finally, GAO interviewed DHS

officials to discuss any practices that

were not fully implemented.

What GAO Recommends

GAO is making 10 recommendations to

DHS to implement selected leading

practices for its transition to Agile

software development. DHS agreed with

GAO’s recommendations and described

actions taken and planned to address

them.

View GAO-20-213. For more information,

contact

Carol C. Harris at (202) 512-4456 or

harriscc

@gao.gov.

Highlights of GAO-20-213, a report to

congressional requesters

Page i GAO-20-213 Agile Software Development

Letter 1

Background 4

DHS Has Made Progress in Implementing Leading Practices, but

Has Not Fully Addressed Others 13

Conclusions 26

Recommendations for Executive Action 27

Agency Comments and Our Evaluation 28

Appendix I Objective, Scope, and Methodology 32

Appendix II DHS Agile Action Plans 39

Appendix III Leading Practices for Adopting Agile Development—Agency

Environment 50

Appendix IV Leading Practices for Adopting Agile Development—Program

Processes 67

Appendix V Leading Practices for Adopting Agile Development—Team

Activities and Dynamics 84

Appendix VI Comments from the Department of Homeland Security 103

Appendix VII GAO Contact and Staff Acknowledgments 107

Tables

Table 1: The Department of Homeland Security (DHS) Roles and

Responsibilities for Agile Development 10

Contents

Page ii GAO-20-213 Agile Software Development

Table 2: Levels of Agile Adoption and Leading Practices

Associated with Each Level 20

Table 3: DHS Agile Action Plan Title, Goal, Lead Organizations,

Level of Difficulty and Impact, and Executive Priority 40

Table 4: DHS Agile Action Plan Problem Statements and

Recommendations 43

Figures

Figure 1: Comparison of Agile and Waterfall Methods for

Developing Software 8

Figure 2: DHS Acquisition Life Cycle Framework (ALF) and

Systems Engineering Life Cycle (SELC) for Major

Acquisition Programs 52

Figure 3: Optional Tailoring of the DHS Acquisition Life Cycle

Framework (ALF) and Systems Engineering Life Cycle

(SELC) for Agile Major Acquisition Program 54

Abbreviations

ALF acquisition life cycle framework

BEE Biometric Entry Exit

C4ISR Command, Control, Communications, and Computers,

Intelligence, Surveillance and Reconnaissance

CBP U.S. Customs and Border Protection

CDR critical design review

CIO chief information officer

DHS Department of Homeland Security

EA enterprise architecture

EAB Enterprise Architecture Board

FITARA Federal Information Technology Acquisition Reform Act

ICE U.S. Immigration and Customs Enforcement

IRR integration readiness review

IT information technology

ITPM COE IT Program Management Center of Excellence

ITR initial technical review

JRC Joint Requirements Council

OCIO Office of the Chief Information Officer

OCTO Office of the Chief Technology Officer

Page iii GAO-20-213 Agile Software Development

ORR operational readiness review

OTRR operational test readiness review

PARM Office of Program Accountability and Risk Management

PDR preliminary design review

PIR post implementation review

PPR project planning review

PRR production readiness review

RCR release cycle review

RPR release planning review

SAR solution analysis review

SDR system definition review

SELC systems engineering life cycle

SEVIS Student and Exchange Visitor Information System

SPDR software preliminary design review

SPR study plan review

STM Strategic Technology Management

USCG U.S. Coast Guard

This is a work of the U.S. government and is not subject to copyright protection in the

United States. The published product may be reproduced and distributed in its entirety

without further permission from GAO. However, because this work may contain

copyrighted images or other material, permission from the copyright holder may be

necessary if you wish to reproduce this material separately.

Page 1 GAO-20-213 Agile Software Development

441 G St. N.W.

Washington, DC 20548

June 1, 2020

Congressional Requesters

The Department of Homeland Security (DHS) and its components invest

billions of dollars each year to acquire information technology (IT) and

other capabilities to support the department’s critical functions. However,

as we have previously reported, many of the department’s major IT

acquisition programs have taken longer than expected to develop and

implement, or have failed to deliver the desired value to mission

operations.

As part of an effort to improve its IT acquisition and

management, in April 2016, the department identified Agile software

development as its preferred approach for all DHS IT programs and

projects. Such an approach—one form of incremental development—calls

for the rapid delivery of software in small, short increments.

You asked us to examine the department’s adoption of Agile software

development. Our specific objective was to assess the extent to which

DHS has addressed selected leading practices for its transition to the use

of Agile software development. To accomplish this objective, we

assessed the extent to which the department adhered to leading practices

See, for example, Homeland Security Acquisitions: Outcomes Have Improved but

Actions Needed to Enhance Oversight of Schedule Goals, GAO-20-170SP (Washington,

D.C.: Dec. 19, 2019); FEMA Grants Modernization: Improvements Needed to Strengthen

Program Management and Cybersecurity, GAO-19-164 (Washington, D.C.: Apr. 9, 2019);

U.S. Secret Service: Action Needed to Address Gaps in IT Workforce Planning and

Management Practices, GAO-19-60 (Washington, D.C.: Nov. 15, 2018); TSA

Modernization: Use of Sound Program Management and Oversight Practices Is Needed to

Avoid Repeating Past Problems, GAO-18-46 (Washington, D.C.: Oct. 17, 2017);

Homeland Security: Progress Made to Implement IT Reform, but Additional Chief

Information Officer Involvement Needed, GAO-17-284 (Washington, D.C.: May 18, 2017);

Immigration Benefits System: U.S. Citizenship and Immigration Services Can Improve

Program Management, GAO-16-467 (Washington, D.C.: July 7, 2016); Information

Technology: FEMA Needs to Address Management Weaknesses to Improve Its Systems,

GAO-16-306 (Washington, D.C.: Apr. 5, 2016); Homeland Security: Oversight of

Neglected Human Resources Information Technology Investment Is Needed,

GAO-16-253 (Washington, D.C.: Feb. 11, 2016); Immigration Benefits System: Better

Informed Decision Making Needed on Transformation Program, GAO-15-415

(Washington, D.C.: May 18, 2015); Border Security: DHS’s Efforts to Modernize Key

Enforcement Systems Could be Strengthened, GAO-14-62 (Washington, D.C.: Dec. 5,

2013); Information Technology: DHS Needs to Enhance Management of Major

Investments, GAO-13-478T (Washington, D.C.: Mar. 19, 2013); Information Technology:

DHS Needs to Enhance Management of Cost and Schedule for Major Investments,

GAO-12-904 (Washington, D.C.: Sept. 26, 2012).

Letter

Page 2 GAO-20-213 Agile Software Development

in two specific areas: organizational change management and Agile

software development adoption.

With regard to organizational change management, we reviewed leading

practices published by the Project Management Institute and GAO on

organizational change management.

Based on this review, we identified

fifteen leading practices. We then grouped these 15 practices into three

broad organizational change management areas: planning, implementing,

and measuring change.

To determine the extent to which DHS addressed leading practice areas

for organizational change management in its transition to Agile

development, we assessed DHS policies, procedures, guidance, plans,

and other working group artifacts and compared them against the leading

practices in the three areas. Our review also included analyzing DHS’s IT

Program Management Center of Excellence (ITPM COE) meeting

minutes, presentation slides, and status update charts. Further, we

interviewed officials from DHS headquarters lines of business to discuss

any practices in the three areas that were not fully addressed.

Specifically, we interviewed officials from the offices of the Chief

Procurement Officer, Chief Readiness Support Officer, Chief Financial

Officer, Chief Human Capital Officer, Chief Security Officer, the Chief

Information Officer (OCIO), Systems Engineer, and Test and Evaluation,

and the Joint Requirements Council.

With regard to leading practices for Agile software development adoption,

we reviewed work performed by GAO to develop generally accepted

leading practices. In developing these leading practices, GAO reviewed

information from a variety of sources related to Agile adoption and

compiled a draft of leading practices commonly mentioned across these

Project Management Institute, Inc., Managing Change in Organizations: A Practice

Guide, (Newtown Square, PA: 2013); GAO, Government Reorganization: Key Questions

to Assess Agency Reform Efforts, GAO-18-427 (Washington, D.C.: Jul 13, 2018); IT

Workforce: Key Practices Help Ensure Strong Integrated Program Teams; Selected

Departments Need to Assess Skill Gaps, GAO-17-8 (Washington, D.C.: Nov. 30, 2016);

Standards for Internal Control in the Federal Government, GAO-14-704G (Washington,

D.C.: Sept. 10, 2014).

Department of Homeland Security, Instruction 102-01-004. DHS defines the line of

business chiefs as officers at the department level with a set of one or more highly related

services (administrative, financial management, human resources, information technology,

procurement, and security), which include the Chief Procurement Officer, Chief Readiness

Support Officer, Chief Financial Officer, Chief Human Capital Officer, Chief Security

Officer, and the Chief Information Officer.

Page 3 GAO-20-213 Agile Software Development

different sources. We then convened a working group of experts from the

public and private sectors and academia. This working group met three

times a year between August 2016 and August 2019 to review and

discuss these leading practices. More than 200 experts participated in the

meetings, including more than 20 officials from DHS. GAO received

comments from many of these experts both during these meetings and by

email after the meetings.

Based on this work, GAO developed a set of nine leading practices for

Agile adoption. The leading practices were described by a series of core

elements and core element expectations that, collectively, can be used to

assess the status of an agency’s implementation.

To determine the extent to which the department had addressed the

leading practices for the adoption of Agile development, we obtained and

assessed DHS policies, procedures, guidance, plans, and other

documentation such as systems engineering life cycle (SELC) technical

review completion letters, and compared them against the nine leading

practices. This included supplementary Agile documentation, such as

training materials prepared by the Homeland Security Acquisition Institute

for acquisition workforce certifications and webinars offered by the

Procurement Innovation Lab. We also interviewed department officials

responsible for the associated policies, procedures, guidance, plans, and

other documentation to discuss any practices that were not fully

implemented.

To supplement our assessment of the extent to which the department

addressed program process, and team activity and dynamics-level

leading practices, we also assessed selected projects’ implementation of

these practices. We selected only the projects supporting programs on

the Major Acquisition Oversight List because DHS expects these

programs to comply with its Agile instruction and acquisition management

policy. We then further limited the scope of projects to those within

components where GAO had not previously assessed a program using

Agile methods or was not in the process of assessing such a program.

We further refined our selection based on the following criteria: software

development life cycle methodology (iterative development only) and

project completion date (in-progress only).

This excluded the U.S. Citizenship and Immigration Services, Federal Emergency

Management Agency, Transportation Security Administration, and U.S. Secret Service.

Page 4 GAO-20-213 Agile Software Development

We then selected a random sample of three projects, with no more than

one project selected from a component. The three case study projects we

selected were the 1) U.S. Coast Guard (USCG) Command, Control,

Communications, and Computers, Intelligence, Surveillance and

Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol

Cutter project, with particular attention to the SeaWatch portion of this

project; 2) the U.S. Customs and Border Protection (CBP) Biometric Entry

Exit (BEE) program Air Exit project, with particular attention to the

Traveler Verification Services portion of this project; and 3) the U.S.

Immigration and Customs Enforcement (ICE) Student and Exchange

Visitor Information System (SEVIS) program 8001 project, with particular

attention to the SEVIS modernization portion of this effort.

To evaluate case studies’ implementation of selected leading practices,

we reviewed artifacts from the selected projects. In particular, we

reviewed artifacts demonstrating a project’s use of Agile including testing

metrics, evidence of Agile meetings, the existence of user stories and a

backlog, and the availability of Agile coaching and training. We then

interviewed officials responsible for program and project management

and representatives of groups responsible for software development for

the three selected case study projects to discuss gaps we identified. See

appendix I for a more detailed discussion of our objective, scope, and

methodology.

We conducted this performance audit from December 2017 to April 2020

in accordance with generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain

sufficient, appropriate evidence to provide a reasonable basis for our

findings and conclusions based on our audit objectives. We believe that

the evidence obtained provides a reasonable basis for our findings and

conclusions based on our audit objectives.

DHS and its components invest billions of dollars each year to acquire IT

and other capabilities to support the department’s critical functions. The

Background

Page 5 GAO-20-213 Agile Software Development

department plans to spend approximately $2.3 billion on major IT

investments in fiscal year 2020.

However, DHS has faced long-standing challenges in acquiring and

managing IT.

We have highlighted the department’s IT management

issues on our high-risk list since 2003 and have made numerous

recommendations to improve its IT management practices.

For example,

in 2013, we testified that, out of 68 major IT investments that the

department had in development, 21 had one or more subsidiary projects

that were not meeting cost and/or schedule commitments due to technical

issues in the development phase, changes in agency priorities, or a lack

of understanding of user requirements, among other things.

Many federal agencies, including DHS, are accustomed to using a

waterfall software development model. This type of model typically

consists of long, sequential phases, resulting in product delivery years

after program initiation. With many federal IT investments in a

development phase, it is important to ensure that agencies are making

According to data that DHS reported to the Office of Management and Budget’s Federal

IT Dashboard in June 2019, the department planned to spend approximately $2.3 billion

across 40 major IT investments in fiscal year 2020. (A major program is defined by DHS

as one with a life cycle estimate of $300 million or more.) This included costs associated

with developing, modernizing, enhancing, and operating and maintaining IT investments.

According to Office of Management and Budget guidance, an IT investment may include a

project or projects for the development, modernization, enhancement, or maintenance of a

single IT asset or group of IT assets with related functionality, and the subsequent

operation of those assets in a production environment.

GAO, Information Technology: Customs Automated Commercial Environment Program

Progressing, but Need for Management Improvements Continues, GAO-05-267

(Washington, D.C.: Mar. 14, 2005), Border Security: US-VISIT Program Faces Strategic,

Operational, and Technological Challenges at Land Ports of Entry, GAO-07-248

(Washington, D.C.: Dec. 6, 2006), Secure Border Initiative: DHS Needs to Address

Significant Risks in Delivering Key Technology Investment, GAO-08-1086 (Washington,

D.C.: Sept. 22, 2008), Department of Homeland Security, Transportation Security

Administration: Secure Flight Program, GAO-09-169R (Washington, D.C.: Nov. 10, 2008),

Border Security: DHS’s Efforts to Modernize Key Enforcement Systems Could be

Strengthened, GAO-14-62 (Washington, D.C.: Dec. 5, 2013), Homeland Security:

Oversight of Neglected Human Resources Information Technology Investment Is Needed,

GAO-16-253 (Washington, D.C.: Feb. 11, 2016).

See, for example, GAO, High-Risk Series: An Update, GAO-03-119 (Washington, D.C.:

Jan. 1, 2003) and High-Risk Series: Substantial Efforts Needed to Achieve Greater

Progress on High-Risk Areas, GAO-19-157SP (Washington, D.C.: Mar. 6, 2019).

GAO, Information Technology: DHS Needs to Enhance Management of Major

Investments, GAO-13-478T (Washington, D.C.: Mar. 19, 2013).

Overview of Incremental

and Agile Software

Development

Page 6 GAO-20-213 Agile Software Development

the most efficient use of their financial resources through effective

management practices. However, as we have previously reported and

testified, historically federal IT projects often fail—that is, even after

exceeding their budgets by millions of dollars and delaying the schedules

by years—and the results do not meet requirements.

Recognizing the severity of challenges related to the government-wide

management of IT, in December 2014, federal IT acquisition reform

provisions (commonly referred to as FITARA) were enacted as a part of

the Carl Levin and Howard P. ”Buck” McKeon National Defense

Authorization Act for Fiscal Year 2015.

One of the provisions requires

that the Office of Management and Budget (OMB) require in its annual IT

capital planning guidance that each covered agency’s chief information

officer (CIO) certify that IT investments are adequately implementing

incremental development,

as defined in capital planning guidance

issued by OMB.

Agile software development—one form of incremental development—

calls for the rapid delivery of software. Probably the most well-known

feature of Agile software development is iterative product development

and delivery; that is, development of software in segments that are

continuously evaluated against requirements. This method is well suited

for programs in which the final product is to include distinct features,

some of which may be discovered during the process rather than planned

at the beginning. These frequent iterations can effectively measure

progress and allow developers to respond quickly to feedback from

customers, thus reducing technical and programmatic risk. With its

GAO, Information Technology: Implementation of IT Reform Law and Related Initiatives

Can Help Improve Acquisitions, GAO-17-494T (Washington, D.C.: Mar. 28, 2017) and

Information Technology: Additional Actions and Oversight Urgently Needed to Reduce

Waste and Improve Performance in Acquisitions and Operations, GAO-15-675T

(Washington, D.C.: June 10, 2015).

Carl Levin and Howard P. ”Buck” McKeon National Defense Authorization Act for Fiscal

Year 2015, Pub. L. No. 113-291, division A, title VIII, subtitle D, 128 Stat. 3292, 3438-50

(Dec. 19, 2014).

Incremental or modular development is where an investment may be broken down into

discrete projects, increments, or useful segments, each of which are undertaken to

develop and implement the products and capabilities that the larger investment must

deliver. Dividing investments into smaller parts helps to reduce investment risk, deliver

capabilities more rapidly, and permit easier adoption of newer and emerging technologies.

Pub. L. No. 113-291, § 831 as codified at 40 U.S.C. § 11319(b)(1)(B)(ii).

Page 7 GAO-20-213 Agile Software Development

emphasis on early and continuous delivery of working software, Agile can

be a valuable tool for agencies in mitigating schedule and budget risks.

Figure 1 compares requirements, design, development, and testing using

Agile software methods versus a traditional waterfall approach; illustrating

how requirements, design, development, and testing are performed

concurrently in smaller time-boxed iterations for Agile and sequentially in

waterfall development. As a result, using an Agile framework should

result in producing high-quality software with frequent reviews and

customer feedback to ensure that the highest value requirements are

being met. The figure assumes that planning for both Agile and waterfall

development has already occurred.

Page 8 GAO-20-213 Agile Software Development

Figure 1: Comparison of Agile and Waterfall Methods for Developing Software

Page 9 GAO-20-213 Agile Software Development

In February 2016, the DHS Under Secretary for Management announced

an effort to pilot the use of Agile development methodologies to improve

the department’s execution and oversight of IT acquisitions.

This

resulted in five Agile pilot programs. Each pilot program was overseen by

a component integrated program team. Collectively, the first pilot

programs were also overseen and supported by a DHS integrated

program team. In April 2016, the department issued an Agile instruction,

which identified Agile software development as the preferred approach for

all DHS programs and projects that are to deliver an IT, or embedded-IT,

capability.

The department also set an expectation for its component

CIOs to develop plans to increase the use of Agile development and

justify any major IT programs that did not intend to use Agile development

practices. Many DHS programs were already using Agile or similar

incremental development methods before the department identified it as

the preferred approach.

The DHS CIO, as the individual delegated departmentwide responsibility

for approving, managing, and overseeing all of the department’s IT

programs, sets the policies and procedures to help ensure Agile practices

meet the department’s goals and comply with acquisition management

policy. The DHS CIO is supported in this effort by the heads of other

major DHS lines of business, such as the Chief Procurement Officer.

Department of Homeland Security, Acquisition Decision Memorandum: Information

Technology Acquisitions Agile Pilots (Feb. 18, 2016). The memorandum signed by the

DHS Under Secretary for Management identifies practices, such as Lean and Agile

incremental development, as the preferred methods for acquiring and delivering DHS IT

capabilities. For the purposes of this report, we refer to Lean and Agile incremental

development as Agile development.

DHS defines a program as a group of related projects managed in a coordinated way to

obtain benefits and control not available from managing them individually. A project is

defined as a planned undertaking of something to be accomplished or produced, or an

undertaking having a finite beginning and finite end. A project is a temporary endeavor

undertaken to create a unique product, service, or result; it involves the definition,

acquisition, and fielding of a unique product, service, or result in accordance with specified

resources and requirements. For the purposes of this report, we will use the term

“program” to refer to a program or a project.

DHS defines a line of business chief as an “officer” at the department level with a set of

one or more highly related services (administrative, financial management, human

resources, information technology, procurement, and security). The lines of business

chiefs include the Chief Procurement Officer, Chief Readiness Support Officer, Chief

Financial Officer, Chief Human Capital Officer, Chief Security Officer, and the Chief

Information Officer.

DHS Adopted Agile

Software Development to

Address IT Challenges

Roles and Responsibilities

for Agile Programs

Page 10 GAO-20-213 Agile Software Development

Table 1 describes the roles and responsibilities that support Agile

development within the department.

Table 1: The Department of Homeland Security (DHS) Roles and Responsibilities for Agile Development

Role

Responsibility

Chief Information

Officer (CIO)

Sets the policies and procedures to ensure Agile development best practices are leveraged to meet the

department’s goals and are consistent with acquisition policy established by Directive 102-01.

Certifies that Department of Homeland Security (DHS) IT programs and projects are appropriately

implementing incremental software development.

Reviews IT investments to ensure they are appropriately tailoring and executing Agile methodologies within

the context of the specific programs and domains.

With the Chief Procurement Officer; Component Acquisition Executives; Science & Technology Directorate’s

Director, Office of Test and Evaluation; and component CIO, sets Agile outcomes and target measures;

monitors the progress of DHS in achieving Agile outcomes; and reports (as required) to Office of Management

and Budget and GAO on DHS attainment of outcome metrics and associated benefits.

Supported by the Chief Procurement Officer; Component Acquisition Executives; Science & Technology

Directorate’s Director, Office of Test and Evaluation, and the Office of Program Accountability and Risk

Management (PARM), provides guidance, training, and mentoring for the adoption and execution of Agile

development.

Chief Procurement

Officer

Supports DHS contracting organizations in implementing Office of Management and Budget guidance on

modular contracting.

Supported by the CIO, Component Acquisition Executives, and PARM, provides guidance, training, and

mentoring for adopting and executing modular contracting in support of modular development

programs/projects.

Supported by the CIO, sets modular contracting implementation metrics and reporting requirements.

Supported by the CIO, Component Acquisition Executives, and PARM, assesses training opportunities and

identifies appropriate Agile methodology training for acquisition professionals, including program/project

managers, test and evaluation personnel, system engineers, contracting officers, and logisticians.

Chief Financial Officer

As necessary, tailors Office of Management and Budget guidance regarding flexible budget and funding

models that support Agile development of IT acquisitions and distributes it to applicable parties within DHS.

Director, Office of

Test and Evaluation,

Science &

Technology

Directorate

Provides independent test and evaluation oversight for major acquisition programs, procurements, or capital

investments using approved development methodologies based on authority and responsibility as directed in

DHS Directive 026-06 and Delegation 10003.

Works with acquisition programs using Agile methodologies to develop integrated test and evaluation

strategies tailored to support Agile development in accordance with DHS test and evaluation policy.

Provides test and evaluation consultation to non-oversight acquisition programs dependent on available

Science & Technology Directorate’s Director, Office of Test and Evaluation staff resources.

Executive Director,

PARM

Supports the Chief Acquisition Officer in managing DHS-wide acquisition program policy, governance, and

oversight in accordance with Directive 102-01.

Source: Agile Development and Delivery for Information Technology | GAO-20-213

The Agile instruction also defines roles and responsibilities at the component level, including

component CIOs, chief financial officers, component acquisition executives, lead business and

technical authorities, and program and project managers.

Page 11 GAO-20-213 Agile Software Development

Additionally, DHS established a headquarters-level team—the ITPM

COE—to collaborate across the department on improvements to policy,

governance, and acquisition guidance. In April 2017, the ITPM COE

assumed responsibilities for the department’s transition to Agile

development. The Office of the Chief Technology Officer (OCTO)

Strategic Technology Management (STM) division within the OCIO

facilitates the ITPM COE and serves as the official liaison between other

OCIO divisions, other partner headquarters directorate and management

offices, and operational components as needed.

We have reported on various programmatic and technical challenges that

were limiting DHS’ efforts on Agile programs. For example,

• In 2016, we reported that the U.S. Citizenship and Immigration

Services Transformation program, which was using Agile software

development to modernize citizenship and immigration benefits

processing, needed to improve testing of its software code and ensure

its approaches to interoperability and end user testing met leading

practices.

We made 12 recommendations to improve

Transformation program management, including ensuring alignment

among policy, guidance, and leading practices in areas such as Agile

software development and systems integration and testing. DHS

concurred with the recommendations and has thus far implemented

eight of them.

• We reported in October 2017 that the Transportation Security

Administration Technology Infrastructure Modernization program had

not defined key roles and responsibilities, prioritized system

requirements, or implemented automated capabilities that were

essential to ensuring effective adoption of Agile.

We made 14

recommendations including that DHS should prioritize requirements

and obtain leadership consensus on oversight and governance

changes. DHS concurred with the recommendations and to date has

implemented 13 of them.

• In November 2018, we reported that the U.S. Secret Service OCIO

did not fully measure post-deployment user satisfaction with one

GAO, Immigration Benefits System: U.S. Citizenship and Immigration Services Can

Improve Program Management, GAO-16-467 (Washington, D.C.: July 7, 2016).

GAO, TSA Modernization: Use of Sound Program Management and Oversight Practices

Is Needed to Avoid Repeating Past Problems, GAO-18-46 (Washington, D.C.: Oct. 17,

2017).

GAO Previously Reported

on Challenges in DHS’

Management of Agile

Programs

Page 12 GAO-20-213 Agile Software Development

project supporting the Information Integration and Technology

Transformation investment.

We made 13 recommendations to the

U.S. Secret Service including that the Secret Service establish a

process that ensures the CIO reviews all IT contracts, as appropriate;

and identify the skills needed for its IT workforce. DHS concurred with

the recommendations but has not yet implemented them.

• We reported in April 2019 that the Federal Emergency Management

Agency Grants Management Modernization program had not yet fully

established plans for implementing new business processes or

established completed traceability of IT requirements.

We made

eight recommendations to implement leading practices related to

reengineering processes, managing requirements, scheduling, and

implementing cybersecurity. DHS concurred with the

recommendations and has thus far implemented two of them.

According to the Project Management Institute, the practice of change

management is a comprehensive, cyclic, structured approach for

transitioning individuals, groups, and organizations from a current state to

a future state with intended business benefits.

It helps organizations to

integrate and align people, processes, structures, culture, and strategy.

The Project Management Institute and GAO have both described leading

practices for effective organizational change management.

GAO, U.S. Secret Service: Action Needed to Address Gaps in IT Workforce Planning

and Management Practices, GAO-19-60 (Washington, D.C.: Nov. 15, 2018).

GAO, FEMA Grants Modernization: Improvements Needed to Strengthen Program

Management and Cybersecurity, GAO-19-164 (Washington, D.C.: Apr. 9, 2019).

Project Management Institute, Inc., Managing Change in Organizations: A Practice

Guide, (Newtown, Square, PA: 2013).

Project Management Institute, Managing Change in Organizations: A Practice Guide;

GAO, Government Reorganization: Key Questions to Assess Agency Reform Efforts,

GAO-18-427 (Washington, D.C.: Jul 13, 2018); IT Workforce: Key Practices Help Ensure

Strong Integrated Program Teams; Selected Departments Need to Assess Skill Gaps,

GAO-17-8 (Washington, D.C.: Nov. 30, 2017); Standards for Internal Control in the

Federal Government, GAO-14-704G (Washington, D.C.: Sept. 10, 2014).

Organizational Change

Management

Page 13 GAO-20-213 Agile Software Development

Leading practices in organizational change management advise an

agency to (1) plan for, (2) implement, and (3) measure the impact when

undertaking a significant change, such as a transition from one software

development approach to another.

Since DHS committed to its

transition to Agile software development in policy in April 2016, the

department has fully developed plans to facilitate the transition. However,

DHS has not fully implemented these plans and has experienced

challenges in measuring progress against its intended goals. In addition,

many of the plans are part of a larger effort to improve overall IT

acquisitions rather than specific to a transition to Agile development, an

approach that may delay DHS’s execution of these plans.

Leading practices for Agile software development adoption advise an

agency to focus on three organizational levels of adoption: (1) agency

environment, (2) program processes, and (3) team activities and

dynamics. DHS has partially adopted practices at all three organizational

levels. For example, the agency activities fully supported Agile methods

through actions such as developing policies and procedures that called

for the alignment of software, program goals, and agency goals.

However, the department’s culture can better support Agile methods by,

among other things, demonstrating an incentives and rewards structure to

incentivize Agile teams.

Planning for organizational change involves defining the activities that the

agency will need to undertake that integrate the planned changes with

business operations, identify areas where specific support is required,

and socialize practices to enable employees to make sense of what is

happening during the change. DHS fully implemented this practice area.

• Defining activities for integrating the planned changes with

business operations. DHS defined activities for integrating Agile

software development with business operations. Specifically, the

department began by compiling lessons learned based on Agile pilot

programs to inform its approach to adopting Agile development across

all of DHS. In June 2017, after completing the pilot effort, the Acting

Under Secretary for Management approved a set of 18 Agile action

plans to guide the transition to Agile software development.

Within

these actions plans, the department planned to update acquisition

Project Management Institute, Managing Change in Organizations: A Practice Guide;

GAO, GAO-18-427, GAO-17-8, GAO-14-704G.

The action plans are described further in appendix II.

DHS Has Made

Progress in

Implementing

Leading Practices,

but Has Not Fully

Addressed Others

DHS Fully Defined Planning

Activities for Transitioning to

Agile Development

Page 14 GAO-20-213 Agile Software Development

policy and guidance to ensure alignment with Agile software

development and supplement the updates with a series of artifacts

and templates.

• Defining activities for identifying areas where specific support is

required. DHS defined activities for identifying areas where specific

support is required. Within the Agile action plans, DHS planned to

meet with programs to determine their maturity levels and create an

Agile baseline for the department and develop and document a

strategy for prioritizing programs to receive support. Moreover, one

responsibility of the ITPM COE is to establish and institutionalize a

holistic IT program support intake function for identified risks and

issues to determine when assistance is warranted.

• Defining activities for socializing practices to enable employees

to make sense of what is happening during the change. DHS

defined activities for socializing practices to enable employees to

make sense of what is happening during the change. Within the Agile

action plans, the department intended to develop a generic

communications standard operating procedure for how information will

be socialized, shared, stored and received. DHS also planned to

socialize and post all templates and artifacts where components and

programs could access them.

Implementing organizational change involves executing the planned

activities and developing a human resource management plan based on

the skills, size, and availability of staff resources. DHS partially

implemented this practice area.

• Implementing the planned activities. DHS partially implemented the

planned activities for its transition to Agile software development.

According to officials from OCTO STM, contractor support staff

maintain a spreadsheet to track the status of each planned activity. As

of September 2019, of the 202 activities associated with the 18 Agile

action plans, 134 (approximately 66%) were complete, 30

(approximately 15%) were in progress, and 38 (approximately 19%)

had not been started.

ITPM COE officials stated that the activities of the ITPM COE, such as

completing the outstanding activities associated with the Agile action

plans, were sometimes delayed because the group’s activities were

not its members’ primary duties. This was reinforced in the fiscal year

2019 planning session, which identified the need for more of an

obligation from stakeholder engagement to ensure members attend

and assign themselves deliverables.

DHS Did Not Implement All of

the Defined Activities for

Transitioning to Agile

Development

Page 15 GAO-20-213 Agile Software Development

Nevertheless, DHS closed all of its aforementioned 18 Agile action

plans as implemented based on the near-term definition of done, as

agreed to by the Deputy Under Secretary for Management. As a result

of this decision, some of the activities were deferred until a later date.

According to the Director of STM, the Deputy Under Secretary for

Management concurred with this proposal, although DHS did not

provide supporting documentation to substantiate this statement.

The outstanding activities remain important to a successful transition

to Agile software development. For example, one Agile action plan

originally called for the department to publish updates to Agile policy,

procedures, and guidance. This included completing activities such as

updating the Agile instruction manual, as well as SELC guidance.

DHS closed this action plan as implemented because the department

had completed updating the acquisition management instruction and

associated instruction manual and sent it for executive approval.

However, according to the Director of STM and officials from PARM,

as of September 2019, these updates were not complete. Such

updates could help to further integrate Agile development with

business operations.

Until the department implements its planned activities for transitioning

to Agile, DHS risks increasing the chance that it will face challenges

that could adversely affect the transition to Agile.

• Developing a human resource management plan based on the

skills, size, and availability of staff resources. DHS initially

prepared a human resource management plan for completing the

Agile action plans. Following approval of the 18 action plans, the

Under Secretary for Management tasked a working group with

developing a schedule for executing the action plans. This included

estimating the number of staff required to complete each action plan,

how those staff positions would be filled, the time required to complete

each action plan, and any delays in other DHS initiatives that might

result from reprioritization. The working group determined that it would

need 88 full-time staff to complete the action plans as intended and

estimated completing all 18 action plans by October 2018. According

to the Director of STM, the Under Secretary for Management did not

approve the request for 88 full-time staff.

However, DHS did update the human resource management plan to

reflect ongoing fiscal year planning sessions and completing the

remaining Agile action plan activities. In September 2018, the ITPM

COE held a planning session to identify priorities for fiscal year 2019.

These priorities included completing some of the outstanding action

plan activities along with some new activities identified by members of

Page 16 GAO-20-213 Agile Software Development

the group. However, DHS did not demonstrate that the ongoing

planning sessions and the subsequent schedule for the upcoming

fiscal year were based on an assessment of the skills, team size, and

availability of the ITPM COE or other working groups supporting the

transition to Agile.

By assessing whether the upcoming plans for the ITPM COE are

realistic based on the skills needed to complete its planned

responsibilities associated with the transition to Agile and the

available resources, DHS can improve the likelihood of completing the

actions necessary to finish the transition of the department to Agile

development methods.

Measuring the impact of organizational change involves establishing the

need for the change, clarifying expected outcomes of the change that are

tied to target measures, measuring adoption rate, and measuring results

through its impact on the agency. DHS partially implemented this practice

area.

• Establishing the need for the change. DHS established the need

for its transition to Agile development. In the February 2016

Acquisition Decision Memorandum that approved five Agile pilots, the

DHS Under Secretary for Management stated that the department

needed the pilot programs because the department’s IT programs

were taking too long to develop and implement or were failing to

deliver the desired value to mission operations.

• Clarifying expected outcomes of the change that are tied to

target measures. The DHS Agile instruction required the DHS CIO

to, among other things, set Agile outcomes and target measures.

Consistent with leading practices and this requirement, DHS clarified

the expected outcomes of the transition to Agile development. In the

February 2016 Acquisition Decision Memorandum, the Under

Secretary for Management set a goal for the pilot programs to

improve the execution and oversight of DHS IT acquisitions using

industry best practices. In a white paper on streamlining DHS

acquisition and establishing a foundation for Agile program delivery,

DHS expanded on this goal and defined five outcomes for the

transition to Agile development:

1. Increased customer value: Deliver capabilities that are better

aligned with mission and user needs.

2. Reduced risk: Reduce probability of large, expensive failures.

DHS Could Not Measure the

Benefits of the Transition to

Agile

Page 17 GAO-20-213 Agile Software Development

3. Faster time-to-market: Deliver capabilities as quickly as possible

without sacrificing quality.

4. Increased accountability and oversight: Provide detailed,

continuous insight to progress and risks.

5. Economic value: Deliver capabilities as inexpensively as possible

without sacrificing quality.

However, the expected outcomes were not tied to target measures, as

required by the Agile instruction. The Director of STM stated that the

department did not initially define target measures for outcomes of its

transition because DHS was initially focused on adoption and

anticipated the need to refine the metrics over time based on lessons

learned. The Director added that there were challenges for some

programs in adopting Agile and target measures could have dis-

incentivized programs. The Director stated that this has been a

learning process and the department is now more comfortable

associating targets with the metrics, but did not provide a date for

doing so. By identifying target measures tied to its expected

outcomes, the department will be able to better determine whether the

transition is achieving its desired outcomes.

• Measuring adoption rate. DHS did not initially measure the Agile

adoption rate because it had not specifically identified projects that

were using Agile development. In 2018, the OCTO STM was focused

on measuring the adoption of incremental development in order to

comply with the requirements of the federal IT reform provisions

(commonly known as FITARA).

This effort included OCTO STM

working with programs to ensure they were accurately identifying

software projects and separating that effort from other projects

supporting the program. The office also focused on ensuring that

programs accurately reported the delivery times of those software

projects to headquarters via the Investment Evaluation, Submission, &

One of the provisions required the Chief Information Officer to certify whether an IT

investment is adequately implementing incremental development, as defined in capital

planning guidance issued by the Office of Management and Budget. Pub. L. No. 113-291,

§ 831 as codified at 40 U.S.C. § 11319(b)(1)(B)(ii). In June 2015, the Office of

Management and Budget released guidance (Management and Oversight of Federal

Information Technology, Memorandum M-15-14) describing how agencies are to

implement FITARA. Although Office of Management and Budget guidance and FITARA

only prescribed the use of incremental, and not specifically Agile, development, DHS uses

Agile development as a mechanism for complying with the requirements for incremental

development.

Page 18 GAO-20-213 Agile Software Development

Tracking system.

The delivery time impacts the agency rating on the

Federal Information Technology Acquisition Reform Act scorecard.

According to the Director of the OCTO STM, DHS has been validating

data reported to the Investment Evaluation, Submission, & Tracking

system to comply with capital planning and investment control

requirements from the Office of Management and Budget. The

Director stated that the fields in the Investment Evaluation,

Submission, & Tracking system were incorrect at first, but, through

interaction with the Office of Management and Budget, DHS was able

to change the fields through which Agile adoption is measured.

In March 2019, the department provided an updated dashboard for

measuring programs that included identifying programs adopting Agile

development or another form of incremental development.

In May

2017, the department published an initial set of Agile core metrics.

The purpose of these metrics was to provide DHS component IT

programs with direction on the core software delivery metrics that

DHS headquarters would subsequently require programs to collect

and report. DHS also intended for these core metrics to inform the

department on program or software delivery health, maturation, and

stability in delivering their intended capabilities and outcomes. Among

other things, the updated dashboard identified Agile programs based

on those that completed and submitted the Agile core metrics to their

department.

The Investment Evaluation, Submission, & Tracking system is a central repository for

data on DHS acquisitions and investments, such as budget, schedule, and performance

information. Data in this system are used to oversee both major and non-major

acquisitions and to satisfy internal and external reporting requirements.

Beginning in November 2015, the House of Representatives Committee on Oversight

and Government Reform released its first Federal Information Technology Acquisition

Reform Act (FITARA) scorecard that assigned letter grades to federal agencies on their

implementation of FITARA. See GAO, Information Technology: Effective Practices Have

Improved Agencies’ FITARA Implementation, GAO-19-131 (Washington, D.C.: Apr. 29,

2019) for additional information.

The DHS OCIO Agile Certification Report for March 2019 tracked two categories: 1) the

current Federal IT Acquisition Reform Act score and 2) the current CIO certification score.

The Federal IT Acquisition Reform Act score tracks the percentage of IT programs that

report using an iterative, incremental, Spiral, or Agile software development methodology

and report “yes” to releasing to production every six months in the Investment Evaluation,

Submission, & Tracking system. The CIO certification score tracks the percentage of IT

programs that meet the requirements for the Federal IT Acquisition Reform Act score and

also report a last delivery date within six months, that the program is in the ‘obtain’ phase,

and that the Agile core metrics is completed. As of March 2019, the Agile certification

report included a current CIO certification score of 29%.

Page 19 GAO-20-213 Agile Software Development

However, the Agile core metrics that DHS relied on to measure the

Agile adoption rate were not consistently reported to the Investment

Evaluation, Submission, & Tracking system as required. According to

the Director of STM, the department was still working with programs

and capital planning and investment control administrators to increase

compliance with reporting the core metrics. Until DHS can ensure that

all Agile programs are consistently reporting the core metrics, the list

of Agile programs will be incomplete. In addition, until DHS can

identify Agile programs and begin to measure results, it risks not

being appropriately informed about whether its efforts are having a

positive impact on product and performance results.

• Measuring results through its impact on the agency. The

department did not measure the results associated with the transition

to Agile development or the success of the transition based on its

impact on the department. According to the Director of STM, the

department had intended to measure results no later than April 2019

as part of a particular action plan. This plan was to pursue text and

business analytics solutions and leverage automation capabilities to

increase effectiveness of program analysis, planning, and oversight

reporting. However, the department closed this action plan in April

2019 without demonstrating the value of the transition to Agile.

In written comments, the Director of STM stated that the action plan

was closed based on completing an experiment to show that analytics

could lead to measuring success of incremental acquisition

techniques. According to the Director, the department is now pursuing

tools and techniques to put the results of this action plan into practice.

The Director added that further investment will likely be required to

fully meet this anticipated outcome.

By measuring and communicating the results of the transition to Agile

development, DHS can determine whether Agile is achieving its

desired results and if Agile programs are performing better or worse

than programs performed prior to the transition to Agile development.

The department may also increase the acceptance and adoption of

Agile among people throughout the department because they may

better understand the associated results.

Page 20 GAO-20-213 Agile Software Development

Leading practices that we developed for Agile software development

adoption are organized into three areas, called organizational levels:

agency environment, program processes, and team activities and

dynamics.

The organizational levels are further divided into nine leading

practices. Table 2 identifies the three organizational levels and nine

leading practices associated with these levels (three practices within each

area). A detailed assessment of DHS’s implementation of each of the

nine leading practices can be found in appendixes III, IV, and V.

Table 2: Levels of Agile Adoption and Leading Practices Associated with Each Level

Practice level

Leading practice

Leading practice description

Agency

Environment

Agency activities support Agile methods

The agency should establish appropriate life cycle activities and

ensure that goals and objectives are clearly aligned.

Agency culture supports Agile methods

The agency’s sponsorship for Agile development should cascade

throughout the agency and sponsors should understand Agile

development. The agency should also establish an environment

supportive of Agile development. Incentives and rewards should be

aligned to Agile development methods.

Agency acquisition policy and procedure

support Agile methods

Agency guidance should be appropriate for Agile acquisition

strategies

Program

Processes

Staff are appropriately trained in Agile

methods

Agency policy or guidance should ensure that all program staff are

trained in Agile methods and call for Agile teams to have the

appropriate technical expertise needed to perform their roles.

Technical environments enable Agile

development

Agency policy or guidance should call for technical and project tools

being available to support Agile development. In addition, policy or

guidance should call for system design that will support iterative

delivery.

Project planning controls are compatible

with Agile development

Agency policy or guidance should call for teams to maintain a

sustainable development pace and track and monitor that

development pace. In addition, policy or guidance should call for non-

functional requirements and critical features to be defined and

incorporated in development.

Team Activities

and Dynamics

Team composition supports Agile

methods

Agency policy or guidance should call for self-organizing Agile teams

and define the role of a product owner to support Agile methods.

Work is prioritized to maximize value for

the customer

Agency policy or guidance should call for Agile teams to use user

stories to define work, requirements to be prioritized in a backlog

based on value, including tracking and monitoring the value of work

accomplished, and for Agile teams to estimate the relative complexity

of user stories.

See appendix I for more information regarding the process for compiling the leading

practices.

DHS Has Made Progress

in Implementing Nine

Leading Practices for Agile

Software Development

Adoption, but Has Not

Fully Implemented All

Page 21 GAO-20-213 Agile Software Development

Practice level

Leading practice

Leading practice description

Repeatable processes are in place

Agency policy or guidance should call for Agile teams to meet daily to

review progress and discuss impediments, and observe end-iteration

demonstrations and end-iteration retrospectives. In addition, agency

policy or guidance should call for Agile projects to employ continuous

integration and confirm mechanisms are in place to ensure the quality

of code being developed. This includes setting expectations for

automated testing and code quality and tracking and monitoring

against these expectations.

Source: GAO analysis of DHS documentation. | GAO-20-213

We refer to the leading practices related to an agency’s processes,

culture, and acquisition strategies as agency environment practices. For

an agency to successfully transition from an agency that supports

traditional development methods, it should ensure that its activities,

culture, and acquisition policy and procedures support Agile methods.

DHS partially implemented the agency environment practice level by fully

implementing two leading practices and partially implementing the

remaining one. A more detailed assessment of DHS’s agency

environment leading practices can be found in appendix III.

• Agency activities support Agile methods–fully implemented. DHS

established appropriate life cycle activities to support Agile methods.

For example, the department has outlined its policies, procedures,

and guidance in several documents to assist its components in the

acquisition and implementation of Agile software development. The

department also developed policies and procedures that called for the

alignment of software, program goals, and agency goals.

• Agency culture supports Agile methods–partially implemented.

DHS established an environment that supported Agile development,

and senior stakeholders supported its development throughout the

agency. However, DHS did not take sufficient steps to ensure that

senior stakeholders serving as executive sponsors understood Agile

development, as called for by leading practices that are described in

further detail in appendix III. The Director of STM stated that Agile

sponsors were considered to be chief executive officers (e.g.

Executive Director of PARM and the Deputy Under Secretary for

Management). These parties oversaw the actions of the ITPM COE

and approved the Agile action plans in June 2017.

In addition, the department did not require training for senior

stakeholders serving as executive sponsors, as called for by leading

practices. In a written response, the Office of the Chief Human Capital

Officer said that there were no Agile training requirements for officials

at this level. By training executive-level sponsors in Agile

DHS partially implemented

practices at the agency

environment level

Page 22 GAO-20-213 Agile Software Development

development, the department can mitigate the risk of setting

expectations for programs and projects that do not align with the

values and principles of Agile software development.

DHS also did not demonstrate that it established an incentives and

rewards structure to incentivize Agile teams, as called for by leading

practices. Officials from the Office of the Chief Human Capital Officer

stated that the department’s existing rewards structure allowed for

incentivizing team and individual performance even though it was not

focused specifically on Agile methods. These officials stated that they

did not believe that additional policy, guidance, or modifications to

their existing policy were necessary. The Director of STM within OCIO

stated that rewarding Agile teams was not a topic the ITPM COE was

currently considering, but that OCIO might be interested in pursuing

the topic after completing existing, higher-priority activities. By

considering modifications to policy and guidance governing the

incentives and rewards structure to promote team performance, DHS

could improve team productivity and output.

• Agency acquisition policies and procedures support Agile

methods–fully implemented. DHS guidance for acquisition

strategies supported the unique needs of Agile programs. For

example, DHS offered guidance for preparing acquisition strategies

through its Procurement Innovation Lab and published Agile guidance

that discussed contracting and acquisition strategies.

Program processes involve staff being appropriately trained in Agile

methods, technical environments enabling Agile development, and project

planning controls being compatible with Agile development. DHS partially

implemented the program processes practice level by fully implementing

one leading practice and partially implementing the remaining two. A

more detailed assessment of DHS program process leading practices can

be found in appendix IV.

• Staff are appropriately trained in Agile methods–partially

implemented. DHS training policy and guidance called for some of

the acquisition management program staff to be trained in Agile

methods. DHS has also taken steps to incorporate Agile concepts into

required training for members of the acquisitions workforce. In

addition, DHS offered elective training covering Agile methods and

The Department of Homeland Security Procurement Innovation Lab, Office of the Chief

Procurement Officer, experiments with innovative techniques for increasing efficiencies in

the procurement process and institutionalizing best practices.

DHS partially implemented

practices at the program

process level

Page 23 GAO-20-213 Agile Software Development

guidance for Agile teams, including contractors, to have the

appropriate technical expertise needed to perform their role.

The department also took steps to identify the necessary

competencies for Agile teams and individuals. In April 2019, the

Strategic Workforce Planning team within OCIO published a white

paper identifying 27 competencies necessary for teams and

individuals to use and training courses associated with the

competencies. The white paper also made recommendations to help

DHS address challenges in implementing Agile methods, such as

establishing communities of practice for Agile practitioners to identify

best practices and provide workshops. According to a written

response by OCIO, the Strategic Workforce Planning team will create

an implementation and communication plan for any deliverables

associated with the white paper.

However, the department did not provide policy or guidance to ensure

that all program staff were trained in Agile methods, as called for by

leading practices described in further detail in appendix IV. Existing

Agile training requirements covered only the acquisitions workforce.

DHS did not establish training requirements for program staff outside

of the acquisitions workforce—such as a product owner or other

staff—who may be assigned to an Agile program. As a result,

individual programs must independently decide on and enforce

training requirements if they want to ensure that all staff receive the

needed training.

DHS officials stated that the department focuses on key acquisition

career fields in part because those career fields are defined in policy

and procedures.

According to the Director of STM, the department

also encourages programs to independently find coaching and

training because the components are more likely to have funding. By

providing policy or guidance to ensure that all personnel staffed to an

Agile program or project receive appropriate training, the department

can better prepare program staff to plan and execute appropriately,

and increase the likelihood of achieving the expected outcomes of the

transition to Agile.

• Technical environments enable Agile development–fully

implemented. DHS guidance called for technical and project tools to

be available to support Agile development. For example, DHS test

Department of Homeland Security, Directive 064-04, Acquisition Professional Career

Information, Revision 00 (Oct. 30, 2008).

Page 24 GAO-20-213 Agile Software Development

and evaluation guidance stated that automated testing should be

implemented where practical.

In addition, DHS guidance called for system designs that will support

iterative delivery. For example, DHS enterprise architecture guidance

and supplementary design considerations for acquisition programs

discussed loose coupling and different methods for establishing a

modular system.

• Project planning controls are compatible with Agile

development–partially implemented. DHS guidance called for

defining and incorporating non-functional requirements and critical

features throughout development. In addition, DHS provided guidance

for establishing a sustainable development pace. For example, the

Agile instruction manual identified the benefits of monitoring the

amount of work completed by Agile teams across each iteration in

order to monitor ongoing team progress.

However, DHS was not tracking and monitoring the pace of Agile

team development as called for by DHS guidance and described

further in appendix IV. According to the Director of STM, programs

were not consistently reporting the Agile core metrics associated with

development team pace as required. The Director of STM stated that

the department was taking steps to begin tracking and monitoring the

pace of Agile teams. In addition, the Director stated that he allocated

staff to assist programs with consistently reporting the Agile core

metrics.

According to the Director of STM, the department was in the process

of updating the core metrics and intended to publish a new version of

them in the future, which would include tracking the pace.

Nevertheless, DHS did not provide assurance that the metrics

associated with development pace would be included in this revised

set of metrics or that programs would consistently report that

information in order for the department to track and monitor the pace

of Agile teams. Until the department consistently tracks and monitors

Agile programs and projects, it will not have the information needed to

help ensure the development pace is maintained.

Practices at the team activities and dynamics level include team

composition supporting Agile methods, work being prioritized to maximize

value for the customer, and repeatable processes being in place. DHS

partially implemented the team activities and dynamics practice level by

fully implementing one leading practice and partially implementing the

remaining two. A more detailed assessment of DHS team activity and

dynamics leading practices can be found in appendix V.

DHS partially implemented

practices at the team activity

and dynamics level

Page 25 GAO-20-213 Agile Software Development

• Team composition supports Agile methods—fully implemented.

DHS established guidance that called for self-organizing teams and

defined the role of a product owner. For example, the Agile instruction

and Agile instruction manual both explain that collaborative, self-

organizing, and cross-functional teams help achieve the flexibility

needed for the iterative development that characterizes Agile

development methods. In addition, the Agile instruction manual states

that the product owner is responsible for representing stakeholders

and should be available to the development team throughout the

iteration to answer questions and clarify requirements on behalf of the

stakeholders.

• Work is prioritized to maximize value for the customer—partially

implemented. DHS guidance called for Agile teams to craft user

stories to define work. The guidance also called for user stories to be

prioritized in a backlog based on value.

However, the guidance did not describe how Agile teams can

estimate the relative complexity of the user stories as called for by

leading practices and described in further detail in appendix V. The

Director of STM stated that relative estimation is a basic exercise and

that guidance on this topic can be found in a number of sources

outside of DHS. However, without providing guidance or directing

Agile teams to external sources for additional information on relative

estimation, OCIO risks that teams supporting Agile projects will not

appropriately estimate user stories relative to each other.

By providing guidance on estimating the relative complexity of user

stories, the department can help Agile teams to effectively commit to

an appropriate amount of work during a given iteration.

• Repeatable processes are in place—partially implemented. DHS

guidance addressed holding daily meetings to review progress and

discuss impediments, using a demonstration for the acceptance of a

user story and conducting a retrospective to evaluate progress. In

addition, the department’s guidance called for Agile programs to

employ continuous integration and emphasized the need for

mechanisms to help ensure code quality.

However, DHS did not set expectations for automated testing and

code quality, as called for by DHS guidance and described further in

appendix V. DHS’s Agile core metrics included a series of metrics that

addressed automated testing and code quality. The core metrics

included targets but the targets were notional and, therefore, not

expectations that DHS required a program to meet. According to the

Director of STM, the initial core metrics were intended to assess the

Page 26 GAO-20-213 Agile Software Development

level of DHS team achievement without imposing artificial industry-

based target measures for each. The Director stated that, on receiving

the metrics for a period of time, the department would then adjust the

core metrics and begin to include target measures based on the

results achieved. According to the Director, this effort is currently

underway and an updated set of core metrics will be distributed in

early fiscal year 2020.

Moreover, the department did not track and monitor automated testing

or code quality against expectations. As discussed under project

planning controls, DHS intended to track and monitor Agile practices,

such as automated testing and code quality, through the Agile core

metrics. However, according to the Director of STM, programs and

projects were not consistently reporting these core metrics and those

that were reporting did not collect data or report on particular metrics.

By setting expectations for automated testing and code quality and

beginning to track and monitor project performance against these

expectations, DHS can increase the likelihood that Agile programs

and projects are delivered within cost, schedule, and performance

estimates.

DHS has taken many positive steps in its transition to Agile software

development. It has implemented activities and artifacts that support all

levels of adoption, from the department and component offices to Agile

programs, projects, and teams. These activities and artifacts include

providing opportunities for Agile programs and projects to streamline

acquisition and life cycle processes to allow for iterative delivery and

exhibiting senior support for the transition to Agile.

The department successfully planned for the transition to Agile software

development and completed many of its intended implementation

activities. However, because DHS did not assess the skills and resources

needed to complete deferred activities, it risks continued delays in

completing these. In addition, without identifying target measures tied to

expected outcomes, the department is limited in determining whether the

transition is achieving its desired outcomes. Moreover, until DHS can

ensure that all programs are consistently reporting on Agile core metrics,

the department will not be able to track programs’ development

techniques. Further measuring and communicating the benefits of the

transition can enable the department to know whether Agile programs are

performing better than those used prior to the transition.

Conclusions

Page 27 GAO-20-213 Agile Software Development

DHS has demonstrated significant progress in implementing leading Agile

practices. The department can further improve its performance through

full execution of the remaining partially implemented practices. At the

agency environment level, DHS can mitigate risk and improve productivity

through executive level training and modifications to policy to incentivize

Agile teams. For program level practices, addressing training

requirements for all necessary staff and tracking and monitoring the pace

of Agile team development can help ensure teams’ success.

With respect to team-level practices, DHS has not established guidance

for estimating the relative complexity of user stories. As a result, Agile

teams are hampered in effectively committing to an appropriate amount of

work during a given period of time. Finally, because DHS has not set

expectations for performance metrics for monitoring and tracking the use

of automated testing and code quality, DHS is at a greater risk for

programs breaching their cost and schedule expectations.

We are making the following 10 recommendations to the Secretary of the

Department of Homeland Security (DHS).

The Secretary should ensure that the Director of Strategic Technology

Management (STM), in collaboration with other members of the

Information Technology Program Management Center of Excellence

(ITPM COE), identifies the skills and resources needed to complete the

work intended for the upcoming fiscal year, including the availability of

supplementary staff, such as subject matter experts. (Recommendation

The Secretary should ensure that the Executive Steering Committee

overseeing the activities of the ITPM COE establishes target measures

for the department’s desired outcomes of its transition to Agile

development. (Recommendation 2)

The Secretary should ensure that the DHS Chief Information Officer (CIO)

defines a process and associated set of controls to ensure that Agile

programs and projects are reporting a set of core required performance

metrics for monitoring and measuring Agile adoption. (Recommendation

The Secretary should ensure that the ITPM COE, in coordination with the

CIO, begins measuring results associated with the transition to Agile and

the success of the transition based on its impact on the department.

(Recommendation 4)

Recommendations for

Executive Action

Page 28 GAO-20-213 Agile Software Development

The Secretary should ensure that the CIO, in collaboration with the Chief

Procurement Officer, through the Homeland Security Acquisition Institute,

establish Agile training requirements for senior stakeholders.

(Recommendation 5)

The Secretary should ensure that the Chief Human Capital Officer, in

collaboration with the CIO, consider modifications to the current employee

recognition and performance management governance to ensure that

teamwork and team performance of Agile programs and projects are

incentivized. (Recommendation 6)

The Secretary should ensure that the CIO, in collaboration with the Chief

Procurement Officer, through the Homeland Security Acquisition Institute,

establish Agile training requirements for staff outside of the acquisition

workforce but assigned to Agile programs. (Recommendation 7)

The Secretary should ensure that the CIO, upon establishing a set of core

performance metrics, tracks and monitors the pace of Agile team

development. (Recommendation 8)

The Secretary should ensure that the CIO, in collaboration with the

Executive Director of the Office of Program Accountability and Risk

Management (PARM), update or develop new guidance on Agile

methodologies to describe how Agile teams can estimate the relative

complexity of user stories. (Recommendation 9)

The Secretary should ensure that the CIO, upon establishing a set of core

performance metrics, sets expectations for automated testing and code

quality, and tracks and monitors against those expectations.

(Recommendation 10)

DHS provided written comments on a draft of this report. In its comments

(reproduced in Appendix VI), the department agreed with our 10

recommendations and described actions that it had completed and

planned to address them.

Based on the actions DHS said it had taken, the department requested

that we close the first three recommendations as implemented. For

example, the department described steps it had taken to address our

recommendation that it identify the skills and resources needed to

complete the work intended for the upcoming fiscal year, including the

availability of supplementary staff such as subject matter experts. In

addition, the department stated that it had addressed our

Agency Comments

and Our Evaluation

Page 29 GAO-20-213 Agile Software Development

recommendation to define a process and controls to ensure that Agile

programs and projects are reporting a set of core required performance

metrics for monitoring and measuring Agile adoption. We plan to follow up

with DHS to assess the sufficiency of its actions to address our

recommendations.

The department also described actions that it plans to take to address the

other seven recommendations. For example, DHS stated that it will use

the results of its Agile core metrics and Agile Software Delivery Maturity

Model to measure the success of the transition to Agile and its impact on

the department. According to the department, it expects this action to be

completed by June 30, 2021.

Further, DHS stated that it will identify Agile training requirements for staff

in Agile programs, and will use that to establish Agile training

requirements for staff outside of the acquisition workforce but assigned to

Agile programs. Specifically, DHS stated that the DHS OCIO will gather

requirements from components via its IT workforce planning integrated

project team to identify training resources available across the

department that also address the skill sets needed for Agile programs.

The department added that the DHS OCIO will utilize information from the

April 2019 white paper, titled “OCIO Agile White Paper” to inform

proposed Agile program training requirements. The department estimated

that these actions are to be completed by September 30, 2020.

DHS also provided technical comments, which we have incorporated as

appropriate.

We are sending copies of this report to the Acting Secretary of Homeland

Security and interested congressional committees. In addition, the report

will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact

me at (202) 512-4456 or [email protected]. Contact points for our Offices

of Congressional Relations and Public Affairs may be found on the last

page of this report. GAO staff who made key contributions to this report

Page 30 GAO-20-213 Agile Software Development

are listed in appendix VII. In addition, the report is available at no charge

on the GAO website at http://www.gao.gov.

Carol C. Harris

Director

Information Technology Acquisition Management Issues

Page 31 GAO-20-213 Agile Software Development

List of Requesters

The Honorable Xochitl Torres Small

Chairwoman

Subcommittee on Oversight, Management, and Accountability

Committee on Homeland Security

House of Representatives

The Honorable J. Luis Correa

House of Representatives

The Honorable Scott Perry

House of Representatives

Appendix I: Objective, Scope, and

Methodology

Page 32 GAO-20-213 Agile Software Development

Our objective was to assess the extent to which the Department of

Homeland Security (DHS) addressed selected leading practices for its

transition to the use of Agile software development. To accomplish this

objective, we assessed the extent to which the department adhered to

leading practices in two specific areas: organizational change

management and Agile software development adoption.

With regard to organizational change management, we reviewed leading

practices published by the Project Management Institute and GAO.

Based on this review, we identified 15 leading practices. We then

grouped these 15 practices in three broad organizational change

management areas: planning, implementing, and measuring change.

To determine the extent to which DHS addressed leading practices for

organizational change management in its transition to Agile development,

we assessed DHS policies, procedures, guidance, plans, and other

working group artifacts and compared them against leading practices. In

particular, we reviewed working group charters for the DHS headquarters

Agile Acquisition Integrated Program Team and IT Program Management

Center of Excellence (ITPM COE), and any plans developed by these

working groups, including the DHS Agile Action Plans and associated

implementation plans. We then reviewed working group meeting minutes,

presentation slides, and status update charts to assess the progress of

the transition to Agile, identified artifacts prepared to support the transition

to Agile, and assessed the status of plans for the transition to Agile. We

reviewed all Agile artifacts prepared by or supporting the Agile working

groups, such as a preliminary software development maturity model, the

DHS Agile Acquisition Software Delivery Core Metrics (Agile core

metrics), and an updated test and evaluation master plan template for

Agile, among other artifacts.

We also interviewed officials from DHS headquarters line of business

representatives explicitly identified in the Agile Development and Delivery

Project Management Institute, Inc., Managing Change in Organizations: A Practice

Guide, 2013. GAO, Government Reorganization: Key Questions to Assess Agency

Reform Efforts, GAO-18-427 (Washington, D.C.: Jul 13, 2018); IT Workforce: Key

Practices Help Ensure Strong Integrated Program Teams; Selected Departments Need to

Assess Skill Gaps, GAO-17-8 (Washington, D.C.: Nov. 30, 2016); Standards for Internal

Control in the Federal Government, GAO-14-704G (Washington, D.C.: Sept. 2014).

Department of Homeland Security, Agile Acquisition Software Delivery Core Metrics,

Version 1.0 (May 23, 2017).

Appendix I: Objective, Scope, and

Methodology

Appendix I: Objective, Scope, and

Methodology

Page 33 GAO-20-213 Agile Software Development

for Information Technology instruction (Agile instruction).

This included

officials from the Office of the Chief Procurement Officer, Office of the

Chief Financial Officer, Office of the Chief Information Officer (OCIO),

Office of Program Accountability and Risk Management (PARM), and the

Science and Technology Directorate, and offices of Test and Evaluation

and Systems Engineering. Within OCIO, we interviewed officials from the

Office of the Chief Technology Officer (OCTO) within the Strategic

Technology Management (STM) division, among others, as STM is the

entity tasked with facilitating the ITPM COE and serves as the official

liaison between other OCIO divisions, other partner headquarters

directorate and management offices, and operational components. We

also interviewed representatives from groups participating in ITPM COE

activities but not explicitly called out in the Agile instruction, including the

Privacy Office and Joint Requirements Council. In addition, we

interviewed representatives from other groups not represented on the

ITPM COE but potentially impacted by the transition to Agile. This

included officials from the Office of the Chief Readiness Support Officer

and Office of the Chief Human Capital Officer.

With regard to leading practices for Agile software development adoption,

we reviewed work performed by GAO to develop generally accepted

leading practices. In developing these leading practices, GAO reviewed

information from a variety of sources related to Agile adoption and

compiled a draft of leading practices commonly mentioned across these

different sources.

We then convened a working group of experts from the

public and private sectors and academia. This working group met three

times a year between August 2016 and August 2019 to review and

discuss these leading practices. More than 200 experts participated in the

Department of Homeland Security, Instruction 102-01-004; Instruction Manual 102-01-

004-01. DHS defines the line of business chiefs as officers at the department level with a

set of one or more highly related services (administrative, financial management, human

resources, information technology, procurement, and security), which include the Chief

Procurement Officer, Chief Readiness Support Officer, Chief Financial Officer, Chief

Human Capital Officer, Chief Security Officer, and the Chief Information Officer.

See, for example, Booz Allen Hamilton, Agile Playbook, Version 2.0 (Washington, D.C.:

June 2016); California Department of Technology, California Project Management Office,

Understanding Agile, Version 1.0 (California: Dec. 5, 2016); National Association of State

Chief Information Officers and Accenture, Agile IT Delivery: Imperatives for Government

Success (Washington, D.C.: 2017); Office of Management and Budget, U.S. Digital

Services, Playbook (version pulled on Dec. 22, 2017); TechFAR: Handbook for Procuring

Digital Services Using Agile Processes (version pulled on Mar. 8, 2018); Project

Management Institute, Inc., Agile Practice Guide (Newtown Square, PA: 2017); Software

Engineering Institute. The Readiness & Fit Analysis: Is Your Organization Ready for

Agile? (Pittsburgh, PA: Apr. 2014).

Appendix I: Objective, Scope, and

Methodology

Page 34 GAO-20-213 Agile Software Development

meetings, including more than 20 officials from DHS. GAO received

comments from some of these experts both during these meetings and by

email after the meetings.

Based on this work, GAO developed a set of nine leading practices for

Agile adoption. GAO grouped these leading practices into three

organizational levels: (1) agency environment, (2) program processes,

and (3) team activities and dynamics. The leading practices were further

described by a series of core elements and core element expectations

that, collectively, can be used to assess the status of an agency’s

implementation.

To determine the extent to which the department had implemented the

leading practices for the adoption of Agile development, we obtained and

assessed DHS policies, procedures, guidance, plans, and other

documentation and compared them against the nine leading practices. In

particular, we reviewed department acquisition policy, procedures, and

guidance, such as acquisition management directive 102-01; software

engineering life cycle policy, procedures, and guidance, such as those

published in the software engineering life cycle guidebook; requirements

policy, procedures, and guidance, such as the Joint Requirements

Integration and Management System and Requirements Engineering

User’s Guide; testing policy, procedures, and guidance, such as the Test

and Evaluation Master Plan template and Test and Evaluation

Management Guide; technical assessment and enterprise architecture

policy, procedures, and guidance; program health assessment policy,

procedures, and guidance such as the Acquisition Program Health

Assessment instruction and CIO Program Health Assessment Scoring

Guideline; and Agile-specific policy, procedures, and guidance, such as

the Agile instruction and the Agile Development and Delivery for

Information Technology Instruction Manual (Agile instruction manual),

among other policy, procedures, and guidance.

In addition to reviewing the department policy, procedures, and guidance,

we obtained and assessed supplementary Agile documentation. In

particular, we reviewed training materials prepared by the Homeland

Security Acquisition Institute for acquisition workforce certifications and

webinars offered by the Procurement Innovation Lab; ITPM COE Agile

artifacts discussed under our assessment of the implementation of

Department of Homeland Security, Instruction Manual 102-01-004-01, Agile

Development and Delivery for Information Technology Instruction Manual, Revision 00

(Jul. 15, 2016).

Appendix I: Objective, Scope, and

Methodology

Page 35 GAO-20-213 Agile Software Development

organizational change management leading practices, such as the Agile

core metrics; and Agile-specific technical review completion letters, such

as the release planning review.

We also interviewed officials from the components responsible for the

associated policy, procedures, and guidance and those specifically cited

in the Agile instruction. This included officials from the Office of the Chief

Procurement Officer, Office of the Chief Financial Officer, OCIO, PARM,

Science and Technology Directorate, offices of Test and Evaluation and

Systems Engineering, the Joint Requirements Council, Office of the Chief

Readiness Support Officer, and Office of the Chief Human Capital Officer.

As with our assessment of DHS implementation of organizational change

management practices, within OCIO, we interviewed officials from the

OCTO STM division, among others.

We assessed a core element as being “met” if the department provided

supporting documentation that demonstrated it met all of the expectations

associated with the core elements. We assessed a core element as being

“partially met” if the department provided supporting documentation that

demonstrated some, but not all, aspects of the underlying expectations.

We assessed a core element as “not met” if the officials did not provide

any supporting documentation for the core element, or if the

documentation provided did not demonstrate any aspect of the underlying

expectations. The expectations associated with each core element are

described more fully in appendixes III, IV, and V.

We assessed each leading practice and practice level as being “fully

implemented” if DHS provided evidence that it had met all of the core

elements. We assessed each leading practice and practice level as being

“not implemented” if DHS did not provide evidence that it had met or

partially met any of the core elements. We assessed each leading

practice and practice level as being “partially implemented” if DHS

provided evidence that it had not met all core elements and partially met

at least one core element.

To supplement our assessment of the department’s implementation of the

leading practices for adopting Agile development, we also assessed

selected projects’ implementation of selected program process and team

activity and dynamics leading practices. We updated the core element

test plans to include general control objectives, associated controls, and

associated test steps in order to reach a determination on the extent to

which these projects implemented a particular aspect of a leading

practice.

Appendix I: Objective, Scope, and

Methodology

Page 36 GAO-20-213 Agile Software Development

We identified potential case study projects based on data provided by

DHS from the Investment Evaluation, Submission, & Tracking system.

We determined that the data in the Investment Evaluation, Submission, &

Tracking system was sufficiently reliable for our use in selecting projects

for our case studies. We selected case study projects, rather than

programs, because, according to DHS officials from OCIO, programs

report software development life cycle data to the Investment Evaluation,

Submission, & Tracking system at the project level only.

We selected only the projects supporting programs on the Major

Acquisition Oversight List because these programs are expected to

comply with the Agile instruction and acquisition management policy. We

then further limited the scope of projects to those within components

where GAO has not previously assessed a program using Agile methods

or was not in the process of assessing such a program. This excluded the

U.S. Citizenship and Immigration Services, Federal Emergency

Management Agency, Transportation Security Administration, and U.S.

Secret Service.

We then further refined our selection based on the following criteria:

• Software development life cycle methodology (iterative development

only)

• Project completion date (in-progress only)

• DHS component (selection of only one project per component)

We then selected a random sample of three projects, with no more than

one project selected from a component. The three case study projects we

selected were the U.S. Coast Guard (USCG) Command, Control,

Communications, and Computers, Intelligence, Surveillance and

Reconnaissance (C4ISR) program New Asset Acquisition Offshore Patrol

Cutter project, with particular attention to the SeaWatch portion of this

GAO, Immigration Benefits System: U.S. Citizenship and Immigration Services Can

Improve Program Management, GAO-16-467 (Washington, D.C.: July 7, 2016); TSA

Modernization: Use of Sound Program Management and Oversight Practices Is Needed to

Avoid Repeating Past Problems, GAO-18-46 (Washington, D.C.: Oct. 17, 2017); U.S.

Secret Service: Action Needed to Address Gaps in IT Workforce Planning and

Management Practices, GAO-19-60 (Washington, D.C.: Nov. 15, 2018); FEMA Grants

Modernization: Improvements Needed to Strengthen Program Management and

Cybersecurity, GAO-19-164 (Washington, D.C.: Apr. 9, 2019).

Appendix I: Objective, Scope, and

Methodology

Page 37 GAO-20-213 Agile Software Development

project;

the U.S. Customs and Border Protection (CBP) Biometric Entry

Exit (BEE) program Air Exit project, with particular attention to the

Traveler Verification Services portion of this project;

and the U.S.

Immigration and Customs Enforcement (ICE) Student and Exchange

Visitor Information System (SEVIS) program 8001 project, with particular

attention to the SEVIS modernization portion of this effort.

In preliminary

interviews, we confirmed that these projects were applying Agile practices

in order to validate data reported to the Investment Evaluation,

Submission, and Tracking system.

These case studies were used to supplement our findings from our

program process and team activity and dynamics-level evaluations of the

department’s implementation of leading practices for adopting Agile

development. To evaluate case studies’ implementation of these leading

practices, we reviewed artifacts from the selected projects. In particular,

we reviewed artifacts demonstrating a project’s use of Agile including

testing metrics, evidence of Agile ceremonies, the existence of user

stories and a backlog, and the availability of Agile coaching and training.

We then interviewed officials responsible for program and project

management and representatives of groups responsible for software

development for the three selected case study projects to discuss gaps

we identified. We shared our initial assessment with DHS, USCG, CBP,

and ICE and obtained feedback and additional supporting documentation.

Regarding our analysis of project implementation of the program process

and team activity and dynamics core elements, we followed the

aforementioned process in assessing a core element as being “met”,

“partially met”, or “not met”. These assessments were used to gain insight

into the extent to which DHS policy, procedures, and guidance prepared

We initially selected the U.S. Coast Guard Command, Control, Communications,

Computers, Intelligence, Surveillance and Reconnaissance Program National Security

Cutter project; however, in an interview, U.S. Coast Guard officials stated that this project

was not implementing Agile practices and, therefore, would not be suitable for our

purposes. Officials suggested looking at the SeaWatch project, a non-major project

supporting the Offshore Patrol Cutter project, which does support a larger program on the

Major Acquisition Oversight List, or a logistics management system project that supported

a non-major program.

GAO has previously reported on the Biometric Entry Exit program. However, these

reports have not assessed Agile practices within the program or underlying projects.

There is only one project associated with the larger program in the Investment

Evaluation, Submission, & Tracking system.

Appendix I: Objective, Scope, and

Methodology

Page 38 GAO-20-213 Agile Software Development

programs and projects for the successful adoption of Agile leading

practices. We did not evaluate the projects in order to make specific

recommendations to the individual projects.

We conducted this performance audit from December 2017 through April

2020, in accordance with generally accepted government auditing

standards. Those standards require that we plan and perform the audit to

obtain sufficient, appropriate evidence to provide a reasonable basis for

our findings and conclusions based on our audit objectives. We believe

that the evidence obtained provides a reasonable basis for our findings

and conclusions based on our audit objectives.

Appendix II: DHS Agile Action Plans

Page 39 GAO-20-213 Agile Software Development

In June 2017, Department of Homeland Security (DHS) senior

stakeholders endorsed Recommendations Action Plans: Agile Acquisition

Pilots, developed by the Agile Acquisitions Working Group. These

recommendations were an effort to sustain the success of the information

technology (IT) acquisition and delivery pilot program.

The action plans

were developed in response to the February 18, 2016, Acquisition

Decision Memorandum from the Under Secretary for Management, which

recognized the expressed need for both components and headquarters

directorates to continue driving organizational change and process

improvement to DHS IT acquisitions and delivery. The action plans were

intended to codify lessons learned and recommendations based on

independent interviews and retrospective meetings with those who

participated in the five acquisition pilots. These plans were organized by

priority: 12 critical, three high, and three moderate. The recommendations

were weighted against one another based on impact, level of difficulty,

and alignment with the original five goals of the Agile acquisition pilot

program charter: reduce risk, increase customer value, faster time to

market, economic value, and increased accountability and oversight.

Table 3 describes the DHS Agile action plans, including the associated

goal, primary organization(s), level of difficulty, impact, and executive

priority.

Senior stakeholder endorsement refers to the commitment of their office and staff’s

support to the higher level goal of improving acquisitions for IT programs throughout the

department. This endorsement does not refer to the appointment of any one office the

responsibility for the execution of some or all of the recommendations.

Appendix II: DHS Agile Action Plans

Page 40 GAO-20-213 Agile Software Development

Table 3: DHS Agile Action Plan Title, Goal, Lead Organizations, Level of Difficulty and Impact, and Executive Priority

Action

plan

number

Action plan title Goal

Primary

organization

Level of

difficulty

Impact

Executive

priority

Improve the process for acquisition

document review, adjudication, and

approval, enabled by workflow

management and process automation

technology solutions

Faster time to

market

Office of Program

Accountability and

Risk Management

(PARM)

Office of the Chief

Information Officer

(OCIO)

Critical

Establish a unified authority to govern,

institutionalize, and manage the

implementation of Agile Acquisitions

Working Group action plans and enable

continuous improvement of IT

acquisitions and delivery

Increased

accountability

and oversight

OCIO

PARM

Critical

Establish a scalable future operating

model for support of level 1 and 2

acquisition and IT programs

Reduce Risk

PARM

OCIO

Science and

Technology

Directorate

Critical

Define roles and responsibilities for

each step or phase of the acquisition life

cycle framework (ALF) and systems

engineering life cycle (SELC)

Increased

accountability

and oversight

PARM

OCIO

Science and

Technology

Directorate

Critical

Incorporate Agile governance and

review models to increase transparency

and feedback throughout the obtain

phase and operations and maintenance

Reduce risk

PARM

OCIO

Science and

Technology

Directorate

Office of the Chief

Financial Officer

Critical

Modify principle acquisition decision

points and production reviews, including

Acquisition Decision Events 2A and 2B,

initial operating capability, full

operational capability, and production

readiness reviews

Reduce risk

PARM

OCIO

Joint Requirements

Council

Critical

Review DHS acquisition guidance,

policy, and practices for the

identification and management of

requirements through the Joint

Requirements Council

Increase

customer value

PARM

OCIO

Joint Requirements

Council

Science and

Technology

Directorate

Critical

Appendix II: DHS Agile Action Plans

Page 41 GAO-20-213 Agile Software Development

Action

plan

number

Action plan title Goal

Primary

organization

Level of

difficulty

Impact

Executive

priority

Update DHS acquisition guidance,

policy, and practices for testing and

evaluation to enable modern best

practices in automated testing and

continuous integration

Economic

value

PARM

OCIO

Science and

Technology

Directorate

Critical

Update the DHS acquisition guidance,

policy, and practices for evaluation of

technical solutions and vendors,

including a lean Analysis of Alternatives

Faster time to

market

PARM

OCIO

Science and

Technology

Directorate

Critical

Update the DHS acquisition guidance,

policy, and practices for initial cost

estimation and lifecycle cost estimate

reviews for multiyear IT programs

Economic

value

Office of the Chief

Financial Officer

PARM

OCIO

Critical

Update the DHS acquisition guidance,

policy, and practices for cybersecurity

considerations for IT acquisitions

Increase

accountability

and oversight

OCIO

Components

Science and

Technology

Directorate

PARM

Joint Requirements

Council

Office of Intelligence

and Analysis

Critical

Map current, future, and ideal state

process relationships across the entire

ALF/SELC to identify continuous

improvement opportunities

Increase

accountability

and oversight

PARM

OCIO

Science and

Technology

Directorate

Critical

Remove redundant requirements for

program documentation and provide

clarifying expectations for Agile tailored

ALF artifacts

Faster time to

market

PARM

OCIO

Office of the Chief

Procurement Officer

Science and

Technology

Directorate

Office of the Chief

Financial Officer

Joint Requirements

Council

High

Appendix II: DHS Agile Action Plans

Page 42 GAO-20-213 Agile Software Development

Action

plan

number

Action plan title Goal

Primary

organization

Level of

difficulty

Impact

Executive

priority

Establish performance-based delivery

metrics and measures to monitor

program delivery health

Increase

accountability

and oversight

PARM

OCIO

Office of the Chief

Financial Officer

High

Enforce IT enterprise architecture

touchpoints within Management

Directive-103-02, ALF, and SELC

ensuring enterprise architecture

practices are embedded

Reduce risk

Science and

Technology

Directorate

OCIO

PARM

High

Develop strategic sourcing strategy for

Operational Test Agent vendors

Faster time to

market

Science and

Technology

Directorate

Office of the Chief

Procurement Officer

Moderate

Codify, implement, and apply the

software delivery maturity model, Agile

maturity model, in program health

assessments for DHS component

organizations and programs

Reduce risk

PARM

OCIO

Moderate

Pursue text and business analytics tools

leveraging automation capabilities to

increase effectiveness of program

analysis

Faster time to

market

PARM

Office of the Chief

Procurement Officer

Science and

Technology

Directorate

Joint Requirements

Council

Moderate

Source: DHS Recommendations Action Plans: Agile Acquisition Pilots | GAO-20-213

Appendix II: DHS Agile Action Plans

Page 43 GAO-20-213 Agile Software Development

Each DHS action plan included a problem statement and

recommendation, as detailed in table 4.

Table 4: DHS Agile Action Plan Problem Statements and Recommendations

Action plan

number

Problem statement Recommendation

The document review process lacks transparency

and is extensively manual, disjointed, and inefficient,

which poses a challenge for both programs and for

oversight bodies. Stakeholders are uncertain about

who should review, when reviews are completed,

and how comments were adjudicated and tracked.

Ambiguity in workflow management results in

process delays.

Baseline, re-engineer, and codify the review, adjudication, and

approval processes for acquisition life cycle framework (ALF)

and systems engineering life cycle (SELC) artifacts. Evaluate,

select, and implement a technology solution to enable

visualization of the workflow, tracking of completed reviews,

and tracking of comments and adjudicated decisions. The

selected solution must enable collaborative document reviews,

workflow management and notifications, and performance

monitoring based on the re-engineered process.

Without an overarching Value Stream Champion to

remove impediments, approve process

modifications, and drive action, implementing

process improvements that cross components and

headquarters directorates will be challenging. A

unified authority is necessary to govern,

institutionalize, and manage the changes needed;

otherwise, improvements may become stalled or

isolated within a component or stakeholder

organization.

Use the newly-established Transformation Executive Council

authority defined in Department of Homeland Security (DHS)

Management Directive-262-10 “DHS Digital Transformation.”

Leverage the role of S2 and the Under Secretary for

Management to establish a subordinate authoritative body that

includes Office of the Chief Procurement Officer, Office of the

Chief Financial Officer, Office of the Chief Technology Officer

(OCIO)/Chief Information Security Officer, Office of Program

Accountability and Risk Management (PARM), and as

necessary, component acquisition executives and component

chief information officers (CIOs) to enforce this unified authority.

This Unified Authority will be responsible for consolidating

efforts of PARM, OCIO/Office of the Chief Technology Officer

(OCTO)/Enterprise Architecture, Science and Technology

Directorate, Joint Requirements Council, and broader program

transformation activities, ensuring that investment, acquisition,

Joint Requirements Council, program, and delivery governance

are fully integrated.

The current framework and scope of the Agile

Acquisitions Working Group is not sustainable and

cannot be scaled across DHS. Agile Acquisitions

Working Group oversight bodies are strained due to

the level of effort required to establish and support

each of the component integrated project teams.

However, component programs have expressed the

need for continued support in managing and meeting

ALF and SELC requirements in a manner that

promotes modern best practices, process efficiency,

reduced risk, and program success.

Expand the existing DHS OCTO IT Project Management Center

of Excellence (ITPM COE) structure to more directly engage

PARM, Science and Technology Directorate, and other

oversight groups to establish a sustainable support model to

support components developing and acquiring IT solutions. The

support model should:

Define intake and exit criteria

Define service offerings and levels of support

Evaluate support needs of the component program to be

provided by headquarters

Align Component needs with provided offerings and skill sets

Enable “Just in time” support

Provide a dedicated project manager as staffing allows

This model also must place a limit on the number of programs

receiving support based on staffing bandwidth, and prioritize

requests in a headquarters Agile Acquisition Assistance queue.

Appendix II: DHS Agile Action Plans

Page 44 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Headquarters lines of business and component

programs do not have a clear understanding of the

roles, responsibilities, and equities of engaged

oversight bodies throughout the ALF and SELC

processes. Additionally, a lack of consistent

empowerment among integrated project team

members, sometimes without present and engaged

product owners and stakeholders, often resulted in

delays, rework, and inconsistent guidance to

programs during the Agile pilots.

Assess, revise, and communicate organizational roles and

responsibilities to Lines of Business and Components in

alignment with existing DHS policy. Develop recommendations

for enforcing lines of business boundaries, mitigate conflicting

responsibilities, and resolving cross-organization disputes.

Within program integrated project teams, validate that the

correct members and stakeholders are invited, engaged, and

empowered. Update the current ITPM COE charter, develop a

responsibility assignment matrix for each phase of the ALF and

SELC (and at the beginning of each new program support

engagement), and establish team rules for clear communication

channels. Use the forthcoming workflow tool to facilitate action,

roles enforcement, and communication.

In the transition to Agile development, traditional

approaches toward program oversight and

governance (along with related requirements placed

on programs) are often disconnected from the way in

which programs plan, develop, deploy, and

implement functionality. As a result, governance and

oversight bodies risk creating additional work for

programs and/or being disconnected from the work

being performed, solutions being developed, and

value being provided. This limits the effectiveness of

oversight groups in performing support functions as

a part of their governance role, potentially inhibiting

the overall intended mission of risk mitigation, value

creation, and enabling transparency.

Develop a revised governance approach that aligns with the

iterative nature of Agile and enables programs to establish

continuous integration continuous delivery pipelines and

emphasize communication of program planning, processes,

issues, and risks through naturally occurring artifacts.

Implement Agile governance and review models to increase

transparency and feedback throughout the obtain phase and

operations and maintenance. The updated DHS governance

and review models should account for portfolio, program, and

project level considerations. This will improve the value of

headquarters support as well as provide transparency in

program management and delivery.

For IT programs, the principle acquisition decision

points occur prematurely relative to the maturity of

the program and proposed technical designs. By

forcing premature decisions, programs and oversight

groups rely on assumptions of capabilities,

development plans, and risks. Additionally, existing

DHS policy definitions of initial operating capability

and full operational capability, and their alignment to

decision events, is not sufficiently defined for Agile IT

programs.

For IT programs, the Acquisition Decision Event 2A and 2B

decision points and related requirements should be adjusted in

relation to the SELC to ensure programs have adequate

information to support acquisitions. Programs must be enabled

to complete acquisition processes faster and with an

understanding that detailed engineering requirements will not

be determined up front. DHS policy and instruction should be

modified to adjust the decision points and related requirements

for IT software programs and clarify definitions of initial

operating capability and full operational capability for Agile

programs.

Appendix II: DHS Agile Action Plans

Page 45 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Traditional strategies for acquisition planning and

requirements generation across DHS and its

components have focused on large, upfront planning

with emphasis on a specific investment or technical

solution as opposed to focusing on the required

mission-based capabilities. This approach promotes

inefficient allocation of funding to specific IT

programs at the expense of desired capabilities;

overlap or redundant requirements within and across

the department; delays in requirement verification;

rigidity in development cycles; constraints on

development approaches and technical strategies;

and an inability to adequately plan, allocate, and re-

allocate resources to enable strategic investment

against evolving operational requirements. Current

expectations and requirements for upfront solution

definition and planning prevent programs from using

modern development strategies.

The implementation of the Joint Requirements Council and the

Joint Requirements Integration Management System reflects an

on-going change management process that continues to

mature. Components are designating Chief requirements

executives, establishing component-level requirements

structures and processes, and putting in place the necessary

human capital with competencies and capacity to fully support

the Joint Requirements Council and Joint Requirements

Integration Management System. DHS directives (071-02 and

107-01) and the Joint Requirements Integration Management

System Instruction Manual (107-01-001- 01) are the

authoritative guidance for the DHS requirements process.

Within the existing structure of the ALF, drive early artifacts to

focus on operational outcomes, gaps, needs and operational

level requirements, and not solution-specific details. Prevent or

discourage programs from defining functional requirements

within their early capability documentation (capability analysis

report, concept of operations, mission needs statement) to

enable solution flexibility. Programs and DHS Oversight groups

must use Acquisition Decision Event 1 and 2A documentation

to state the specific capability gaps, needs, and operating

concept gaps or improvements that will be derived from a given

investment, what critical decisions must be made through the

Analyze/Select Phase, what measurable outcomes will be

achieved, and define the epics and stories the program will

deliver.

In alignment with Enterprise Architecture (EA), support the Joint

Requirements Council in its efforts to assist Components and

programs in the upfront assessment of emerging operational

requirements. Focus early assessments on the desired impact

of reducing duplication of capabilities across the department

and within Components. Increase the role of EA in the early

requirements processes to ensure alignment to DHS IT

Strategy and awareness of existing IT assets through Portfolio

Teams, including Enterprise Management Portfolio Team, and

approaches. For artifacts other than the Capability Analysis

Report, Mission Needs Statement, Concept of Operations and

Operational Requirements Document, rely on the Science and

Technology Directorate Office of Systems Engineer, OCIO, EA,

and other subject matter experts within headquarters to ensure

sufficient oversight and program planning has occurred. Review

and propose updates and improvements to the Joint

Requirements Integration Management System process to

ensure it is executing as efficiently as possible.

Appendix II: DHS Agile Action Plans

Page 46 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Current DHS programs inconsistently implement

testing and evaluation. Guidance and policies do not

effectively support modern best practices in

automated testing and continuous integration.

Testing documentation including the Testing and

Evaluation Master Plan, Operational Test Agent, and

Authority to Operate are developed at different times,

resulting in outdated or misaligned criteria. Testing

requirements and procedures are not fully integrated

into the development pipeline and program

documentation must be continuously updated to

reflect changes. These conditions result in increased

audit risk and administrative workload, as well as

delays in releases of functionality.

Review and modify Management Directive-026 Test and

Evaluation Master Plan requirements to explicitly enable

programs to pursue the integration of test and evaluation

processes into the development pipeline. Support programs in

the adoption of modern best practices for automated testing

and continuous integration by developing a DHS wide strategic

view of integrated testing practices and case examples of

successful programs. Engage Components in the establishment

of approved processes to provide programs the ability to utilize

a continuous or ongoing Authority to Operate for iterative

development and releases.

The analysis of alternatives solutions is cumbersome

and unnecessarily delays the time to market for

required capabilities. Current requirements for

Analysis of Alternatives should be scalable based on

the needs and resource requirements of the

program.

Develop department level best practices and supporting

strategies for engaging with industry to build market intelligence

to gain insight into private sector capabilities and practices.

Update the DHS acquisition guidance and policy to streamline

the processes for evaluation of technical solutions and vendors

to include opportunities for programs to utilize a lean Analysis of

Alternatives. Provide clear and consistent guidance to programs

developing an Analysis of Alternatives, including opportunities

to tailor document and process requirements, conduct a lean

Analysis of Alternatives, or perform an Alternatives Analysis.

Currently, all Major Acquisition Oversight List

programs require an Acquisition Decision Event 2A

decision to establish the overall Acquisition Program

Baseline cost, schedule, and performance

parameters. This Acquisition Review Board decision

precedes the program’s development, testing, and

evaluation of their selected alternative solution to

meet the gap in the business capability. A total

program life-cycle cost estimate is required at this

decision to support the Acquisition Program

Baseline. Agile programs seek to make incremental

solutions towards a business capability without an

overall program solution defined years in advance.

If Agile programs had a modified agile governance

process and if the principal acquisition decision

points were modified to better align decisions with

the agile incremental approach, then the life-cycle

cost estimate can be scoped to address these

incremental decisions needs and the effort to

develop a life-cycle cost estimate would be reduced.

Following the efforts and accomplishments of Action Plan 5 and

6, Office of the Chief Financial Officer and OCIO will coordinate

adjustments to the cost estimating processes to align to the

adjusted governance process. The life-cycle cost estimate will

need to continue to support department budget decisions and

acquisition decisions for agile programs. The acquisition

decisions are expected to focus on incremental updates that

deliver a standalone business capability. The budget decisions

require estimated costs for the program to address the Future

Year Homeland Security Program.

Appendix II: DHS Agile Action Plans

Page 47 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Existing DHS guidance and polices do not include

proper requirements or instructions for programs at

various stages of ALF to properly incorporate DHS

cybersecurity policies within the required acquisition

artifacts. It has been shown that programs,

especially in early ALF stages, would benefit from

properly refined and repeatable instruction/guidance

on how to incorporate the policies when formulating

the program’s artifacts (e.g., Preliminary Mission

Needs Statement, Mission Needs Statement,

Capability Development Plan, Operational

Requirements Document, Analysis of Alternatives,

Concept of Operations).

Establish a working group incorporating all stakeholders with

the stated goal of reforming the existing DHS cybersecurity

policies in order to provide instructions and guidance on how

programs properly apply cybersecurity policies during the

acquisition artifact development process. A full mapping, value

stream analysis, and review of the ALF documents overlaying

cybersecurity activities as mandated in DHS policy. It is critical

for this working group to be enabled with proper authority.

Without a clear picture of how the entire ALF and

SELC process currently operates and should operate

in the future, it is difficult to identify the projects that

will close the gap.

Capture and map the ALF and SELC value stream. The power

of value stream mapping lies in looking at an entire business

process. It is critical to have this overall perspective for

selecting what projects to tackle. Value stream mapping not

only includes defining the current state, but also includes

defining the ideal and future state and the gaps between them.

By defining the overarching goal for the ALF and SELC

process, IT programs can guide and drive the design.

Multiple documents within the DHS ALF and SELC

contain overlapping or duplicative information and do

not align with Agile methodologies. Requirements

placed on programs to repackage or recollect

information into new artifacts for department level

oversight bodies forces rework for programs,

increases administrative burdens, and risks delays to

Acquisition Decision Events. Providing Agile clarity

around all ALF and SELC artifact requirements and

development would increase first pass approval.

Review and modify all ALF and SELC required documents and

artifacts. Create templates and have ‘best in show’ examples

where possible to provide clarity on the critical thinking, base

content, and format for Acquisition documentation.

Allow for greater flexibility in program documentation, including

the requirements for recollecting and repackaging existing

program documentation for select artifacts by consolidating into

core artifacts. Review each document with the appropriate

document owner(s) to ensure it contains the minimal actionable

information required to make an informed go/no go decision for

the Acquisition Review Board (or other appropriate decision

event), enabling programs to focus on solution development as

opposed to comprehensive documentation.

Programs are required to develop key performance

parameters as part of their Operational

Requirements Document, but the key performance

parameters are often unclear or difficult to track.

Additionally, while programs each track their own

metrics, there is not a set of core metrics to help

oversight bodies gauge the performance of the IT

development.

Provide additional guidance on how to develop metrics.

Develop a set of core metrics needs to be collected and enable

improved tracking of Agile IT development.

Appendix II: DHS Agile Action Plans

Page 48 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Currently, there are no IT EA touchpoints within

existing policies such as Management Directive-103-

02. There is also a lack of enterprise and component

target architecture plans and diagrams illustrating

DHS-wide enterprise architecture and system inter-

dependencies to support programs as they integrate

with the mission and one another.

Create an enforcement mechanism for IT EA touchpoints within

existing policies such as Management Directive-103-02.

Establish DHS target architectures, and require each program

to articulate their target architecture, so that as programs

implement Agile, the critical program materials and artifacts can

be reviewed at a higher level, ensuring EA practices and

principles are embedded. Develop an Enterprise wide blueprint

depicting the interrelationship of systems and critical

dependencies.

Programs struggle to identify, source, and select

suitable Operational Test Agent vendors in an

efficient manner that delivers the appropriate level of

support within the cost, schedule, and technical

requirements of the program. Programs face delays

due to Component acquisition processes and

required involvement of Science and Technology

Directorate Director of the Office of Testing and

Evaluation review of Operational Test Agents

through the vendor selection and award process.

Clarity around Operational Test Agent vendors would

eliminate the need for rework or re-contracting.

Programs also face difficulty in utilizing Operational

Test Agents and test and evaluation processes in a

manner that supports continuous development and

delivery.

DHS’s Science and Technology Directorate Director of the

Office of Testing and Evaluation should collaborate with the

Office of the Chief Procurement Officer Strategic Sourcing

Program Office to establish strategic sourcing vehicles to

provide accelerated access for programs to pre-approved

Operational Test Agent vendors and services. Offering use of

multi-award Blanket Purchase Agreements Indefinite Delivery

Indefinite Quantity contracts for Operational Test Agent

services will provide rapid access to Operational Test Agent

vendors aligned to Science and Technology Directorate

objectives, drive cost efficiency through consolidated

procurement vehicles, and increase visibility into Operational

Test Agent related spending at the department level. Direct

programs to engage the Science and Technology Directorate

early in the Analyze/Select Phase and involve the Operational

Test Agent / test and evaluation subject matter experts in

development of Operational Requirements Document and Test

and Evaluation Master Plan documentation.

Headquarters organizations, components, and

programs lack a defined method to deliver

consistency in the way products are developed and

delivered, the extent to which Agile best practices

have been adopted across the organization, and the

general health of their programs. These groups also

lack a common framework to measure their abilities

in these areas, to strategically request support for a

given deficiency, and to develop action plans

targeted towards addressing the specific gaps.

Uniformity is also needed in providing program

evaluations against the Office of Management and

Budget CIO Evaluation requirements described in

the Annual Budget-Capital Planning Guidance and to

support DHS CIO in meeting Federal IT Acquisition

Reform Act (FITARA) requirements.

Codify, implement, and apply the software delivery maturity

model, Agile maturity model, and Integrated program health

assessments to DHS component organizations and programs.

By establishing DHS-level maturity models for organizations

and programs in these areas, the agency and its components

will have common insight into process and delivery maturity

capabilities across the enterprise. Implemented maturity models

will also support intake processes for future Agile program

support, provide a logical road map towards capability

improvements in software development and agile adoption, and

facilitate DHS Digital Transformation. The maturity models also

will improve the capacities and capabilities of DHS OCIO to

conduct program health assessments of all major IT and special

interest investments.

Appendix II: DHS Agile Action Plans

Page 49 GAO-20-213 Agile Software Development

Action plan

number Problem statement Recommendation

Current processes for analyzing requirements

against existing IT capabilities within the DHS space

or in evaluating program requirements traceability to

mission needs or enterprise standards is highly

dependent on manual review and analysis.

Performance of manual program screening and

document review is time intensive and poses risks

associated with manual processing. Oversight

bodies are also limited in their ability to focus on

high-value analysis due to resource constraints in

data collection, document screening, and preliminary

reviews. Additionally, the review process does not

produce reusable products to inform future analysis

or determination. Oversight groups are also hindered

by incomplete or low fidelity data residing in reporting

systems (for example: the Investment Evaluation,

Submission, & Tracking system). Lack of visibility

into capabilities and requirements across the

department also restricts the ability of the

department and its components to pursue portfolio

level management against mission needs and

available resources and to make assumptions of risk,

complexity, and cost of future programs based on

historical insights.

OCIO (OCTO/EA) should pursue text and business analytics

solutions and leverage automation capabilities to increase

effectiveness of program analysis, planning, and oversight

reporting. The ideal state of operations would enable analysts

and decision makers to have comprehensive, data based

insight into existing capability gaps, mission needs, existing

solutions, and program profiles that utilizes visualizations and

reporting to inform decisions, identify dependencies across

programs and mission spaces, and monitor incoming

requirements for completeness, overlap, and alignment to

department and component objectives. The data solution

should account for existing DHS systems of record, program

health and maturity assessments, and ingestion of ALF and

SELC document text and meta-data.

Source: DHS Recommendations Action Plans: Agile Acquisition Pilots | GAO-20-213

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 50 GAO-20-213 Agile Software Development

This appendix describes in detail our evaluation of the three leading

practices for agency environment when adopting Agile development,

including a further explanation of expectations for each practice as well as

some of the findings associated with each practice. We do not present

any additional recommendations from these findings; this information is

intended to assist the Department of Homeland Security (DHS) in

implementing the recommendations described in our report.

Agency environment refers to leading practices related to an agency’s

processes, culture, and acquisition strategies. For an agency to

successfully transition from an environment that supports traditional

development methods, it should ensure that the

• activities support Agile methods by

• establishing appropriate life cycle activities

• clearly align goals and objectives

• culture supports Agile methods through

• cascading sponsorship for Agile software development

• sponsorship understanding of Agile software development

• establishing an environment supportive of Agile development

• aligning incentives and rewards to Agile methods

• acquisition policy and procedures support Agile methods

Establish appropriate life cycle activities

Agency activities should support Agile methods by allowing for

incremental and iterative software delivery that is tailored to the cadence

of Agile software development and by incorporating technical reviews that

occur throughout the development process. These activities and

supporting policy and guidance should allow for requirements to be

changed during development and the requirements change approval

process should not impede the cadence of iterative and incremental

development. Life cycle activities should also be user-focused and call for

collaboration between the development team and users.

To manage its multi-billion dollar investments, DHS has established

policies, procedures, and guidance for IT program management. These

publications govern the complete life cycle of a system, from technology

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Agency activities support Agile

methods

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 51 GAO-20-213 Agile Software Development

development through integration and testing and, finally, implementation

and operations and maintenance.

DHS has outlined its policies, procedures, and guidance in several

documents to assist its components in the acquisition and implementation

of software development. Policies for managing its major acquisition

programs are primarily set forth in a directive and supporting instruction.

These policies outline an acquisition life cycle framework (ALF) that

includes a series of predetermined milestones—known as acquisition

decision events—at which the Acquisition Decision Authority reviews a

program to assess whether it is ready to proceed to the next phase of the

ALF. DHS’s Under Secretary for Management serves as the Acquisition

Decision Authority for the department’s major acquisition programs.

A separate DHS instruction and associated guidebook outline a

framework of major systems engineering activities and technical reviews,

collectively considered the systems engineering life cycle (SELC), which

should be conducted by all DHS programs, both major and non-major.

The SELC helps to ensure that appropriate systems engineering activities

are planned and implemented and that a program’s development effort is

meeting business needs. The SELC consists of major activities and a set

of related technical reviews and artifacts that fit within the acquisition life

cycle.

Figure 2 depicts the acquisition life cycle and associated technical

reviews established in DHS acquisition management policy.

DHS issued the initial versions of the directive and instruction in November 2008 and has

subsequently issued multiple updates—the current version of the directive in February

2019 and the current version of the instruction in May 2019. Combined, these documents

are intended to provide a framework for consistent and efficient management of DHS’s

major acquisition programs. However, they also provide the Acquisition Decision Authority

the flexibility to tailor the framework for programs as needed.

Department of Homeland Security, Instruction 102-01-103, Systems Engineering Life

Cycle, Revision 00 (Nov. 5, 2015); Guidebook number 102-01-103-01, Systems

Engineering Life Cycle Guidebook, Revision 00 (Apr. 18, 2016).

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 52 GAO-20-213 Agile Software Development

Figure 2: DHS Acquisition Life Cycle Framework (ALF) and Systems Engineering Life Cycle (SELC) for Major Acquisition

Programs

DHS provided programs with flexibility in their SELC technical reviews.

Within the ALF, Agile processes are applied primarily within the obtain

phase, where design, development, testing, and implementation of a

system takes place. Prior to entering the obtain phase, a program selects

its software development approach, such as Agile. The agreed-upon

approach is then codified in an SELC Tailoring Plan, which is approved at

acquisition decision event 2A. The SELC Tailoring Plan identifies the

technical reviews and artifacts that the program is responsible for

completing based on its unique characteristics (e.g., scope, complexity,

and risk).

To assist in tailoring efforts and further guide the implementation of Agile,

DHS published an Agile instruction that includes the scope, definitions,

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 53 GAO-20-213 Agile Software Development

roles and responsibilities, and procedures for establishing an Agile

framework for developing all DHS IT acquisitions.

DHS supplemented

this instruction with an Agile instruction manual and provided a template

that Agile programs can follow to tailor their activities.

For example,

instead of holding a system definition review, an Agile program is

encouraged to conduct a release planning review (which encompasses

the development and release of a segment of software). This optional

approach to tailoring a technical review is depicted in figure 3.

Department of Homeland Security, Instruction 102-01-004, Agile Development and

Delivery for Information Technology, Revision 01 (Apr. 16, 2018).

Department of Homeland Security, Instruction Manual 102-01-004-01, Agile

Development and Delivery for Information Technology Instruction Manual, Revision 00

(Jul. 15, 2016) and Systems Engineering Center of Excellence, SELC Tailoring Examples

for Selected Types of DHS Acquisition Programs, Version 2.0 (Nov. 2016).

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 54 GAO-20-213 Agile Software Development

Figure 3: Optional Tailoring of the DHS Acquisition Life Cycle Framework (ALF) and Systems Engineering Life Cycle (SELC)

for Agile Major Acquisition Program

Outside of technical reviews, DHS updated acquisition policy in February

2019 and associated guidance in May 2019 to allow programs greater

flexibility in the larger ALF. The Director of Strategic Technology

Management (STM) stated that, under the previous acquisition policy and

guidance, IT programs were using in-house expertise due to limited

funding to prepare for the 2B decision, when full program funding was

received. He noted that, by the time a contract was awarded for

development following a 2B decision, the contractor might or might not

have been using planning artifacts developed by the program and instead

might have recreated them, thereby rendering 2 to 3 years of work

useless. The Director stated that programs were unable to fully flesh out

the program architecture and other key aspects of the program because

programs did not receive funding until the 2B decision and in-house

expertise was limited. For example, if a program had not proven out its

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 55 GAO-20-213 Agile Software Development

architecture prior to a 2B decision, it could continue to refine and modify

the architecture during the course of development, thereby impacting

productivity and quality. DHS updated acquisition management policy and

guidance to modify the requirements for the acquisition decision events

and addressed a related GAO recommendation.

DHS policy and guidance also allowed for programs to modify

requirements over the course of development. The traditional process for

requirements may be modified as part of tailoring the SELC in order to

allow for increased flexibility. The DHS Requirements Engineering User’s

Guide detailed requirements engineering steps, activities, and methods

for performing those steps.

DHS developed this user guide to supplement SELC policy and guidance.

One section of this guide focused on Agile development. According to the

guide, requirements are broken down over the course of the ALF and

commitments are made at different levels of specificity. Fundamental

capability gaps are defined in the mission needs statement presented at

acquisition decision event 1. Subsequently, the analyze/select phase

would ultimately define the high-level features and functions of each

required capability, define the fundamental performance of those high-

level features and functions, and establish the business case to support

approving the acquisition at an acquisition decision event 2A. Often, a

preliminary concept of operations is developed and delivered with the

mission needs statement.

The guide also states that the activities to evaluate these potential

alternatives will ultimately result in a preferred solution with defined

business practices, methods, and processes that allow the development

of business epics and associated architecture epics. Business epic is an

Agile term that defines the high-level “stories” that describe a capability,

or what the new system is required to perform. Architectural epic is an

Agile term that defines the architecture the system will be incorporated

into. In addition, the preferred solution would have defined high-level

GAO, Homeland Security Acquisitions: Earlier Requirements Definition and Clear

Documentation of Key Decisions Could Facilitate Ongoing Progress, GAO-17-346SP

(Washington, D.C.: Apr. 6, 2017). In this report, GAO recommended that the Secretary of

Homeland Security direct the Undersecretary for Management to require that major

acquisition programs’ technical requirements be well defined and key technical reviews be

conducted prior to approving programs to initiate product development and establishing

acquisition program baselines, in accordance with acquisition best practices. This

recommendation was intended to mitigate the risk of poor acquisition outcomes and

strengthen the department’s investment decisions.

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 56 GAO-20-213 Agile Software Development

performance requirements (stated from the operational perspective) in

terms of how well the solution must perform to be operationally effective

and suitable. Key constraints such as security, Section 508 compliance,

privacy, reliability, etc., should also be identified. These top-level

requirements will be documented in the operational requirements

document.

According to the guide, Agile teams capture the capabilities and

constraints (essentially the functional and non-functional requirements

that reflect the business epic level of performance) in an artifact called the

capabilities and constraints document. Requirements statements in this

document should follow the standard “shall statement” format for ease of

translation between the operational requirements document and the

capabilities and constraints document. The capabilities and constraints

document and its contents mature over time and, as the document

matures, business and architectural epics decompose to

features/functions or themes, and ultimately to user stories that reflect the

specific tasks that users will perform. Officials within the DHS Joint

Requirements Council noted that headquarters involvement occurred at

this level to approve the high-level operating requirements.

After headquarters oversight and approval, the program may then

decompose requirements as part of planning for and executing technical

reviews. If tailored into an Agile program, the capabilities and constraints

document should drive the development of a backlog. The backlog is a

list of all the user stories that describe what the system needs to do. The

backlog should become more refined as the program decomposes the

high-level features (a service that fulfills a user need) and functions down

to specific stories that an individual software developer will code and test

during a specific iteration.

To prevent the backlog from becoming unmanageable, DHS guidance

stated that backlogs may be established at different levels. For example,

the business and architectural epics along with the associated operating

requirements would constitute the “program backlog.” Sub-epics are

usually broken down into “high-level features” with business epics broken

down into business features and architectural epics broken down into

architectural features. Features or functions are decomposed into detailed

stories that are then allocated to a “release”. The list of user stories in a

specific release constitutes the release backlog. This process of

decomposing stories continues to the iteration backlog.

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 57 GAO-20-213 Agile Software Development

DHS guidance places an emphasis on end user needs. The

Requirements Engineering User’s Guide raised the importance of

identifying stakeholders, including system users, and capturing the needs

of those users via requirements or, in the case of Agile, user stories. The

Agile instruction manual placed an emphasis on the importance of users

to a program and articulated that the product owner represent the user

community and was expected to continually seek ongoing feedback and

elicit requirements from users. The Agile core metrics also strongly

recommended the use of a net promoter score. This score was one

mechanism for measuring customer satisfaction through asking users to

rank how likely they would be to recommend the system or application to

a friend or colleague, based on a score of 1 to 10.

Clearly align goals and objectives

Program goals should clearly reflect stakeholder needs and concerns

based on input from stakeholders and stakeholder review and approval.

Program goals should align with strategic IT objectives. Software-related

goals should be defined and clearly aligned with program goals. The

agency should collect objective measures that are well defined to track

progress towards achieving software goals so the agency knows which

features and capabilities have been achieved.

The Requirements Engineering User’s Guide described program

expectations for tracing from mission needs to operating and functional

requirements, or user stories. The guide recognized that, as a program

progressed through the ALF and SELC, it was important to trace

requirements from the top-level mission needs or capabilities and/or

business requirements down to the system/sub-system, component, or

configuration item level that enabled those requirements to be met. This

helped ensure continuity across various DHS artifacts, such as the

program’s mission need statement, concept of operations, and

operational requirements document, to vendor specifications (or

applicable equivalent artifacts). Although an Agile program will modify the

SELC to accommodate its needs, generally programs were expected to

follow the same conceptual approach to the requirements of planning,

development, and management.

The user’s guide stated that collaboration among the various

stakeholders was important and the program requirements team must

continuously work to establish partnerships and networks. To do so, the

guide stated that the program team must identify all individuals and

organizations that may be impacted by their program and ensure those

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 58 GAO-20-213 Agile Software Development

stakeholders were engaged throughout development to facilitate

understanding of their perspectives and needs. The first step was to

identify applicable stakeholders, which would include end users, program

sponsors, developers, maintainers, trainers, and other affected individuals

or organizations. The program requirements team then solicited input

from these stakeholders to understand their needs, policies, processes,

and operations to begin the requirements definition effort. It identified

some ways a team might begin the process of eliciting requirements from

the stakeholders.

After collaborating with stakeholders, the stakeholder needs must be

translated into the program requirements, or goals. The guide stated that

the program requirements team should take the inputs from the various

stakeholders and decompose, prioritize, de-conflict, and validate the

needs identified. It clarified that a “good” requirement was achievable,

testable, clear, concise, technology-independent, feasible, and able to

stand alone.

The guide grounded all of the requirements elicitation and development

process in the overall contribution to the agency mission, recognizing the

need for general strategic alignment. In particular, the guide noted that

requirements were “mission need” driven as opposed to “solution” driven.

Requirements were developed throughout the life of a program, with the

first formal requirements being the operating requirements documented in

the operational requirements document.

To ensure that DHS’s mission or strategic goals were key inputs for

decision making, DHS relied, in part, on its enterprise architecture

process. DHS policy for enterprise architecture stated that the enterprise

architecture program provided a vehicle to tie the strategic mission goals

and objectives of DHS to the business processes, information resources,

and technology investments necessary to reach key performance

outcomes. This methodology was intended to integrate IT into the mission

and strategic priorities of DHS, which provided the core foundation for all

subsequent processes. DHS capital planning and investment control

guidance reinforced this fact, stating that the Federal Acquisition

Streamlining Act of 1994 required capital investments to align with

mission and strategic goals. This included the framework within which the

department formulated, managed, and maintained its portfolio of

investments as critical assets for achieving success in the DHS mission

and alignment to the DHS IT Strategic Plan and the DHS Strategic Plan.

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 59 GAO-20-213 Agile Software Development

Cascading sponsorship for Agile software development

Senior stakeholders should support and model the use of Agile, along

with its values and principles, through explicit policy or guidance

impacting the business and should take steps to complete responsibilities

defined in agency Agile policy or guidance. Agile should also be

supported in all relevant areas of the business impacting a software

development project through the use of Agile sponsors. These sponsors

should represent the lines of business in key agency decisions on Agile.

Senior stakeholders at DHS demonstrated support for Agile through the

publication of policy and guidance that established Agile development as

the department’s preferred approach for software development. As

discussed previously, the department published Instruction 102-01-004

Agile Development and Delivery for Information Technology (Agile

instruction), which provided the scope, definitions, roles and

responsibilities, and procedures to establish an Agile framework for the

development of IT acquisitions at DHS.

Specifically, the Agile instruction

established responsibilities for the CIO, the Chief Procurement Officer,

the Chief Financial Officer, the Director, Office of Test and Evaluation

within the Science & Technology Directorate, and the Executive Director

of PARM.

Each of these five stakeholders and their associated components

demonstrated their support for Agile development by taking steps to

complete their responsibilities defined in the Agile instruction. For

example, the Office of the Chief Information Officer (OCIO), the Office of

the Chief Procurement Officer, and the Director, Office of Test and

Evaluation within the Science and Technology Directorate all had

responsibilities related to providing guidance for the implementation of

Agile within their specific area of expertise. All three components had

taken steps to execute these responsibilities, such as by publishing the

Agile instruction manual, providing supplementary guidance for test and

evaluation in an Agile environment, and offering elective training on

contracting strategies for Agile services.

Representatives from offices with a role in software development also

supported Agile via membership in the IT Program Management Center

of Excellence (ITPM COE). In addition to the stakeholder organizations

Department of Homeland Security, Instruction 102-01-004, Agile Development and

Delivery for Information Technology, Revision 01 (Washington D.C.: Apr. 16, 2018).

Agency culture supports Agile

methods

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 60 GAO-20-213 Agile Software Development

identified in the Agile instruction, the ITPM COE membership included

representatives from the Joint Requirements Council

and the Chief

Privacy Officer.

According to the ITPM COE charter, the ITPM COE

served as a cross-functional team to identify and promote best practices,

provide tools, and coordinate assistance for programs and projects to

maximize the successful management of DHS IT investments. This

included making progress towards the 18 Agile action plans that resulted

from the Agile acquisition pilots.

The ITPM COE membership requirements called for representatives of

the member organizations to be involved in key decisions regarding Agile.

According to the Director of STM within OCTO, ITPM COE members

were selected and approved by their organization’s executives. The ITPM

COE charter stated that these representatives must be authorized to

represent or make decisions on behalf of their officers or organizations.

Officials from all ITPM COE member organizations expressed support for

the ITPM COE and confirmed that their component was appropriately

represented in decision making. This was represented, in part, by the fact

that at least one representative for each ITPM COE member group

attended at least half of the meetings. For example, at least one

representative from the Science and Technology Directorate attended

approximately 95% of the meetings.

According to Directive 102-01, Acquisition Management, the Joint Requirements Council

provides oversight of the DHS requirements generation process, harmonizes efforts

across the department and makes prioritized recommendations to the Deputy’s

Management Action Group for those validated requirements. The Deputy’s Management

Action Group provides recommendations to the Deputy Secretary for consideration in the

annual program and budget review, which reflects DHS’s investment priorities. The

Deputy’s Management Action Group reviews Joint Requirements Council-validated

capability needs and recommendations; provides direction and guidance to the Joint

Requirements Council, and endorses or directs related follow-on Joint Requirements

Council activities.

According to Instruction 102-01-001, Acquisition Management, the DHS Chief Privacy

Officer is responsible for, among other things, reviewing and approving privacy

compliance documentation for DHS acquisitions and ensuring that the department follows

DHS privacy policy, applicable privacy laws, and government-wide privacy policies for

collecting, using, maintaining, disclosing, deleting, or destroying personally identifiable

information.

Department of Homeland Security, Charter Information Technology Program

Management Center of Excellence (ITPM COE), Version 2.1 (Washington D.C.: May 15,

2018).

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 61 GAO-20-213 Agile Software Development

Sponsor understanding of Agile software development

Sponsors should understand and communicate changes resulting from

Agile development. Sponsors should attend training or receive coaching

on Agile and the agency’s framework, the agency should monitor

completion of training, and sponsors should transmit learning from

training to staff. Sponsors should also commit to achieving those intended

results and sponsor performance should be tied to achieving those

intended results.

The Director of STM stated that Agile sponsors were considered to be

chief executive officers (e.g. Executive Director of PARM and the Deputy

Under Secretary for Management). They oversaw the actions of the ITPM

COE and approved the Agile action plans in June 2017.

DHS did not ensure that Agile sponsors attended training or received

coaching in Agile development. The department made training available

for Agile, including courses such as those required for acquisition

professionals. However, in a written response, the Office of the Chief

Human Capital Officer stated, and the Director of STM confirmed, that the

department did not administer mandatory training on Agile for Agile

sponsors.

The department also did not monitor the completion of sponsor training in

Agile. Although DHS employees leveraged the Federal Acquisition

Institute Training Application System to track their training and

certifications, the department was not using this system to monitor

sponsor training in Agile. According to a written response from the Office

of the Chief Human Capital Officer, the department did not keep a record

of whether sponsors completed training in Agile because the department

did not require Agile training specifically for sponsors.

DHS Agile sponsors exhibited support for achieving the intended results

from the transition to Agile. Agile sponsors committed to achieving these

results through an endorsement of the 18 Agile action plans and the

associated implementation plans.

However, DHS did not demonstrate that Agile sponsor performance was

tied to achieving the intended results of the transition to Agile. According

to a written response from the Office of the Chief Human Capital Officer,

the department’s employee performance management policy did not

specifically address Agile. This written response further stated that

addressing Agile in these policies was unnecessary because the Office of

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 62 GAO-20-213 Agile Software Development

the Chief Human Capital Officer incorporated goals derived from project

plans in individual performance plans. DHS policy and guidance for

performance management identified individual performance goals as a

component of employee performance, but the department did not provide

evidence that specific performance plans for the sponsors were linked to

such goals.

Establish an environment supportive of Agile software development

Team dynamics should be facilitated through access to common team

rooms and/or modern communication and social media methods and

headquarters infrastructure operations should allow for communal spaces

and co-location in program offices. A headquarters technical environment

should allow access to tools by programs to foster distributed

communication, and there should be a process for continuous feedback

on the Agile environment and modifications to that process (e.g.

communities of practice, routine working group sessions). Agency

governance bodies should allow programs greater autonomy and

flexibility within existing acquisition processes through the modification of

gate reviews and other touchpoints in the acquisition process for Agile

projects and increased transparency for governance bodies into project

operations when necessary.

DHS policy and guidance allowed for team dynamics to be facilitated

through access to common team rooms and modern communication

methods. In addition, department policy promoted and allowed program

offices to support team dynamics through the use of communal spaces

and co-location. Specifically, the Director of Systems and Information

Integration within the Chief Readiness Support Office confirmed that DHS

had modified policy related to infrastructure operations to allow any office

to reorganize their space, citing the USCIS Transformation program as an

example of this reorganization. The Director of Systems and Information

Integration also noted that he was not aware of any restrictions to this

practice in policy. With respect to facilitating access to modern methods

of communication, DHS offered programs the option of using a suite of

tools that included those for distributed communication.

DHS took multiple steps to establish a process for continuous feedback

related to the department’s Agile environment and process modifications.

According to the Director of STM, OCIO built support for Agile through the

Centers of Excellence, communities of interest, brown bag lunches, and

public speaking engagements. The Director added that these sessions

facilitated the discussion of Agile and could be used to compile feedback.

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 63 GAO-20-213 Agile Software Development

The Director of STM explained that, as this feedback came in, it was

either addressed immediately or put into a backlog. Efforts to further

streamline the acquisition process were tracked via Agile action plan 6.

The department’s governance bodies also increased transparency into

project operations when necessary. The Agile Development and Delivery

for Information Technology Instruction Manual (Agile instruction manual)

stated that the program or project manager should coordinate with the

various oversight bodies that govern IT development. These bodies

varied depending on the level of investment, but, for major programs,

executive steering committees were often established to oversee all

aspects of program planning and execution between major acquisition

decision events. In addition, PARM officials stated that DHS increased

the frequency of acquisition review board reviews and modified the

content presented at the reviews to allow it to be more actively involved

with projects earlier in the acquisition life cycle. Specifically, PARM

updated the Acquisition Review Board slide templates and informed us of

its intent to update acquisition management policy to require Agile

projects to hold Acquisition Review Board reviews once every six months,

as opposed to once every 12 months.

Align incentives and rewards to Agile methods

The agency should establish an incentive and reward structure promoting

team successes and the value of individuals within those teams.

Management should establish agency goals to align incentives and

rewards with Agile methods. Goals for incentives and rewards should

align with the agency’s goal(s) and focus on team success. The agency

should allocate incentives and rewards based on team success.

DHS did not establish an incentives and rewards structure that promoted

team successes and did not demonstrate that management had

established agency goal(s) to align incentives and rewards with Agile

methods. Furthermore, the department did not demonstrate that human

resources and others were actively involved in setting goals for incentives

and rewards alignment.

DHS guidance specifically discussed contract incentives for Agile

projects. For example, the Agile instruction manual suggested that

consideration be given to address the duration of the base term and

options, scalability, deliverables, and pricing with a mindset that

contractors need appropriate incentives to encourage them to perform

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 64 GAO-20-213 Agile Software Development

well. The manual also stated that contract award terms could provide a

greater incentive for contractors working on longer-term Agile projects.

Although the department made efforts to adapt incentives and rewards for

contractors supporting Agile projects, it acknowledged that it did not

update existing incentives and rewards for federal employees working on

Agile projects. Officials within the Office of the Chief Human Capital

Officer stated that existing human capital and performance plan policy

allowed for rewarding and incentivizing Agile teams as well as individuals.

These officials further noted that DHS had numerous opportunities to

recognize and reward team or individual performance, regardless of the

development methodology a program relied on. Specifically, these

officials clarified that the Office of the Chief Human Capital Officer used

project plans to set goals and included those goals in employee

performance plans. As these officials felt the existing performance plan

policy was sufficient, they did not believe additional guidance or

modifications to existing policy were necessary.

Guidance is appropriate for Agile acquisition strategies

Agency acquisition policy and guidance should support awarding

contracts for the unique needs of an Agile program. Acquisition strategies

should recognize the need for interim delivery of software, allow for close

coordination between the contracting office and program office staff, and

allow for changing requirements and contract oversight mechanisms to be

tailored to support Agile development methods.

DHS offered guidance for preparing acquisition strategies through its

Procurement Innovation Lab.

Webinars offered by the Procurement

Innovation Lab on acquisition strategies for Agile programs discussed the

need for interim delivery of software, close coordination between

contractors and program office staff, contract oversight mechanisms that

were tailored to support Agile development, and changing requirements.

For example, the “Transportation Security Administration Agile Services

Procurement” webinar discussed planning, executing, and de-briefing

technical demonstrations used to select the contract recipient, paying

particular attention to the value of transparency and modifying contract

oversight mechanisms. Officials from the Office of the Chief Procurement

The Department of Homeland Security Procurement Innovation Lab, Office of the Chief

Procurement Officer, experiments with innovative techniques for increasing efficiencies in

the procurement process and institutionalizing best practices.

Agency acquisition policies and

procedures support Agile

methods

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 65 GAO-20-213 Agile Software Development

Officer clarified that the webinars were available as needed and were not

required training.

DHS also published Agile guidance that discussed contracting and

acquisition strategies. From an oversight perspective, according to the

Agile instruction manual, DHS executive steering committees oversee all

aspects of program planning and execution between acquisition decision

events. This authority extends to assisting programs in developing

acquisition strategies where appropriate. The manual included a section

that specifically called out Agile contracting considerations that pointed

back to Office of Management and Budget Contracting Guidance to

Support Modular Development, the TechFAR handbook, the Digital

Services playbook, and innovative contracting case studies.

Among other useful information in the Agile instruction manual were key

contracting considerations for an Agile program or project manager.

These considerations included, among other things, frequent, iterative

deliveries of software, an ability to monitor changes to maintain contract

and project scope, flexibility to accommodate refinement of requirements,

transparency and collaboration, and prior experience in the Agile

methodology. The manual also highlighted goals for the acquisition to

discuss with a contracting officer, such as rapid contracting processes to

keep pace with Agile development, contracting to accommodate

incompletely defined scope and requirements, and the ability to respond

to requirements changes without requiring extensive change orders.

According to officials within OCTO and the Office of the Chief

Procurement Officer, the department also supported Agile programs in

preparing acquisition strategies through the IT acquisition review process.

This process was established to provide a mechanism for the DHS Chief

Information Officer to review and guide agency IT expenditures. The

process was intended to analyze IT acquisitions to ensure alignment with

DHS missions, goals, policies, and guidelines. This process relied on

subject matter experts to assist in the review of IT acquisitions, including

one for Agile reviews. According to the IT Acquisition Review Essentials

Guide, Agile reviews occurred where software was being developed to

ensure development activities adhered to Agile best practices and DHS

SELC guidance. The Agile subject matter expert was expected to review

acquisition materials against an established set of criteria for both the

acquisition plan and the requirements document. For example, when

reviewing the acquisition plan for approval, the subject matter expert

should consider if the statement of need adequately addresses Agile or

iterative project-specific activities and/or deliverables.

Appendix III: Leading Practices for Adopting

Agile Development—Agency Environment

Page 66 GAO-20-213 Agile Software Development

The Director of STM stated that there is one staff member in STM who

actively participates in the IT acquisition review process and was

responsible for ensuring Agile language was correctly implemented in

contract statements of work. The Director also added that they were

willing to help teams that were having trouble providing explanations of

Agile processes in their statements of work. The Director of STM stated

that there was no policy to guide his staff member reviewing Agile

language in the statement of work, but that he asked his division to put

together a checklist review to govern this process. The Director added

that the department sent programs and projects requiring assistance with

Agile contracting to the Procurement Innovation Lab by request to

streamline the acquisition plan.

According to the Leader of the Procurement Innovation Lab Team, the

Office of the Chief Procurement Officer was primarily focused on

supporting Agile pilot programs, such as the Federal Emergency

Management Agency Grants Management Modernization program. The

team leader noted that, while the procurement office supported these

programs, it relied on the program offices to ensure accuracy. For

example, the program management office ensures that the requirements

are structured and delivered, which could be challenging for Agile

programs. The team leader mentioned that a particular focus at the

moment was defining the pricing for contract line item numbers in such a

way as to afford the flexibility needed for Agile development while still

holding contractors accountable.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 67 GAO-20-213 Agile Software Development

This appendix describes in greater detail our evaluation of the three

leading practices for program processes when adopting Agile

development. It does not present new findings; rather, the information is

intended to assist the Department of Homeland Security (DHS) in

implementing the recommendations described in this report.

Program processes refer to leading practices related to the program office

and technical environment. For programs to successfully transition from

processes used for traditional development projects, programs should

ensure that

• staff are appropriately trained in Agile methods by

• training all program staff

• ensuring Agile teams have the appropriate technical expertise

needed to perform their roles

• technical environments enable Agile development through

• making technical and project support tools available, and

• designing a system that supports iterative delivery

• project planning controls are compatible with Agile methods by

• maintaining a sustainable development pace and tracking and

monitoring that development pace

• defining and incorporating non-functional requirements in

development

• defining and incorporating critical features in development

The department develops an environment that supports these processes.

Within DHS, program management offices are responsible for planning

and executing individual programs and implementing applicable Agile

methodologies. In addition, the DHS Office of the Chief Information

Officer (OCIO) is responsible for setting policies and procedures to

ensure that programs leverage Agile development best practices to meet

the department’s goals and are within acquisition policy. The DHS OCIO

is also responsible for providing guidance for and reviewing the adoption

and execution of Agile development.

Train all program staff in Agile methods

The agency should provide a training program in Agile for staff and track

and monitor the training. All members supporting the team, not only the

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Staff are appropriately trained

in Agile methods

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 68 GAO-20-213 Agile Software Development

software development team, should be trained in the specific Agile

framework they will be using.

DHS required its acquisitions workforce to take training that incorporated

Agile methods. DHS Instruction 102-01-006, Acquisition Program

Management Staffing, established certifications for key acquisition career

fields, which included training requirements.

According to the Associate

Director for Training from the Homeland Security Acquisitions Institute,

the certification requirements included training that has been updated to

incorporate Agile methods. Specifically, the department updated course

content for AQN 101: DHS Fundamentals of Systems Acquisition to

include Agile development concepts, such as small team management

and Agile metrics, following the issuance of department policy governing

Agile development. This course was required training for seven of the

acquisition career fields, including program and project managers,

systems engineers, and test and evaluation managers.

DHS tracked and monitored the completion of training requirements for

the acquisitions workforce. According to DHS Directive 064-04,

Acquisition Professional Career Information, component acquisition

executives were responsible for ensuring that acquisition personnel met

the mandatory training requirements.

Officials from the Homeland

Security Acquisitions Institute within the Office of the Chief Procurement

Officer stated that DHS employees leveraged the Federal Acquisition

Institute’s training application system to track their training and

certifications. According to the catalog of product services of the institute,

members of the DHS acquisition workforce were required to attach copies

of their training certificates to request certification of completion of the

required training.

Because the DHS acquisitions workforce may not cover all personnel

staffed to Agile projects, some program staff may not be subject to

training requirements that incorporate Agile methods. According to the

Director of the Homeland Security Acquisition Institute, certain Agile team

members, such as the product owner, were not necessarily classified as

part of the acquisitions workforce. For example, according to the U.S.

Immigration and Customs Enforcement (ICE) Student and Exchange

Department of Homeland Security, Instruction 102-01-006, Acquisition Program

Management Staffing (Dec. 2, 2016).

Department of Homeland Security, Directive 064-04, Acquisition Professional Career

Information, Revision 00 (Oct. 30, 2008).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 69 GAO-20-213 Agile Software Development

Visitor Information System (SEVIS) program staffing plan, the product

owner role was not part of the acquisitions workforce and did not require

any certifications.

To help address the Agile training needs of all staff, including those who

are not part of the acquisitions workforce, DHS also provided elective

training in Agile methods. The department offered commercial training

through the Homeland Security Acquisition Institute, such as acquisition

of Agile services and Agile requirements for creating user stories. The

DHS Agile instruction manual also identified training offered by the U.S.

Citizenship and Immigration Services Office of Information and

Technology as another resource on Agile concepts, such as user stories

and automated testing.

In addition to elective training, the Agile Development and Delivery for

Information Technology Instruction Manual (Agile instruction manual)

encouraged program managers to seek out an Agile coach to help teams

adopt Agile methods and supplement training.

The instruction manual

suggested that program managers should identify an Agile coach to serve

as an embedded trainer, consultant, and team advisor. This Agile coach

could help the team adapt Agile methods to their environment and work

through challenges. An Agile coach could also help individual team

members understand the responsibilities of their role on an Agile team.

Although DHS did not provide coaches for Agile teams, the department

offered resources that could help programs select and obtain an Agile

coach. First, the department established a blanket purchase agreement

for programs to acquire Agile development support in the form of hands-

on coaching services for the design and use of Agile methods. According

to Homeland Security Acquisition Institute officials, this agreement would

enable programs and projects to acquire Agile coaching. Among other

things, this agreement defined the scope of Agile coaching services and

their pricing so that programs would not need to develop these terms on

their own. Second, the Agile instruction manual included considerations to

help program managers select a qualified Agile coach. For example, the

instruction manual encouraged program managers to collaborate with

contracting officials to identify an Agile coach who had demonstrated

Department of Homeland Security, Instruction Manual 102-01-004-01, Agile

Development and Delivery for Information Technology Instruction Manual, Revision 00

(Jul. 15, 2016). The Agile instruction manual defines an Agile coach as an individual who

has significantly broad experience applying Agile approaches to software and

development efforts.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 70 GAO-20-213 Agile Software Development

successful past performance on projects implementing similar technology

and Agile methodologies.

Case Study Example

The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit

(BEE) program’s Air Exit project provided informal training for new team

members that included a discussion of Agile methods. According to the

Air Exit project manager in the Office of Information and Technology, new

team members received onboarding training that covered CBP’s

approach to Agile methods. The project did not track attendance for this

onboarding training, but an Air Exit project manager noted that team

members were incentivized to attend the training in order to learn how to

satisfy their responsibilities. The Scrum master for the Air Exit project

stated that this training was also available to the team as a refresher

course approximately every fiscal quarter.

The BEE program also relied on an Agile coach to support the Agile

team. According to the Agile coach supporting the Air Exit project, this

role included training for the Agile team on basic Agile topics and working

with the team on their use of a project management software tool.

According to a project manager within the Office of Field Operations Air

Exit project management office, the Agile coach that supported the

project was instrumental in designing the CBP Office of Information and

Technology’s Agile development program beyond the BEE program.

Ensure Agile teams have the appropriate technical expertise needed

to perform their roles

The agency should have policy or guidance in place to help programs

ensure Agile teams have the appropriate technical expertise. A program

should also consider Agile-centric skills when forming teams. In addition,

programs should define requirements for contractor proposals and

evaluate contractor proposals for Agile services (e.g., source selection).

DHS guidance provided programs with considerations for forming teams

with Agile-centric skills. The DHS Agile instruction manual stated that a

development team with experience in Agile practices can mitigate risks to

on-time delivery. This experience included Agile processes as well as

technical skills, such as automated testing. In the context of the Agile

Scrum methodology in particular, the Agile instruction manual stated that

teams needed to be cross-functional and have all of the skills required to

deliver a project from conception to delivery.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 71 GAO-20-213 Agile Software Development

To enable teams to deliver a project from conception to delivery, the Agile

instruction manual stated that program managers should seek team

members with general skills. The manual advised that team members

should contribute to routine development activities and possess cross-

functional expertise that allows the team to achieve work without

depending on individuals outside of the team. For example, in Agile

development, testers are part of the development team and should

therefore possess both testing and development skills. In addition, the

instruction manual stated that, according to industry experts, program

managers should seek some overlap in team member’s skillsets to

mitigate risks associated with a key person becoming temporarily

unavailable.

DHS guidance further provided programs with considerations for defining

requirements in solicitations for contract proposals for Agile services. For

example, DHS supplemental guidance for incorporating testing and

evaluation into contract requirements noted that contracts should specify

government test and evaluation staff, as well as contractors, on the

development team in order to access the test data they need.

The DHS IT acquisition review process also helped to ensure that

requirements were defined in solicitations for contractor proposals.

According to the Information Technology Acquisition Review Essentials

Guide, Agile subject matter experts in the department review proposed

contracts to ensure that they will enable development activities that

adhere to Agile best practices and DHS systems engineering life cycle

(SELC) guidance.

For example, Agile subject matter experts should

assess whether contract requirements documents, such as the statement

Department of Homeland Security, DHS Supplemental Guidance: Incorporating T&E into

Acquisition Contracts, Version 1.0 (Jan. 2019).

According to the Information Technology Acquisition Review Essentials Guide, the

Information Technology Acquisition Review process was established to provide a

mechanism for the DHS Chief Information Officer to review all IT acquisitions and

contracts in accordance with the Federal Information Technology Acquisition Reform Act

of 2014. As part of this process, subject matter experts at the department level, such as

those for Agile and the SELC, review the documents associated with the acquisition

request for alignment with DHS Chief Information Officer’s IT strategy, and for compliance

with laws and policies governing DHS IT. The subject matter experts then make a

recommendation to the DHS Chief Information Officer regarding whether to approve,

approve with conditions, or disapprove the acquisition request.

Department of Homeland Security, Information Technology Acquisition Review (ITAR)

Essentials Guide, Version 6 (Oct. 2017).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 72 GAO-20-213 Agile Software Development

of work, are prepared in terms that will enable vendors to clearly

understand the Agile requirements.

The department also provided guidance to assist programs in evaluating

contractor proposals for Agile services. The Agile instruction manual

noted that programs can consider certifications in various Agile

methodologies and recommended that programs coordinate with

contracting officials to review vendors’ past performance in implementing

Agile methods.

In addition, the department established the Procurement Innovation Lab

within the Office of the Chief Procurement Officer to help programs

address challenges in procuring Agile services, such as validating

contractor qualifications. According to a Procurement Innovation Lab

team leader, the lab shares lessons learned from Agile services contracts

via webinars, which are available to staff on an as-needed basis. Several

of these webinars highlighted the value of using technical demonstrations

to validate the qualifications of vendors.

Case Study Example

The ICE SEVIS program provided training for all team members,

including contractors, to ensure they had the necessary Agile-centric

skills and expertise. A team process agreement for one development

module showed that the technical lead, development team, test engineer,

and Scrum master roles were filled by contractors, while other positions

such as the project manager, product owner, and test automation subject

matter expert roles were filled by government employees.

According to

the ICE SEVIS program manager and Scrum master, the program

provided training for contractors that covered Agile processes as well as

technical and project management support tools.

In addition, some

government employees took role-specific training. For example, the

program’s test automation subject matter expert completed training in

continuous integration and test automation.

To further ensure contractors on ICE SEVIS Agile teams had the

necessary Agile-centric skills, the ICE SEVIS program defined the Agile

U.S Immigration and Customs Enforcement, ICE Team Process Agreement for SEVIS

Information Sharing.

U.S. Immigration and Customs Enforcement, SEVP Agile Overview for Development

Teams (Sept. 2018).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 73 GAO-20-213 Agile Software Development

methodology and necessary technical expertise for contractors in the

contract requirements. For example, the performance work statement for

one development module required contractors to use the program’s

management software tool to track user stories. The performance work

statement also required use of the program’s continuous integration and

automated testing tools. The terms and conditions for this contract also

identified the required experience for key personnel, such as proven

experience working in an Agile environment.

The ICE SEVIS program also evaluated contractor qualifications to

ensure they had the necessary technical expertise. According to the

program manager, contractor qualifications were evaluated in two stages;

first, by assessing the contractor’s proposal, and second, by conducting a

technical challenge to ensure that contractors could demonstrate the

technical skills in the proposal. According to the instructions included in

the request for contractor proposals, this technical challenge required the

contractor to leverage Agile best practices to design, develop, and

demonstrate working software that addressed user stories provided by

the program. Although the instructions stated that contractors were

required to follow Agile methods, the ICE SEVIS program manager stated

that the primary goal of the technical challenge was to assess

development skills rather than knowledge of Agile.

Agency policy or guidance should call for technical and project tools to be

available to support Agile development and for system design that will

support iterative delivery.

Make technical and project support tools available

Project management and technical support tools should be integrated into

a program’s technical environment, where appropriate. The tools within

this technical environment should be readily available to Agile teams.

DHS policy and guidance called for Agile projects’ technical environments

to support Agile methods. The department published guidance for

standing up technical environments specifically for Agile projects. For

example, the DHS Agile instruction manual identified the benefits of using

program support tools for tracking program progress, reporting on that

progress as part of program governance, and automating tests within an

Agile technical environment. The manual stated that a program or project

manager is responsible for fostering an environment that enables the

Agile team to succeed, including obtaining the appropriate tools.

Technical environments enable

Agile development

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 74 GAO-20-213 Agile Software Development

To supplement this guidance, DHS offered a suite of tools that Agile

programs could access. The suite of tools was referenced in a checklist of

activities for program or project managers in the Agile instruction manual.

According to an IT specialist from the Technical Architecture and

Engineering division within the Office of the Chief Technology Officer

(OCTO), the tools available included program management tools as well

as technical tools. The specialist stated that OCTO provided programs

with access to this suite of tools to build support for and familiarity with

the tools, evaluating any requested plug-ins from programs and doing

their best to accommodate them.

Case Study Example

The ICE SEVIS program defined the technical environment to include

technical tools for automated testing and continuous integration. The

team process agreement for one of the program’s development modules

identified technical tools that supported continuous integration and testing

within the program’s technical environment.

This included Jenkins for

continuous integration as well as MUnit and Soap UI for continuous

testing. In addition, the ICE SEVIS Modernization Test and Evaluation

Master Plan discussed that tools for helping to ensure code quality, such

as an automated code analytics tool, should be used to identify test

coverage of code and cybersecurity code vulnerabilities.

The program also defined management support tools in the process

agreement. Specifically, it identified support tools for tracking and

knowledge management, such as JIRA and Confluence. The team

process agreement stated that JIRA should be the main knowledge

management tool and that all changes, discussion, and history should be

tracked in each ticket. This process agreement also stated that JIRA

should be the team’s tracking tool with Confluence used to provide

transparency.

U.S Immigration and Customs Enforcement, ICE Team Process Agreement for SEVIS

Information Sharing.

U.S Immigration and Customs Enforcement, Test and Evaluation Master Plan for the

Student and Exchange Visitor Information System Modernization, Version 1.2 (Feb. 19,

2018).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 75 GAO-20-213 Agile Software Development

Design a system that supports iterative delivery

The agency should adopt policy or guidance that allows project designs to

develop modular system components and the program should establish a

loosely coupled architecture that allows for modular development.

DHS guidance allowed project designs to develop modular system

components through upfront architecture planning. The DHS Technical

Review Guide advises stakeholders to discuss and approve the technical

design of the system, including its top-level architecture, as part of the

system definition review. This review should take place prior to

development work.

For Agile programs, DHS suggested that programs may elect to switch

the system definition review with a release planning review. The SELC

Tailoring Examples for Selected Types of DHS Acquisition Programs

specified that this design discussion should take place as a part of

release planning. The department referred to this design as an

“architectural runway”, a description that should enable the team to

conceptualize how the user stories will be implemented. In exiting the

release planning review, the Technical Review Guide noted that

programs should answer whether or not an architecture exists, if the

architecture enables the deployment of the release, if architecture

collaboration is explained and understood for this development process,

and if the appropriate resources are available.

In addition to transitioning to a release planning review, DHS guidance

urged Agile programs to move away from traditional artifacts associated

with a system definition review. In this shift from traditional artifacts, the

department proposed that programs document software design within a

system design document on a release-by-release basis. According to the

Requirements Engineering Users Guide, in Agile methodologies detailed

design occurs at the iteration level and, as such, the design is

documented in an iterative fashion in the system design document. The

guide further stated that the system design document allows the

development team to communicate the design to others including

customers, managers, and other developers and that industry best

practice was to represent the design through a series of “design views.”

Each software design stakeholder could have a distinct perspective on

what are the essential aspects of a software design. Together, these

views provide a comprehensive description of the design in a concise and

usable form that simplifies information access and assimilation.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 76 GAO-20-213 Agile Software Development

DHS guidance did not discuss the system design document as a

delivered artifact until after the sprint review and demo and a release

readiness review had been discussed.

At the end of each iteration, DHS

guidance stated that the system design document should represent the

design of the feature, function, and/or system as it existed at that

moment. To facilitate communication between Agile teams and to ensure

the most up-to-date description of the design is available, guidance called

for the system design document to be developed and maintained in an

electronic form using any number of programs or web tools that are

available. The Requirements Engineering Users Guide noted that the

system design document is to be considered complete when each

identified design concern is the topic of at least one design view, all

design constraints have been applied, and sufficient detail exists to be an

authoritative and primary “code-to” artifact.

The system design document should also provide traceability to the

feature, epic, and operational requirements document “shall” statements.

The SELC Tailoring Examples for Selected Types of DHS Acquisition

Programs stated that, prior to releasing software to the production

environment, a release readiness review should be conducted. As part of

this guidance, the department stated that the intent of this release

readiness review included ensuring that all elements of the release were

complete, including a system design document.

DHS guidance also discussed designing a loosely coupled architecture,

another important aspect of project design that facilitates modular

development. A member of the contractor support staff for the DHS OCIO

stated that the Enterprise Architecture Team was expected to consider

modularity and loose coupling generally through consideration of

technical complexity. According to DHS Enterprise Architecture principles,

technical complexity is to be mitigated in part by the implementation of

loose coupling.

According to the principles, DHS will incorporate loose

coupling into architecture and systems design to minimize the risk

resulting from changes within one system necessitating changes within

an interoperable system.

For the purpose of this report, we use the terms “sprint” and “iteration” interchangeably.

Figure 3 in the background does not cover a release readiness review. According to

officials from DHS OCIO, this technical review will be renamed the “release cycle review”

but serve the same purpose as the release readiness review.

Department of Homeland Security, Enterprise Architecture Principles (Oct. 2015).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 77 GAO-20-213 Agile Software Development

Case Study Example

The BEE Air Exit project design document defined the planned design for

the system and addressed design and architectural concerns that could

affect the system’s operating environment.

As part of this design consideration, the project established a loosely

coupled architecture. This loosely coupled architecture was illustrated

within the project’s system design document. This system design

document defined the Traveler Verification Services software as

consisting of two distinct components: 1) traveler verification services

core and 2) traveler verification services matcher. The functionality and

responsibility of these two components were distinguished throughout the

document. Moreover, the document detailed how the Traveler Verification

Services software would be delivered as a system of applications,

combining an integration layer, business layer, data access layer, and

data layer.

Agency policy or guidance should call for teams to maintain a sustainable

development pace and track and monitor that pace and for non-functional

requirements and critical features to be defined and incorporated in

development.

Maintain a sustainable development pace and track and monitor that

development pace

The agency should have policy or guidance that calls for Agile projects to

establish a sustainable development pace. This guidance should be

supplemented by tracking and monitoring the pace. The program should

establish a sustainable pace for Agile projects and that pace should be

tracked and monitored.

DHS guidance called for Agile projects to manage the pace of the

software development. The Agile instruction manual stated that Agile

projects should consider velocity and burndown rates to track the overall

project status and update the project plan to reflect this status. In a

separate appendix, the Agile instruction manual also identified metrics for

project and program managers and executives to consider in order to

monitor how a project was progressing, how Agile was optimizing the use

of team members and resources, and where the project stood in terms of

key Agile measures. In the list of Agile metrics, DHS highlighted

burndown rate and velocity, and offered a description and method of

calculation for each.

Project planning controls are

compatible with Agile

development

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 78 GAO-20-213 Agile Software Development

In addition to the Agile instruction manual, the department provided

training that spoke to development team pace. For example, the

curriculum for lesson six of course APM 350 on managing program

execution included a section covering Agile development metrics. Among

the metrics discussed were those associated with progress, including

velocity and burndown charts. Progress metrics were also covered in

other course offerings. However, DHS guidance and training materials did

not cover the concept of ensuring a sustainable pace.

In order to track and monitor the development team’s pace, the

department incorporated several related measures into the Agile core

metrics. Among others, programs executing Agile were expected to report

on the following pace-related metrics after each iteration:

• story points completed,

• story points planned to be completed,

• number of production deployment per quarter, and

• average product deployment lead time.

These measures could provide programs and the department with an

understanding of the development team’s pace and the extent to which it

was or was not sustainable.

However, the department was not tracking and monitoring development

team pace as intended. The Agile instruction required Agile programs to

submit Agile core metrics within six months of the instruction’s

publication. However, according to the Director of STM, programs were

not consistently reporting these core metrics. According to the Director of

STM, the department was still working with programs to ensure they

consistently reported the core metrics to the Investment Evaluation,

Submission, & Tracking system.

Case Study Example

The SeaWatch project at the United States Coast Guard (USCG)

demonstrated that it was monitoring development pace on a monthly

basis. SeaWatch officials stated that they used TAIGA as a tool to

manage the overall project and to auto-calculate pace. Additionally,

SeaWatch officials stated that contractors delivered a monthly progress

report, which contained the accomplishments of each team and a

snapshot from the latest TAIGA report. For example, one monthly report

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 79 GAO-20-213 Agile Software Development

for SeaWatch included a burndown chart for the SeaWatch project’s

development backlog and the monthly output of user stories and

associated story points by development effort that could be used to

assess development pace over time.

SeaWatch officials stated that the teams used velocity to help plan for the

next iteration. Officials added that they tracked the collective velocity of all

four teams as they were all working together on the same ship build. In

the future, officials stated that this tracking of velocity could also be used

to track individual team velocities as necessary.

The project demonstrated that it was adapting in order to achieve a

sustainable pace. According to the April 2018 monthly report, the team

completed 55 user stories worth 500 story points. The following month, in

the May 2018 monthly report, the number of user stories dropped from 55

user stories to 17, worth 130 story points. According to the June 2018

monthly report, the team completed a development effort of 31 user

stories and 278 story points.

According to the SeaWatch acquisition manager, development pace

fluctuated because not all sprints were of equal difficulty. The acquisition

manager added that the number of completed story points per sprint

could also be inconsistent due to inaccurate user story estimates,

changes in staff availability from sprint to sprint, and other external factors

such as weather.

Define and incorporate non-functional requirements in development

The agency should have policy or guidance in place for incorporating

non-functional requirements for Agile projects and the program should

account for non-functional requirements, such as security and privacy, in

the program strategy and throughout development.

DHS guidance addressed the incorporation of non-functional

requirements for Agile projects. According to the Technical Review Guide,

non-functional requirements could be governed via a system definition

review. According to the guide, this review was required at the end of the

The Department of Homeland Security’s Requirements Engineering Users Guide

defines non-functional requirements as requirements that specify criteria that can be used

to judge the operation of a system, rather than specific behaviors. This should be

contrasted with functional requirements that specify specific behavior or functions. Typical

non-functional requirements are reliability, scalability, availability, and cost.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 80 GAO-20-213 Agile Software Development

requirements definition phase to focus on the completeness of the

requirements engineering activities, including the gathering, analysis, and

documentation of functional and non-functional requirements. This review

assessed the traceability of these requirements to the operational

requirements document and concept of operations.

In the case of Agile programs, DHS suggested replacing the system

definition review with a release planning review. In place of traditional

artifacts associated with a system definition review, DHS guidance stated

that the capabilities and constraints document, backlogs, and the system

design document, which are developed iteratively throughout the release,

should document the requirements and provide traceability to the

operational requirements document. These artifacts served the function

and filled in for the functional requirements document and the system

requirements document previously required for a system definition review.

The Technical Review Guide noted that, as the capabilities and

constraints document matures, business and architectural epics should

decompose to features or themes, and, ultimately, user stories that reflect

the specific tasks that users will perform. The Technical Review Guide

cited as exit criteria that a program or project should answer whether the

capabilities and constraints document identified the specific features and

non-functional requirements to be addressed in the release.

DHS requirements engineering guidance expanded on how Agile

programs and projects could manage non-functional requirements. The

guidance explained that there were various ways that the constraints or

non-functional requirements such as security, Section 508 accessibility,

privacy, or reliability could be translated down to the iteration level. It

stated that some Agile teams may include these non-functional

requirements in the backlog, while other teams may include them as part

of acceptance criteria or in an artifact called the “definition of done”.

According to officials from the Science and Technology Directorate Office

of Systems Engineering, once defined, the day-to-day operations and

testing for non-functional requirements were the responsibility of the

operational test agent.

The “definition of done” should identify all of the activities and artifacts, besides working

code, that must be completed for a feature or sub-epic to be ready for deployment or

release, including testing, documentation, training material development, and

certifications.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 81 GAO-20-213 Agile Software Development

DHS maintained some governance over non-functional requirements.

According to the DHS acquisition management instruction, the

operational requirements document should be approved by the

Acquisition Decision Authority after validation by the Joint Requirements

Council. The operational requirements document should include both the

functional and non-functional requirements. Officials from the Office of the

Director of Test and Evaluation said that they do not usually provide

feedback on the decomposed functional or technical requirements for

software development projects, focusing only on the operating

requirements, because that is what directly impacts operations.

Case Study Example

The CBP BEE program’s functional requirements document outlined a

series of non-functional requirements as the requirements used to define

how the system is to behave as opposed to functional requirements that

define what the system should do. The project included 10 non-functional

requirements in the functional requirements document. For example, the

biometric match service should have an overall availability of greater than

or equal to 99%, which included both scheduled and unscheduled

downtime. These ten non-functional requirements comprised five related

to availability, three related to reliability, one related to scalability, and one

related to security. All of these non-functional requirements were

scheduled for release as part of the initial operating capability.

CBP officials noted that non-functional requirements were also captured

within the operational requirements document as measures of

effectiveness. According to project officials, measures of effectiveness

and other security-related parameters translated into the key performance

parameters for the project. Officials noted that these key performance

parameters were tracked on a daily basis and that information was fed

into a monthly report. The operational requirements document stated that

the program’s suitability requirements conformed to the DHS and CBP

enterprise architectures and all DHS and CBP infrastructure policies and

guidelines. Moreover, it noted that National Institute for Standards and

Technology guidance and DHS guidance factored into the development

of security related non-functional requirements. For example, system

security controls should be compliant with National Institute of Standards

and Technology and DHS sensitive system guidelines based on its

Federal Information Processing Standard 199 rating for availability,

integrity, and confidentiality.

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 82 GAO-20-213 Agile Software Development

Define and incorporate critical features in development

The agency should have policy or guidance in place for incorporating

critical features for Agile projects. The program should ensure that its

strategy considers all mission, architectural, and critical safety

components, along with their dependencies, on a regular basis.

DHS policy and guidance addressed the incorporation of critical features

for Agile projects. As discussed in the non-functional requirements

section, programs were expected to document functional requirements

via the systems design review or, as recommended for Agile programs, a

release planning review. Artifacts associated with these reviews served to

capture the functional requirements for the program and should be

evaluated as part of the entrance and exit criteria defined in the technical

review guide. Additional guidance elaborated on the process for

decomposing requirements.

Unlike non-functional requirements, applicable exit criteria on critical

features expanded into the solution engineering review. This criteria

included questions devoted to critical features and how they tied back to

performance measures (e.g. key performance parameters). According to

the Director of STM, headquarters oversight of critical features was

limited to the higher-level requirements defined in the operational

requirements and concept of operations documents.

Case Study Example

The ICE SEVIS program captured critical features in documents required

by department acquisition management policy and guidance. The ICE

SEVIS Modernization Concept of Operations listed specific functional

capabilities associated with mission and mission support scenarios.

The ICE SEVIS Modernization Operational Requirements document

expanded on these functional capabilities and identified the operational

and program-level requirements. These requirements were necessary to

achieve the performance goals and mission of the Student and Exchange

Visitor Program and the Department of State, the primary sponsors for

Immigration and Customs Enforcement, The Student and Exchange Visitor Program:

Concept of Operations for the Student and Exchange Visitor Information System (Aug. 16,

2016).

Appendix IV: Leading Practices for Adopting

Agile Development—Program Processes

Page 83 GAO-20-213 Agile Software Development

the program.

In particular, the SEVIS Modernization Operational

Requirements document identified business capabilities and key

performance parameters that measured system capabilities.

The core capabilities are long-term initiatives intended to span multiple

contracts and deliver the major components necessary for SEVIS

modernization. The SEVIS Modernization Operational Requirements

document stated that these capabilities must be present for the SEVIS

modernization to be considered a success. These business capabilities

represented the core SEVIS functions needed to close outstanding

SEVIS vulnerabilities. According to the ICE SEVIS Modernization SELC

Tailoring Plan, there were 79 sub-capabilities supporting the eight core

capabilities. The sub-capabilities generally fulfilled one or more

stakeholder needs and were delivered within a release or series of

releases. The SEVIS Modernization Operational Requirements document

confirmed that the program should prioritize and sequence the

capabilities for delivery during the release planning and delivery

processes.

The program provided a road map for one development module. This

road map listed areas for development in the order they were intended to

be developed and identified the associated business capabilities. The

business capabilities identified in the road map aligned with the sub-

capabilities listed in the SEVIS Modernization Operational Requirements

document. Examples of business capabilities in the road map that were

also sub-capabilities identified in the operational requirements document

included:

• create nonimmigrant record (including supporting forms),

• align nonimmigrant eligibility information with unique nonimmigrant,

• update nonimmigrant biographical information, and

• add/update dependent information.

Immigration and Customs Enforcement, The Student and Exchange Visitor Program:

Operational Requirements Document for the Student and Exchange Visitor Information

System Modernization (Oct. 3, 2017).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 84 GAO-20-213 Agile Software Development

This appendix describes in more detail our evaluation of the three leading

practices for team activities and dynamics when adopting Agile

development. It does not present new findings; rather, the information is

intended to assist the Department of Homeland Security (DHS) in

implementing the recommendations described in this report.

For teams to successfully transition from processes using traditional

software development methods to Agile methods, leading practices for

team activities and dynamics recommend that

• the composition of the team supports Agile methods by

• self-organizing Agile teams

• defining the role of a product owner

• work is prioritized to maximize value for the customer through

• creating user stories to define work

• prioritizing requirements in a backlog based on value

• estimating the relative complexity of user stories

• repeatable processes are in place by

• meeting daily to review progress and discuss impediments

• observing end-iteration demonstrations

• observing end-iteration retrospectives

• employing continuous integration

• ensuring the quality of code being developed

Within DHS, program management offices are responsible for planning

and executing individual programs and implementing applicable Agile

methodologies. According to Office of the Chief Technology Officer

(OCTO) officials, DHS contracts for Agile services, including

development, rather than performing development in-house. As a result,

Agile teams may be predominantly contractors rather than federal

employees. In addition, DHS Office of the Chief Information Officer

(OCIO) is responsible for setting the policies and procedures to ensure

that programs and, in turn, the teams that make up those programs,

leverage Agile development best practices to meet the department’s

goals and are within acquisition policy. DHS OCIO is also responsible for

providing guidance for and reviewing the adoption and execution of Agile

development.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 85 GAO-20-213 Agile Software Development

Agency policy or guidance should require individual, self-organizing Agile

teams for each segment or iteration and define the role and

responsibilities of the product owner.

Self-organize Agile teams

Agile teams should be self-organizing, meaning they are empowered to

collectively control how to accomplish their work and the resulting

product.

An Agile team’s authority should include lower-level decision

making and team formation and highlight the importance of team stability.

The team’s composition should be cross-functional and consist of

members who possess all the skills needed to produce working software,

including, but not limited to, contract specialists, developers, and testers.

DHS provided guidance to Agile teams on self-governance. The Agile

Development and Delivery for Information Technology instruction (Agile

instruction) and the Agile Development and Delivery for Information

Technology Instruction Manual (Agile instruction manual) both explain

that collaborative, self-organizing, and cross-functional teams help

achieve the flexibility needed for the iterative development that

characterizes Agile development methods.

The Agile instruction manual

notes that most Agile methodologies assume the dedicated involvement

of all stakeholder, developer, and integration staff throughout the project.

DHS guidance also discusses team formation. The Agile instruction

manual recommends that the project team include the roles of the

program or project manager, a product owner, a development team of

approximately five to nine members, testers, and an Agile coach, and any

additional expertise as needed. According to DHS guidance, a program or

project manager is responsible for establishing the project team. The

program or project manager is supported in this by the component

acquisition executive and other component management.

Case Study Example

At DHS, U.S. Immigration and Customs Enforcement’s (ICE) Student and

Exchange Visitor Information System (SEVIS) program had self-

The product is the software functionality assigned to the team for development.

Department of Homeland Security, Instruction 102-01-004, Agile Development and

Delivery for Information Technology, Revision 01 (April 16, 2018) and Instruction Manual

102-01-004-01, Agile Development and Delivery for Information Technology Instruction

Manual, Revision 00 (Jul. 15, 2016).

Team composition supports

Agile methods

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 86 GAO-20-213 Agile Software Development

organizing teams that defined their own processes for completing work.

ICE Agile teams, including those supporting the SEVIS program, were

expected to document their processes in a team process agreement,

where a team had the authority to define its own operational strategy and

make decisions about the product, including when to consider the product

completed according to the program’s “definition of done.” According to

the ICE Agile principles instruction, a program chooses a baseline set of

practices that are documented in a team process agreement and are

adjusted over time.

ICE SEVIS teams were self-managing and included the roles necessary

to deliver what they committed to in a sprint.

ICE’s Agile playbook

suggested minimum levels of experience, knowledge, and certifications

necessary for key personnel to support Agile methodologies.

For

instance, the playbook suggests that Scrum masters be certified and

have a minimum of one year of experience. To help ensure that

contractors have the requisite skills necessary, ICE SEVIS officials stated

that vendors are required to demonstrate their ability to develop a small

software application before a contract is awarded to them.

Define the role of a product owner

A product owner should understand the business and strategic values of

the agency and its alignment with the vision of the product team and

support Agile methods. A product owner’s responsibilities include

availability to the team, authority for making programmatic decisions,

general responsibilities as a member of the team, and the need to

possess subject matter expertise related to the business needs. A

product owner is an authoritative user who manages the requirements

prioritization, communicates operational concepts, and provides continual

feedback to the team.

DHS provided guidance on the role and responsibilities of a product

owner. According to the Agile instruction manual, the product owner is

responsible for representing stakeholders. To do so, the product owner

should be available to the development team throughout the iteration to

Immigration and Customs Enforcement, Management Instruction ICE-OCIO-001

Applying Lean-Agile-DevOpsSec Principles at ICE, Version 1.1 (Jan. 25, 2018).

For the purpose of this report, we use the terms “sprint” and “iteration” interchangeably.

Immigration and Customs Enforcement, ICE Agile/DevOps Playbook, Version 1.0 (May

18, 2017).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 87 GAO-20-213 Agile Software Development

answer questions and clarify requirements on behalf of the stakeholders.

The manual stated that the product owner is also responsible for ensuring

that the product meets user needs and delivers value. This includes, for

example, prioritizing user stories in the backlog and serving as an

acceptance authority for work completed by the team.

The department also provided elective training on the role of a product

owner. For example, the U.S. Citizenship and Immigration Services Office

of Information Technology offered an elective product owner training

course. The USCIS product owner training covered concepts such as the

importance of the product owner’s availability to the team and the product

owner’s authority for making programmatic decisions.

Case Study Example

ICE identified a product owner for SEVIS to represent two user

communities. The program identified one product owner from ICE’s

Student and Exchange Visitor Program and a second product owner from

a stakeholder organization within the Department of State. Both product

owners were identified in the ICE SEVIS staffing plan.

According to a team process agreement for one development module, a

product owner is responsible for, among other things:

• Prioritizing and deciding which user stories will be implemented in

each iteration.

• Making an acceptance decision for each user story based on the

story’s acceptance criteria.

• Ensuring that the intended value of the functionality is delivered.

According to program officials, product owners for the ICE SEVIS

program prioritized user stories during planning sessions. The Student

and Exchange Visitor Program Agile Overview slides stated that the

team, including the product owner, attends sprint planning to review the

prioritized product backlog and to ensure a common understanding of the

product owner’s immediate priorities.

Product owners also exercised authority to validate acceptance criteria

and subsequently close user stories. The program’s “definition of done”

stated that the product owner must test and indicate acceptance of each

user story in order for a user story to be considered complete. In a written

response, ICE SEVIS officials stated that ICE SEVIS product owners

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 88 GAO-20-213 Agile Software Development

indicated a user story had met the acceptance criteria and could be

closed by changing the user story’s status to “closed” using the team’s

program management software tool.

In addition, product owners were available to the development team to

ensure timely input. According to a team process agreement for a

development module, the product owner should work closely with the

development team to communicate the details of requirements and

answer questions about user stories. In an interview, the ICE SEVIS

product owner representing the Student and Exchange Visitor Program

stated that the role was a full-time position and did not have any

competing responsibilities. To ensure availability, the ICE SEVIS product

owner representing the Student and Exchange Visitor Program stated

that there was a designated backup who had the same authority and

responsibilities as the full-time product owner.

Agency policy or guidance should call for Agile teams to create user

stories to define the work; prioritize requirements in a backlog based on

value, including tracking and monitoring the value of work accomplished;

and estimate the relative complexity of user stories. Individual Agile

teams within the respective programs and projects should implement

these aspects of Agile development.

Create user stories to define work

A user story is to reflect a small segment of work that can be completed in

a single iteration. The agency should have policy or guidance in place for

writing user stories for Agile projects. The product owner should

determine the value of a user story in consultation with the development

team, including the acceptance criteria and defining what “done” means.

User story value should then be re-evaluated based on requirements to

ensure the greatest return on investment.

DHS provided guidance that Agile programs and projects could leverage

when writing a user story. The Agile instruction manual, Homeland

Security Acquisition Institute Agile lessons, such as “Managing Program

Execution” and the Requirements Engineering User’s Guide provided a

basic format for how to craft a user story.

These resources noted that a

Department of Homeland Security, Requirements and Engineering User’s Guide, Version

2.0 (Feb. 28, 2014).

Work is prioritized to maximize

value for the customer

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 89 GAO-20-213 Agile Software Development

user story defines where a “role” wants some “goal/desire” accomplished

to result in a “benefit”.

The Requirements Engineering User’s Guide also discusses the role of

acceptance criteria and a definition of done in user story development.

The guide highlighted that acceptance criteria defines the boundaries of a

user story and confirms when a story has been completed and is working

as intended. It specifies that acceptance criteria should be included in an

Agile program or project’s capabilities and constraints document, a DHS

artifact unique to Agile development and highlighted in the systems

engineering life cycle (SELC) tailoring example for Agile. This guide

added that the definition of done identifies all of the activities/artifacts

besides working code that must be completed for a feature or sub-epic to

be ready for deployment or release including testing, documentation,

training material development, certifications, etc.

The Agile instruction manual places much of the responsibility for defining

a user story under the purview of the product owner. The Agile instruction

manual stated that the product owner is the individual tasked with

providing requirements to the development team and is responsible for

determining the features necessary for the product release. The manual

also emphasized that the product owner is only responsible for clarifying

the user story requirements that would meet his or her needs and not

responsible for clarifying how user stories should be implemented to meet

those needs.

Case Study Example

The ICE SEVIS program developed user stories based on business

capabilities and other requirements as determined by the product owner

and the business stakeholders. The SEVIS Modernization Operational

Requirements Document describes eight business capabilities that

represent core SEVIS functions. According to ICE SEVIS officials, these

business capabilities are addressed through user stories, so there is

traceability in the backlog from user stories to epics to business

capabilities/operating requirements. The team’s process agreement for

one development module—Information Sharing—assigned responsibility

DHS defines an epic as a very large user story that is eventually broken down into

smaller stories. Within DHS, business epics describe top-level business processes and

architectural epics describe the architecture the system is incorporated into.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 90 GAO-20-213 Agile Software Development

for writing user stories to the product owner. This agreement also noted

that acceptance criteria would be required for most stories.

User stories for the program were managed through a program

management software tool. An output of the backlog from the program

management software tool for one development module—Managing

Nonimmigrant Information—contained 525 user stories. These user

stories generally followed DHS and ICE guidance for capturing what a

user needs and why. Most of these user stories also included acceptance

criteria.

The program also developed a “definition of done” for all user stories in

the team process agreement.

According to the definition, a user story

was “done” when the following steps were addressed:

• All code to meet the story’s needs was written according to the

system’s development standards.

• Unit tests were written and run successfully.

• All code was checked in and the build completed successfully.

• All database changes (if required) were complete and checked in (a

functional test could be run).

• The software had been deployed to the system test environment and

passed system tests.

• The product owner agreed that the implementation met the

acceptance criteria written in the story as appropriate.

All documentation required to support the story was completed (test

cases, interface updates, etc.)

Prioritize requirements in a backlog based on value

Agile teams should pull work from a prioritized backlog and provide

frequent deliveries of software with immediate value to the user. The

team should determine the value of the user stories, prioritize work in a

product backlog, and provide an ongoing assessment of value expected

As with the capabilities and constraints document, the team process agreement is an

artifact unique to Agile development and is highlighted in the Agile portion of the SELC

Tailoring Examples.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 91 GAO-20-213 Agile Software Development

versus value delivered. The value of the work accomplished by Agile

projects should be tracked and monitored.

DHS guidance called for prioritizing user stories in a backlog. The

department published an example of a SELC tailoring plan for Agile

development that encouraged programs and projects to prioritize user

stories in a backlog as part of each release.

To ensure that programs or

projects took these steps, the Technical Review Guide exit criteria for the

release planning review asks if programs or projects will have a process

in place for prioritizing user stories prior to the development of features for

each release.

Planning sessions were one such process that programs and projects

could use to prioritize user stories in the backlog.

The DHS Agile

instruction manual stated that, during sprint planning, the product owner

meets with the development team in order to identify user stories from the

backlog that should be prioritized for the upcoming sprint and that

prioritization decisions should be made based on value to the users. In

addition, the product owner should ensure that prioritization decisions

maximize mission values. The Requirements Engineering User’s Guide

also states that requirements should be prioritized based on continuous

stakeholder input so that programs can prioritize what users need the

most.

DHS guidance also discussed how to determine the value of individual

user stories. While the Director of STM said that the product owner is

responsible for interpreting the concept of value as it applies to a user

story and the relative prioritization of the backlog, Agile Requirements and

Road Mapping Guidance for DHS includes a discussion on how a

program can sequence its road map for learning, risk, and economic

value. In this section, DHS offers models to consider to assist in user

Department of Homeland Security, SELC Tailoring Examples for Selected Types of DHS

Acquisition Programs, Version 2.0 (Nov. 2016).

Department of Homeland Security, Technical Review Guide, Version 2.0 (Dec. 2015).

Sprint planning meetings occur at the start of a sprint and consist of two parts. First, the

team reviews the product backlog and the product owner describes priorities for the

upcoming sprint. Second, the team decides how the work will be completed and estimates

the size of user stories.

Department of Homeland Security, Requirements Engineering User’s Guide, Version 2.0

(Feb. 28, 2014).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 92 GAO-20-213 Agile Software Development

story prioritization decisions and considerations for the product owner,

such as seeking to balance between business value and cost.

The

Director added that there were venues, such as Agile “chat and chews,”

where program staff could ask questions and receive informal guidance.

DHS modified acquisition procedures to allow for an ongoing assessment

of progress, and indirectly the value of work accomplished, via the

release road map. DHS guidance stated that the release road map is

submitted to the Acquisition Review Board prior to acquisition decision

event 2B, as required by the Agile instruction. The Technical Review

Guide exit criteria for the release planning review and the release

readiness review asked if the development team was following the

release road map and making adjustments that supported the successful

completion requirements defined at acquisition decision event 2B.

Thereafter, programs submitted a road map to the Acquisition Review

Board during regular program reviews.

In addition to tracking and monitoring the value of work accomplished

against a release road map, regular Acquisition Review Board program

reviews allowed for the assessment of value expected versus value

delivered. The presentation template for Acquisition Review Board

program reviews included a slide for programs to report their progress

toward planned features. For each review, programs identified a

percentage of each capability that they planned to complete by the next

review. In addition, programs reported on the percentage of each

capability that they had completed since the last review.

Case Study Example

The U.S. Coast Guard (USCG) SeaWatch Agile teams prioritized

requirements in a backlog based on the team’s ability to complete them

within a sprint. According to the acquisition manager for the Command,

Control, Communications, and Computers, Intelligence, Surveillance and

Reconnaissance (C4ISR) program, the SeaWatch product owner for new

development determined priorities for new requirements with

stakeholders. The product owner then defined those requirements as an

epic or as a user story. The C4ISR acquisition manager stated that the

user stories were prioritized in the backlog during sprint planning primarily

based on whether the Agile team could complete the work in the

upcoming sprint rather than on the value assigned by the stakeholders.

Department of Homeland Security, Agile Requirements and Road Mapping Guidance for

DHS (Mar.12-14, 2018).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 93 GAO-20-213 Agile Software Development

According to SeaWatch officials, user stories that could not be completed

during the current sprint were marked as a priority item for the next sprint.

Although the SeaWatch program assessed value to the user for some

epics, this did affect how the epic or its associated user stories were

prioritized in the backlog. The C4ISR acquisition manager stated that

SeaWatch assigned a value (e.g. extra large, large, or medium) to an epic

based on the epic’s value to the user. However, the acquisition manager

noted that user stories were not typically prioritized by the value of the

associated epic. User stories were instead prioritized based on the Agile

team’s ability to complete the work within the current sprint.

The project reported on its accomplishments via a road map. In May

2018, SeaWatch reported on progress toward milestones in its road map

during an annual briefing for the Non-Major Acquisition Oversight Council.

The program reported that it had installed SeaWatch v3.0 on 65 out of 70

in-service cutters.

Estimate the relative complexity of user stories

The agency should have policy or guidance in place for relative

estimation practices for Agile projects. Teams should use relative

estimation for sizing the effort of work required to satisfy a user story by

estimating its complexity based on work of similar size and complexity.

Relative estimation enables teams to maintain a sustainable software

development pace and predict work commitments. The team should size

user stories relative to one another, assess the complexity of the work,

refine user stories and estimates over time, and use prior estimates to

inform future estimates. The product owner and team should continually

revisit the estimates as they learn more about the business priorities and

as user stories rise in the order of priority.

DHS did not provide policy or guidance for relative estimation. Although

the Agile instruction manual identified estimating user story size as an

integral part to sprint planning, it did not describe the specific techniques

or processes for estimating the relative complexity of user stories.

Instead, the Agile instruction manual discussed how programs could

successfully apply traditional earned value management and cost

estimating principles to Agile projects. DHS guidance noted that programs

had largely moved from measuring story points to feature points to help

programs quantify incremental progress

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 94 GAO-20-213 Agile Software Development

Case Study Example

The U.S. Customs and Border Protection’s (CBP) Biometric Entry Exit

(BEE) program defined practices and guidelines for how the program

expected to estimate user stories. For example, the Traveler Verification

Services process definition document identified a formula for calculating

story point values on the basis that one story point would equate to

approximately four working hours. Moreover, the process definition

document noted that story points must be reconciled to better reflect the

level of effort and task completion at the end of a given sprint.

However, it was not evident that the BEE program had implemented its

own guidance on the estimation of story points. Although the process

definition document outlined procedures for estimating user stories, only

two of 358 user stories in the Air Exit project backlog were estimated

using story points.

Agency policy or guidance should call for Agile teams to meet daily to

review progress and discuss impediments, and to observe end-iteration

demonstrations and end-iteration retrospectives. In addition, agency

policy or guidance should call for Agile projects to employ continuous

integration and confirm mechanisms are in place to ensure the quality of

code being developed. This includes setting expectations for automated

testing and code quality and tracking and monitoring against these

expectations. Responsibility for these aspects of Agile development

should lie with the individual Agile teams.

Meet daily to review progress and discuss impediments

The agency should have policy or guidance in place for holding the daily

stand-up and teams should hold daily meetings in order to stay on track

to meet the iteration goals for Agile projects and adjust as necessary.

DHS guidance defined the general procedure for holding a daily stand-up.

The Agile instruction manual stated that teams should conduct a daily

stand-up meeting for all team members. It can be conducted in person or

via another method of communication (particularly for remote employees)

for a brief, informal meeting every work day. According to the manual, all

team members should discuss the work each has accomplished since the

last daily stand-up, the work to be accomplished by the next daily stand-

up, and highlight any impediments that are preventing the team members

from completing their work. Additionally, the manual suggests that it is

Repeatable processes are in

place

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 95 GAO-20-213 Agile Software Development

necessary to conduct the daily stand-up with strict discipline, so that the

meetings stick to their allotted brief time and are consistently productive.

The Agile instruction manual also highlighted the importance of the daily

stand-up meeting to an Agile process. It called this meeting an essential

collaboration event during which all team members were expected to

participate and discuss their work. The manual suggested that holding

these meetings allowed the team to practice discipline that would assist

them in their work and foster mentoring and partnering relationships

within the team that were reinforced through the constant communication

of meeting every single day. The manual added that this activity allowed

the team to hold its members accountable and be made aware of issues

that may be mitigated through collaboration.

Case Study Example

The Traveler Verification Services team supporting the BEE program Air

Exit project at CBP held daily stand-up meetings. According to project

officials and supporting project artifacts, a daily stand-up meeting was

held each day at 10:00 a.m. Project officials noted that the daily stand-

ups included the entire 40-person team.

Observe end-iteration demonstrations

The agency should have policy or guidance in place for holding

demonstrations or other interactions for acceptance of user stories in

Agile projects. Teams should hold frequent demonstrations to showcase

features that have been implemented and obtain feedback for acceptance

of user stories in Agile projects.

DHS guidance defined the general procedure for holding an end-iteration

demonstration or review. In the SELC Tailoring Plan example for Agile

development, DHS recommended a sprint review and demo as one type

of technical review at the end of each iteration. The purpose of the review

was to demonstrate the working software to end users and other

stakeholders and to obtain feedback that could result in additional items

being added to the backlog. It stated that this review should also ensure

that the software design was documented for inclusion in the system

design document, a proposed DHS Agile-specific artifact. The tailoring

example noted that this review should formally end the iteration’s work

with no further development or testing occurring on any stories. The Agile

instruction manual added that this demonstration should confirm the value

of the incremental piece of software produced.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 96 GAO-20-213 Agile Software Development

DHS guidance also encouraged the use of demonstrations. The Agile

instruction manual states that a demonstration or review could be used to

reach a consensus on whether the work associated with a user story met

expectations or not. The manual also recommended that program and

project managers ensure that the functional software developed during

each iteration was demonstrated to the stakeholder at an iteration review

meeting.

Case Study Example

The ICE SEVIS program held end-iteration demonstrations. The ICE

SEVIS Modernization Systems Engineering Lifecycle Tailoring Plan

stated that sprint demonstrations were tailored into the program to

replace other review activities, such as the preliminary design, critical

design, and integration readiness review. The Test and Evaluation Master

Plan for SEVIS Modernization stated that standard sprint testing results

were to be reported at sprint reviews. According to program artifacts, the

sprint demonstration was to be conducted at the completion of each

sprint, every other Wednesday from 11:00 a.m. to 12:00 p.m.

Observe end-iteration retrospectives

The agency should have policy or guidance in place for holding a

retrospective to adapt and continuously improve on Agile projects. Teams

should hold a retrospective at the end of each iteration to identify areas

for improvement to adapt and continuously improve Agile practices.

DHS guidance defined the general procedure for holding a retrospective.

The program or project manager and team reviewed progress after each

iteration and release. This included the use of a retrospective to discuss

what went well, what didn’t go well, and to identify actions to correct

problems. Guidance noted that the team should immediately incorporate

feedback from the retrospective into future iterations.

The DHS Agile instruction manual highlighted the importance of the

retrospective. The manual stated that the end-iteration retrospective is a

key part of ensuring that teams following Agile methodologies are able to

identify problems and adapt to continuously improve for future sprints.

Additionally, the manual stated that end-iteration retrospectives are useful

in satisfying governance needs. For example, the Agile instruction manual

stated that programs could tailor standard-format SELC artifacts (as

codified in the SELC Tailoring Plan) to instead rely on assessment and

performance data addressed in end iteration retrospectives.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 97 GAO-20-213 Agile Software Development

Case Study Example

The Traveler Verification Services team supporting the BEE program’s Air

Exit project held end-iteration retrospectives. According to the process

definition for this team, a retrospective was to be held between the end-

iteration review and the subsequent planning session for the upcoming

sprint. The process definition defined the goal of the retrospective as

obtaining an honest review of the process with a consensus on how to

adapt it. In an interview, project officials noted that the team documented

the results of retrospectives on a release-by-release basis in a project

management software tool.

Employ continuous integration

The agency should have policy or guidance that defines and emphasizes

the use of automated testing and continuous integration. This guidance

should be supplemented by defining expectations for automated testing

and tracking and monitoring against these expectations. Agile teams

should adopt practices for continuous integration and automated testing

to ensure that software handoffs are repeatable and dependable.

Automated testing should be tracked and monitored based on established

expectations.

The DHS Agile instruction manual defined continuous integration as the

practice where delivery teams frequently integrate their code into a

shared master copy. It noted that these integrations are verified by an

automated build process, which performs testing to detect any integration

errors quickly and automatically. The manual stated that continuous

integration in Agile projects should be planned and recorded on a

release-by-release basis.

The Agile instruction manual also emphasized the importance of

continuous integration and automated testing. With regard to automated

testing, the manual set an expectation for program or project managers

and stakeholders to consider both automated testing tools and

infrastructure support for the Agile software build and test processes as

part of general project planning efforts. Moreover, the manual identified

continuous integration, automated acceptance testing, and automated

unit testing as key practices program or project managers can use for

continuously monitoring and reporting project health. These practices

could also help to identify opportunities for improving project team

performance.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 98 GAO-20-213 Agile Software Development

DHS officials acknowledged that current DHS programs implemented

testing and evaluation inconsistently and that the department’s existing

guidance and policies did not effectively support modern best practices in

automated testing and continuous integration. To address these gaps,

DHS had an Agile action plan that set an expectation for updating DHS

acquisition guidance, policy, and practices for testing and evaluation to

enable modern best practices in automated testing and continuous

integration.

In lieu of more explicit guidance, DHS incorporated training as part of a

curriculum geared toward test and evaluation managers that discussed

both continuous integration and automated testing. According to the

Deputy Director of Policy and Workforce Development in the Test and

Evaluation Division of the Science and Technology Directorate, an

alternative course containing content addressing Agile and continuous

integration and automated testing was recently merged with a required

test and evaluation course, creating a new course. According to the

Deputy Director, the new course was piloted during fiscal year 2019 and

will be standard in fiscal year 2020 as the required course for level II test

and evaluation certification.

In order to track and monitor automated testing, the department

incorporated several measures into the Agile core metrics. Programs

executing Agile were expected to report on the following testing-related

metrics after each iteration:

• Percentage of unit test coverage,

• Percentage of automated tests, and

• Percentage of regression testing coverage.

DHS had not established expectations for these Agile core metrics. The

Agile core metrics included a target. For example, the department

DHS intended for two Agile action items to expand on continuous integration guidance.

The first action item would develop a revised governance approach that enables programs

to establish continuous integration, continuous delivery pipelines. The second action item

would support programs in adopting modern best practices for automated testing and

continuous integration by developing a DHS-wide strategic view of integrated testing

practices and case examples of successful programs. This action plan would include

engaging components in the establishment of approved processes to provide programs

the ability to use a continuous or ongoing authority to operate for iterative development

and releases. As discussed earlier, some activities defined for the initial Agile action items

were still being managed and it was unclear when additional planned activities would be

completed.

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 99 GAO-20-213 Agile Software Development

suggested a program strive for seventy percent of tests to be automated.

However, the instructions accompanying the Agile core metrics stated

that all targets were notional and not expected to be reached. According

to the Director of STM, the initial core metrics were intended to assess

the level of DHS team achievement without imposing artificial industry-

based target measures for each. The Director stated that, on receiving

the metrics for a period of time, the department would then adjust the

core metrics and begin to include target measures based on the results

achieved. According to the Director, this effort was underway and an

updated set of core metrics would be distributed in early fiscal year 2020.

Moreover, the department was not tracking and monitoring automated

testing as intended.

Case Study Example

The CBP BEE program Air Exit project stood up a technical environment

that allowed for continuous integration. This technical environment was

outlined within the process definition of the Traveler Verification Services

team that was developing software. The Traveler Verification Services

process definition identified three operating environments: the

development, test, and production environments. All development

activities during the sprint were conducted within the development

environment. Similarly, all testing activities in preparation for the release

were conducted in the test environment. The final approved software

would then be deployed to the production environment.

CBP officials noted that the BEE program primarily used Jenkins to

integrate code for both continuous builds and deployment. The Air Exit

systems design document also mentioned the role of Jenkins in

continuous integration and continuous deployment for the project.

The Traveler Verification Services team incorporated JaCoCo and

FindBugs automated tests as part of the continuous delivery process and

they were run automatically when the code was checked in. Moreover,

the project’s system design document noted that the Traveler Verification

Services team integrated JaCoCo with the Eclipse Integrated

Development Environment as a code coverage inspection tool for unit

testing. Officials also noted that Selenium was used for automating the

testing within the technical environment.

Customs and Border Protection, Biometric Air Exit Systems Design Document, Version

3.2 (Dec. 11, 2018).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 100 GAO-20-213 Agile Software Development

Ensure the quality of the code being developed

The agency should have policy or guidance for an Agile project on

ensuring the quality of code being developed. This guidance should be

supplemented by defining expectations for code quality and tracking and

monitoring against these expectations. Agile teams should adopt

practices for code quality, such as having a test-driven development, pair

programming, and manual code reviews to supplement automated

testing. Agile teams should incorporate refactoring into code quality

practices and understand the importance of setting aside time for

refactoring.

DHS guidance recognizes the importance of ensuring code quality as part

of the development and testing process. The SELC Guidebook set an

expectation that code review and testing should be part of the software

development environment.

The guide recommended setting up servers

where developers could test code and check whether the developed

application runs successfully with that code. The guide suggested another

level of tests on application reliability to help ensure that the application

did not fail on the production server. The guide stated that the program

manager should ensure that the team takes corrective action for any

hardware and software deficiencies.

In order to find deficiencies early, DHS guidance identified coding and

testing practices that could help development teams. The Agile instruction

manual cited pair programming as one practice where two programmers

work simultaneously on a single task: one programmer observes and

reviews each line of code as it is written. DHS guidance also identified

test-driven development as a practice that could motivate developers to

write effective code. The Supplemental Guidance for Test and Evaluation

stated that this approach consists of writing test cases that define a

desired improvement, then writing the code to meet the desired

functionality, ensuring that the test passes, and refactoring the code as

necessary.

Refactoring, or re-coding, without changing the way the application

functions, is an Agile practice that DHS guidance recommends for

correcting deficiencies in the code. The Agile instruction manual stated

that refactoring aims to improve code readability and reduce the

complexity of previously delivered increments of software. It noted that

Department of Homeland Security, 102-01-103-01, Systems Engineering Life Cycle

Guidebook, Revision 00 (Apr. 18, 2016).

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 101 GAO-20-213 Agile Software Development

refactoring is important because development teams are focused on

adding the desired functionality with each release and may proceed with

making improvements to the code. Refactoring was cited as one way to

address this accumulation of needed improvements to the code, which

are known as technical debt.

The Agile instruction manual further emphasizes the importance of setting

aside time for refactoring to address risks associated with technical debt.

The manual states that refactoring a previously developed increment of

software to improve code quality may force a change in the release

schedule. However, if the team does not make these revisions in a timely

manner, the effort required to correct them later tends to increase. The

manual states that this increasing technical debt is a risk factor to be

addressed as soon as feasible. If the technical debt is allowed to

accumulate unchecked, or if the project team loses track of the scope of

its technical debt, the project could suffer from schedule and performance

problems.

In order to track and monitor the quality of code being developed, the

department incorporated several code quality and testing measures into

the Agile core metrics. Among others, programs executing Agile were

expected to report on the following quality-related metrics after each

iteration:

• Number of critical or major defects fixed.

• Number of critical or major defects in the backlog.

• Number of technical debt issues completed.

• Number of technical debt issues in the backlog.

However, the department was not tracking and monitoring code quality as

intended. These measures could provide programs and the department

with an understanding of the development team’s ability to address

defects and technical debt. In addition to these metrics, programs are

also expected to report quarterly on the number of outages requiring a

rollback or patch after production deployment.

Case Study Example

The ICE SEVIS program used manual testing to ensure code quality. The

definition of done for the program stated that new code should be peer

reviewed to identify risk to the existing code, assess compliance with

Appendix V: Leading Practices for Adopting

Agile Development—Team Activities and

Dynamics

Page 102 GAO-20-213 Agile Software Development

coding best practices, and evaluate refactoring. According to ICE SEVIS

officials, an independent specialist provides internal code reviews and

offers feedback on areas for improvement.

The ICE SEVIS program also employed automated testing to ensure

code quality. The definition of done required that unit tests cover a

minimum of 85 percent of code. Program officials stated that

vulnerabilities and bugs identified through this process were added to the

backlog and classified as technical debt.

The program refactored code to address technical debt, but did not set

aside time for refactoring each sprint. According to ICE SEVIS officials,

the development team refactored code as necessary to improve overall

quality but did not set aside time for refactoring unless they were

addressing a consistent issue. ICE SEVIS officials stated that the

development team could propose refactoring code during sprint planning

if there was a specific technical debt they had identified. However,

according to the Scrum master for the program, addressing technical debt

was additional work for the team to take on beyond the user stories they

planned to complete and this additional work incentivized the

development team to prevent the accumulation of technical debt.

Although DHS allowed Agile programs to tailor the core metrics, ICE

SEVIS submitted some of the code quality-related Agile metrics to the

department. The program included Agile metrics in June 2018

presentation slides for the Acquisition Review Board. For this initial

reporting period, the program reported no critical or major defects in the

backlog and no technical debt issues in the backlog. It also provided a

screenshot of the Agile core metrics reported to DHS via the Investment,

Evaluation, Submission, & Tracking system in February 2019. This

reporting period covered two iterations. The program reported that it fixed

four critical or major defects during the first iteration and did not have any

critical or major defects in the backlog for either iteration. The program

also reported that it completed eight technical debt issues in the first

iteration, out of 14 technical debt issues in the backlog. The program did

not report on the number of outages after deployment as part of the

Acquisition Review Board program review or as part of the metrics

submitted via the Investment Evaluation, Submission, and Tracking

system.

Appendix VI: Comments from the Department

of Homeland Security

Page 103 GAO-20-213 Agile Software Development

Appendix VI: Comments from the

Department of Homeland Security

Appendix VI: Comments from the Department

of Homeland Security

Page 104 GAO-20-213 Agile Software Development

Appendix VI: Comments from the Department

of Homeland Security

Page 105 GAO-20-213 Agile Software Development

Appendix VI: Comments from the Department

of Homeland Security

Page 106 GAO-20-213 Agile Software Development

Appendix VII: GAO Contact and Staff

Acknowledgments

Page 107 GAO-20-213 Agile Software Development

Carol C. Harris at (202) 512-4456 or [email protected]

In addition to the contact named above, the following staff made key

contributions to this report: Michael Holland (assistant director), Mathew

Bader (analyst in charge), Lamis Alabed, Jennifer Beddor, Christina

Bixby, Hannah Brookhart, Chris Businsky, Alan Daigle, Aryn Ehlow,

Nancy Glover, Gina Hoover, Anna Irvine, Hoyt Lacy, Jennifer Leotta,

Alexis Olson, Zsaroq Powe, Martin Skorczynski, Natalie Smith, and

Daniel Spence.

Appendix VII: GAO Contact and Staff

Acknowledgments

GAO Contact

Staff

Acknowledgments

(102504)

The Government Accountability Office, the audit, evaluation, and investigative

arm of Congress, exists to support Congress in meeting its constitutional

responsibilities and to help improve the performance and accountability of the

federal government for the American people. GAO examines the use of public

funds; evaluates federal programs and policies; and provides analyses,

recommendations, and other assistance to help Congress make informed

oversight, policy, and funding decisions. GAO’s commitment to good government

is reflected in its core values of accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no cost is

through our website. Each weekday afternoon, GAO posts on its website newly

released reports, testimony, and correspondence. You can also subscribe to

GAO’s email updates to receive notification of newly posted products.

The price of each GAO publication reflects GAO’s actual cost of production and

distribution and depends on the number of pages in the publication and whether

the publication is printed in color or black and white. Pricing and ordering

information is posted on GAO’s website, https://www.gao.gov/ordering.htm.

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, MasterCard,

Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or Email Updates. Listen to our Podcasts.

Visit GAO on the web at https://www.gao.gov.

Contact FraudNet:

Website: https://www.gao.gov/fraudnet/fraudnet.htm

Automated answering system: (800) 424-5454 or (202) 512-7700

Orice Williams Brown, Managing Director, [email protected]ov, (202) 512-4400,

U.S. Government Accountability Office, 441 G Street NW, Room 7125,

Washington, DC 20548

Chuck Young, Managing Director, young[email protected], (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149

Washington, DC 20548

James-Christian Blockwood, Managing Director, [email protected], (202) 512-4707

U.S. Government Accountability Office, 441 G Street NW, Room 7814,

Washington, DC 20548

GAO’s Mission

Obtaining Copies of

GAO Reports and

Testimony

Order by Phone

Connect with GAO

To Report Fraud,

Waste, and Abuse in

Federal Programs

Congressional

Relations

Public Affairs

Strategic Planning and

External Liaison

Please Print on Recycled Paper.