Technical Evaluation

Tips for Successful Solicitation & Evaluation

Key Question: What techniques can contracting professionals use for evaluating agile contractor capability?

Use presentations as part of the technical evaluation

• Consider including language in the solicitation that the Government intends to require oral presentations as part of the offeror’s technical portion of its quote or proposal. This will enable the Government to determine whether an offeror truly knows Agile software development. This is not mandatory, but has proven to be effective for some agencies.
• Oral presentations need to be tightly controlled and recorded to ensure that all offerors are treated equally, that the Government does not inadvertently open discussions, and to create a defendable record of the agency’s actions.
• If using oral presentations, consider using them after the competitive range is established.
• The Government should clearly spell out the intended use of oral presentations in the Evaluation Criteria if it chooses to use them.

Integrate agile into the technical factors in the Request for Quote (RFQ)

Example:

• Factor 1 – Performance Work Statement: Offerors shall provide a Performance Work Statement in response to the Statement of Objectives (SOO) and this RFQ. The proposed solution shall include an explanation of how project and contract management, communication/collaboration with the Government, security and privacy requirements, documentation, and reporting will function in conjunction with the proposed Agile methodology.

• Factor 2 – Product Development Roadmap: Offerors shall propose an Agile product development roadmap which correlates how the stated objective aligns with the timeframe for implementation and the offeror’s proposed Agile methodology. The product development roadmap shall demonstrate where testing, training, security, privacy, and cut over planning, will be included.

• Factor 3 – Notional Quality Control Plan (QCP): Offerors shall describe the QCP and Performance Measurement approach, including how proposed performance standards will be monitored, evaluated, and reported. The purpose of the notional QCP is to provide evaluators with an understanding of how measures and metrics will be applied based on the proposed technical solution.

Request agile software development-specific information from offerors

• As part of the technical evaluation, request information from the offerors addressing how they manage Agile implementation, techniques for release planning, plans for engaging end users, methods for capturing and applying lessons learned, testing processes, reasons behind the composition of their Agile teams and the rationale behind the proposed development talent and project oversight (tied to Product Vision), how they will make resources available within schedule and budget constraints, and their approach to configuration management.

Evaluate demonstrated experience with agile

• As part of the past experience evaluation criterion, include demonstrated experience with successfully developing software using an Agile approach. This could include a sample of code written and the results of that code.

The contract requirements description for Agile software development contracts and the fact that technical requirements are developed through the Agile process should not increases the risk of protest. Requirements must be defined to the SOO level to allow for vendors to know if they want to compete, and the Government must evaluate based on stated evaluation factors and apply the evaluation criteria consistently among offerors. This is really no different than long-standing requirements applicable to all competitively awarded contracts. Of course, a good debriefing may also help to ward off protests by helping an unsuccessful offeror to understand its weaknesses and how it can be more competitive in future competitions.

Key Question: If system requirements are refined after the contract has been awarded, how can an agency ensure work was evaluated as part of the initial competition and is not considered an out of scope modification in violation of FAR 6.001(c)?

To ensure that all work is within the scope of the contract, as requirements are refined, the software releases (including the end product) must fall within the scope of the Product Vision described in the statement of work, and the agency must give offerors reasonable notice that the scope of the project includes using Agile techniques.

FAR generally requires competing the work that is called for after contract award if the work is not within the scope of the contract. Under long-standing case law, a modification falls within the scope of the original procurement if potential offerors would have reasonably anticipated such a change prior to initial award. In the context of Agile software development, this ultimately means that the MVP, which emerges out of user testing, must fall within the scope of the Product Vision described in the statement of work or statement of objectives. As explained above in Section C, to give offerors reasonable notice, the agency should describe the requirements sufficiently so potential offerors can understand the scope of the project (which should be enough to produce a high level budgetary estimate) and make clear that Agile techniques will be used. This will alert offerors that refinement of technical requirements will occur post-award using a highly-disciplined process of testing and customer feedback. Furthermore, the Government should alert offerors to technical constraints (e.g., platforms) that may be applicable to the effort.

Software requirement changes are expected to be refined and managed as part of the process agreed to up front, addressed in the solicitation, and reflected in the resulting contract. The mere fact that system requirements are refined to reflect the experience from completed iterations does not mean they are out of scope.

To ensure work remains in scope, the Government and contractor should agree to performance standards and the method for assessing the contractor’s work against these standards – which should be aligned with the Product Vision. A service level agreement will be used to specify the levels of service including minimum acceptable service level, target service level, performance standards applicable to each level of service, how service will be measured, the weight assigned to each measure, the frequency of measurement, and the office responsible for measurement. As long as the features support the Statement of Objective (SOO) and are delivered within the set schedule and budget, they should be considered within the scope of the contract. This process is analogous to that used for performance-based acquisitions in FAR 37.6 where the Government issues a SOO defining its outcomes that allows for industry to provide innovative solutions that are measured under a quality assurance surveillance plan.

FAR requires that the Government must describe the general scope, nature, complexity and purpose of what it is acquiring so potential offers can make an informed decision as to whether they want to submit an offer. To enable a prospective offeror to decide whether to submit an offer, the Government needs to clearly state in the SOO that Agile methodology sprint/release framework will be used and that the contractor must propose a method for planning and sizing the work.

The scope should state broad goals and also include functional areas. The Government should fix the schedule/timeframe in which the work is to be completed, so the contractor can propose the number of sprints and the work required in the sprints. If a baseline is available from prior similar work, the Government may be able to propose the set number of sprints to be completed in the schedule/timeframe. To ensure that the prospective offerors understand the environment in which they may be working, the SOO should contain all applicable IT and environmental constraints, including legacy software, privacy, and security requirements or policies that may impact the development of the software. See Sample Language for Government Contracts for Agile Software Development Services in Resources>Learning Center.

Additional resources

• Innovative Evaluation Techniques

  Discover new ways to evaluate Agile proposals in the Evaluation column of the Periodic Table of Acquisition Innovations.

• Technical Challenges

  Use this Technical Challenge Playbook for a creative method of evaluating a vendor's technical agility.

Next: Price Evaluation