Veterans Benefits Administration (VBA) Salesforce Center of Excellence, User-Centered Design Challenge
By VA's Technology Acquisition Center (TAC)
Background
The Department of Veterans Affairs (VA) typically uses traditional approaches such as written technical evaluations to select and procure services from vendors. Evaluations of this type tend to consume significant time and effort both by the VA evaluation teams as well as the vendor teams developing the proposals. Additionally, written approach evaluations rely solely on the writing abilities of the vendors without allowing evaluation teams to get a feel for the culture and creativity of the team that will be performing the work. The current VA landscape consists of rapidly changing and evolving requirements/backlogs which require truly agile implementations and execution by vendors supporting VA. Traditional evaluation methods can lead to vendors with superior writing skills winning contracts over vendors that consist of creative, collaborative, and dynamic technical teams.
Jointly, VA’s Technology Acquisition Center (TAC), US Digital Service (USDS), VA’s Office of Information & Technology (OIT), and the Veterans Benefits Administration (VBA) developed an innovative technical evaluation approach to ensure VA and its customers awarded to the highest qualified vendor for a recent Salesforce Development effort which was looking to use User Centered Design (UCD), a methodology very new to VA. The details of how this came to be and our successes and challenges, follow.
The Problem
Typical technical evaluations focus on the end result of the Offeror’s technical approach, generally via a written technical proposal. However, as this project was to follow an agile methodology in accordance with VA’s Digital Transformation Center (DTC) policies, the exact requirements could not be determined up front. We needed to evaluate how a vendor would put the needs of the user first, and not an end state even we didn’t yet understand. After discussion between the procurement team at VA’s TAC and VA’s DTC, User-Centered Design was determined to be the answer.
The team, after further discussions with representatives from other Contracting Teams at the Department of Homeland Security’s (DHS) Procurement Innovation Lab (PIL) and DHS Digital Service, recognized the importance of UCD as a process that could meet our needs and which could act as the differentiator for the award. UCD is a commonly used term to describe the process of focusing on user’s needs throughout a development cycle including: defining a problem from the user’s perspective, designing a solution that prioritizes the user’s needs, testing the solution with real end-users throughout the development process, and defining success in terms of outcomes for users. The team contacted the USDS for technical advice to evaluate vendor UCD capability for the award.
The Solution
What worried the team was that it’s easy to sound like you understand UCD in a written proposal without actually knowing how, or being capable of, actually doing it. Thus we needed to come up with an evaluation strategy that allowed us to see the vendors actually perform UCD. From this need our UCD-Design Off was born. By utilizing a demonstration in lieu of a full written technical approach, we could make the Offerors show us they could do the work versus just telling us, increasing our confidence in their ability to successfully perform. The next challenge was determining what we actually needed to evaluate (and how).
Evaluate the Right Things
After several discussions around the PWS requirements and the constraints of the in-person presentation format, the team came up with the following approach to evaluate UCD. This approach adapted the evaluation approaches by both the VA Appeals team and the DHS FLASH team to focus entirely on UCD capability and meet the limitations of an in-person demonstration. Focusing on UCD was also facilitated by the use of the GSA’s Salesforce BPA which provided a pre-vetted set of highly skilled firms specializing in Salesforce configuration and support. This removed the need to evaluate each Offeror with such scrutiny.
Design Challenge
Vendors were given a simple scenario or “use case” that was meant to test their innovative capabilities to respond to real situations faced at VA. They were given access to two VA employees (one playing the role of product owner and the other playing the role of end-user) to use at their own discretion throughout the scenario. Vendors were asked to create a prototype and provide any documentation created during the design process.
Use Case
“VBA needs a way to manage system design documents. This process involves drafting documentation, collection and consolidation of feedback, and an approval chain. Help VBA determine how to implement this process in Salesforce.”
Format
The in-person demonstrations had a few key characteristics that helped to set the boundaries on the UCD demonstration.
- Each vendor was allowed a maximum of 5 personnel
- Vendors were given 30 minutes on either end for set-up/tear-down of equipment
- The in-person demonstration was 90 minutes (from getting the use case to pencils-down)
- At the conclusion of the 90 minutes, vendors were given 15 minutes to prepare a presentation of their work from the demonstration
- The presentation was limited to 15 minutes and had to cover the following:
- The prototype created during the scenario
- Next steps they would have taken given more time
- How the prototype would be implemented in Salesforce
- How the UCD process and created documentation were used to inform the prototype
- After the presentation, the government took a brief intermission to generate questions followed by a 30 minute Q&A with the vendor to clarify any parts of its performance or presentation. It should be noted that no updates were allowed to the Offeror’s proposal as a result of the Q&As; they were only for the purposes of clarification.
- After the Q&A period all materials created and used during their demonstration were collected for retention by VA in the contract file (e.g. wireframes, screen shots of their prototype, etc.), which were emailed to the Contract Specialist before the Contractors left the building. It should also be noted that materials that were not either created or used in the demonstration were not collected nor evaluated by VA.
Evaluation Criteria
The following criteria were used to evaluate vendor performance during the in-person demonstrations as well as the resulting prototype. Vendors were provided with only the high-level bullets (like Visual Design) prior to the competition. The bullets below each factor were meant as guidelines for the evaluators to help round-out definitions so factors could be evaluated consistently, however, bullets were not meant to be evaluated individually and could cross from one area to the next.
Visual Design
- Design consistency across application
- Hierarchy of information
- Appropriate use of visual elements such as images and icons
- Affordance and discoverability
User Research
- Preparation for user interviews and user research
- Choice and appropriateness of methods
- Quality of methods
- Documentation and presentation of findings
- Translation of findings into design
Design Process
- Integration of user feedback into agile process steps
- Balancing priority of features based on user research and stakeholder needs
- Using agile design methods including prototyping
- Iteration of designs and prototypes
- Assessing multiple design alternatives
Content
- Minimalism in design through prioritization of content
- User driven navigation and labels
- Information Architecture consistent with user needs and industry best practices
- Use of plain language and writing for the web
- Clear errors messaging and follow up actions
Interaction Design
- Appropriate use of navigation
- Indication of system status
- Clear user paths through application
- Contextual help
Conformance with DTC Standards
- Conformance with any design standards from the DTC GitHUB
Bottom Line
The goal of the in-person UCD challenge was to determine if the vendors could, on short notice, assemble a team of competent designers, developers, and product owners which could demonstrate an agile, user-centered design approach to a simple problem.
Evaluating this capability in-person avoided previous concerns expressed by Contracting Officers and Program Managers about vendors providing sound written proposals but lacking real depth in UCD methods and best practices. By reviewing a live design process, the evaluation team was able to quickly recognize how designers were integrated into the development process and whether vendors were following design and user research best practices. This approach also showcased the vendor’s ability to communicate, collaborate, and adapt their UCD process to fit the sample task on the fly. Review of the prototypes allowed the team to assess the vendor’s competence in producing prototypes from user research and further development iterations.
Lessons Learned
- The evaluation factors should focus more on the team dynamics observable in the scenario and less on the prototype developed. Potential additional factors include: agile team dynamics, user feedback integration, and design iteration.
- Set a specific format for the prototype provided at the end of the session, recommend requesting a “Functioning clickable-prototype” which would open up the options to either HTML/CSS code or a prototyping tool like InVision but close out static wireframes.
- Clarify, in the solicitation, the request for not providing any supporting documentation or artifacts other than what was created or used during the demonstration. The government team expected documents like problem definitions, wireframes, user stories, user testing observations, etc. created during the demonstration but received a wide variety of documentation that was not necessary and which it couldn’t accept.
- In the solicitation, overly articulate the difference between an in-person demonstration of UCD from the standard oral capability presentation. The Government has not asked for this type of demonstration in the past and therefore, vendors provided some presentation-style elements that were not evaluated.
- Emphasize the US Digital Service Playbook in the solicitation so that vendors can see the process expectations prior to the demonstration.
- Add additional complexity into the Sample Task either from the onset or at some point during the demonstration to understand how the vendor adapts.
- Ensure the vendors understand that the evaluation team is not part of the demonstration to avoid “sales pitches” during the demonstration.
- Ensure you have subject matter experts from all of the facets that will be evaluated during the demonstration.
- Ensure the evaluation team confers and baselines what are strengths, weaknesses, and deficiencies to ensure a smooth evaluation session.
Task Order Details
Title: VBA Salesforce Center of Excellence
Total Task Order Value (Firm Fixed Price): $62,161,458.83
Period of Performance: 12 month Base plus four 12 month Option Periods and four Optional Tasks
Scope: The Contractor shall provide user centered design, Salesforce configuration and integration support for VBA including project management, requirements elaboration, design, configuration, content creation, testing support, and training support, for the configuration and enhancement of VBA Salesforce applications. The Contractor shall provide support throughout the lifecycle including compliance testing, deployment, operations and maintenance.
Milestones
Solicitation Issued: August 30, 2017
Proposals Received: September 8, 2017
Evaluations Complete: September 22, 2017
Award: September 26, 2017