VA dot gov Modernization - Comparative Analysis

By VA’s Technology Acquisition Center (TAC)

POCs

Joshua Cohen, Contracting Officer, TAC, Joshua.cohen2@va.gov
Kelly Reale, Contracting Specialist, TAC, Kelly.Reale@va.gov
Johnston Williamson, Director, Engineering Division, TAC, Johnston.Williamson@va.gov Digital Service at VA, Various

Contract Information

Title: VA.gov Modernization # Applicable FAR Part(s): FAR Part 13.1, Simplified Acquisition Procedures
Total Task Order Value (Firm Fixed Price): $5,998,319.98 (inclusive of options)
Period of Performance: 6-month Base period followed by two 6-month option periods.
Scope: The Contractor will implement a Content Management System and provide additional support for the VA Digital Modernization Strategy through improvements to VA.gov.

Background

VA is committed to dramatically upgrading its user-facing digital tools, becoming the first federal agency to deliver a digital experience on par with the private sector. In recognition of the need to modernize, VA developed the VA Digital Modernization Strategy. The subject of this Case Study is a project central to supporting this strategy through improvements to VA’s primary Veteran-facing property, VA.gov, through the implementation of an open-source content management system (CMS). An innovative project with the potential to drastically improve the digital experience of Veterans required an equally modern and innovative method to evaluate and contract for those services. The approach detailed below was the result, jointly developed by VA’s Technology Acquisition Center (TAC) and the Digital Service at VA (DSVA).

The Problem

The Web Brand Consolidation Working Group (a subcommittee of VA’s Web Governance Board) was created to take action on the modernization of VA's primary Veteran-facing web properties: va.gov, vets.gov, myhealth.va.gov, ebenefits.va.gov, and explore.va.gov. The group identified four key problems with the current Veteran experience across these web properties:

• Veterans cannot find the tools and services they need online
• Users are confused by the disjointed navigation between sites
• VA web sites are designed for the Administrations, not users
• Cross-administration governance of VA.gov is complicated

To address these problems, the Working Group decided to design and build a new Veteran-first experience for VA.gov. This experience will include a new homepage for VA.gov; updated global header, footer, and navigation; new benefit hub pages; and new landing pages for many of the top VA services. The new pages will be served from the Amazon Web Services (AWS) GovCloud (VA Enterprise Cloud) infrastructure currently supporting vets.gov. In addition to the new pages, the content and tools that currently exist on vets.gov will be rolled under the va.gov URL structure and the vets.gov brand will be retired. This work was started under an existing contract and was launched on Veteran’s Day 2018. The effort discussed in this Case Study initiated the implementation of a CMS within the updated VA.gov webpage and will support that implementation over the next eighteen months.

From a contracting perspective, the challenge was to find a capable vendor who had the necessary experience with an open source CMS and significant expertise in agile development and user centered design. Traditionally, this would have taken the form of a request for written proposals which document each Offeror’s overall technical approach and their expertise with an open source CMS technology. However, in the past this has proven an imperfect solution as a written proposal only allows evaluators to read a proposed approach which does not always effectively validate the ability of a vendor to actually perform in accordance with that written approach.

The Solution

To ensure we had the largest possible pool of talent from which to choose, based on market research, it was determined an open market acquisition strategy using Simplified Acquisition Procedures (SAP) in accordance with FAR Part 13.1 was best. Since SAP provides significant flexibility in the evaluation and award of commercial contracts, we tailored an innovative multi-step approach that used Case Studies, a short Written Technical Solution, an in-person technical demonstration, and a final PWS negotiation round with the apparent winner to transition from a Statement of Objective (SOO) (used in the solicitation) to a Performance Work Statement based on the Offeror’s technical approach. This approach allowed us to not only ensure the proper technical expertise and approach would be implemented upon award, but as we were able to actually see the proposed vendor team perform, provided us the confidence they would be capable of meeting our innovative technical needs after the award of a contract. Additionally, SAP provides many streamlined procedures, like comparative analysis, which has the potential to significantly improve VA’s ability to find the right vendor to meet its technical needs. Below are highlights on some of the more innovative aspects of the solicitation/process as well as some lessons learned for how we would do things better next time.

Comparative Analysis

Standard FAR Part 15 acquisitions require each proposal be evaluated against a pre-determined set of technical discriminators included in the solicitation. The problem is, especially for agile services, it can be very difficult to predetermine technical discriminators that are flexible enough to fairly evaluate often very different agile processes. To ensure vendors are not forced to augment their agile approaches to meet often arbitrary solicitation provisions, which can drastically affect their ability to perform, it was determined that the comparative analysis techniques allowed under FAR Part 13 were the best path forward. Comparative analysis allows us to compare proposals to each other, rather than against a pre-determined set of discriminators, so vendors were not forced to fit their proposals into a “box” created by our discriminators. Comparative Analysis was used at each Step of the evaluation process and was instrumental in the success of this acquisition. The acquisition utilized a combined synopsis/solicitation in accordance with FAR 12.603 and it was publicized on the Federal Business Opportunities (FBO) web portal.

Case Studies as Step 1

Among the challenges with an open market approach is you never know how many proposals you will receive, and given that we needed to have this effort awarded in a short period of time, we wanted to make sure we had a solid method to cut the pool of vendors down to a manageable number while also ensuring we had the highest quality vendors to choose from. To facilitate this, the solicitation requested each Offeror’s proposed open-source CMS and a set of Case Studies highlighting each Offeror’s relevant past-experience with respect to that CMS including the following information:

A. Client organization name
B. Period of performance
C. Quoter’s role
D. Goals and outcomes
E. Technology solution
F. Delivery Methodology

This allowed us to quickly sort through those that provided detailed and relevant case studies showing expertise supporting the proposed open source CMS, and those that provided compliant CMSs, but vague or non-relevant Case Studies that indicated they didn’t have the specific skill sets needed to support this project. As expected, we were able to quickly cut down to a small subset of vendors with the necessary expertise and move onto the remaining steps of the process only with those vendors. Specifically, of the seven initial respondents, three were determined to be suitable and capable of meeting VA’s needs and were moved into the next round of the process, Due Diligence.

Due Diligence

The FAR encourages acquisition teams, even under FAR Part 15, to conduct meaningful and informative communications with Offerors throughout an acquisition. However, generally this is either a very formal process like discussions or clarifications, or when posting responses to solicitation or pre-solicitation questions/comments, conducted through written question/answer documents which are shared with all Offerors regardless of who asked the question. This helps ensure fairness in the responses, but can have the unwanted consequence of questions remaining unasked as Offerors are reticent to hint at their proposed approach to competitors. However, meaningful communications are critical to a successful acquisition, and it’s to the Government’s detriment when reasonable questions go unasked. Luckily, the FAR is not as prescriptive in how communications with offerors must be handled, especially prior to the receipt of proposals, as common Government procedures would suggest. Specifically, FAR Part 15.201 encourages exchanges of information prior to receipt of proposals, and for FAR Part 8 and 13 acquisitions, as FAR Part 15 procedures do not apply, even after receipt of proposals, one-on-one discussions (what we refer to here as Due Diligence) are acceptable as long as they are reasonable and held with all technically acceptable offerors.

In our case, to allow for a free flow of information between the vendor and Government, a real-time one-on-one process was determined best. This took the form of separate phone calls with each vendor so they were not concerned with tipping off their proposed approach to their competitors. The calls each lasted 30 minutes and were very helpful to VA and the vendors understanding each other better and allowing for a more informed acquisition process. They were also carefully controlled and monitored to ensure fairness and compliance with the FAR. The Due Diligence was a non-evaluated step so it was not a part of the final award decision or the technical evaluation. After the Due Diligence Step, VA requested Written Technical Solution and Price Volumes and scheduled each vendor’s Technical Demonstration.

Written Technical Solution

Each Offeror was asked to send a 10-page Technical Solution related to how they would implement the CMS within the new VA.gov website. Specifically, each Offeror was asked to describe the following:

1. Overall methodology and approach to plan, implement, and configure the proposed CMS for VA.gov.
2. Knowledge and approach to Agile software development.
3. Knowledge and approach to User Centered design.
4. Knowledge and approach to Development Operations (DevOps).
5. Knowledge and approach to Content Writing.
6. What the Quoter would need from the Government to ensure success and any barriers that would reduce or delay success.
7. How success and end user satisfaction will be determined and the strategy for capturing both product metrics and process metrics.
8. The proposed Labor Mix and Level of Effort by Iteration supporting the proposed firm-fixed-price. This description shall indicate whether the Labor Category is being proposed for the Prime or a subcontractor including which proposed subcontractor. Please include Labor Category descriptions for each Labor Category proposed including the experience, skill sets, and education for each Category.

The Written Technical Solutions were similar to the typical Written Proposals, however, since we had a Technical Demonstration we could use to measure each Offeror’s agile process and capability and suitability to do the work, we were able to keep the Written portion significantly shorter which helped speed up the process but also allowed our technical evaluators to see at a high level how each vendor would tackle the specific technical needs of the project. In addition to the Written Technical Solution, each Offeror provided a price proposal volume based on its Technical Approach.

Price Proposal

It is difficult in advance of seeing the exact agile methodology of each vendor, where things like the length of iterations and make-up of the proposed team may differ greatly between contractors, it can be difficult to figure out how to set up a Schedule of Deliverables (Section B) for the solicitation that does not box a vendor in and force them to augment their normal technical approach. To allow vendors the flexibility to provide pricing in line with their agile approach, vendors were provided a template which they could tailor. Specifically, the solicitation requested the following:

Vendors shall fill into the provided Section B document their proposed Contract Line Items (CLINs) and provide fill-ins as included in each CLIN, and a unit price and extended price for each CLIN. Vendors are free to add additional CLINs to support their proposed price. Additionally, a price proposal shall be submitted in Microsoft Excel spreadsheet format. The first tab shall be a summary to include a top-level rollup of the total dollars and percentages by labor, materials, travel, Other Direct Costs, and total Task Order price. Labor shall further be broken out by labor categories, labor rates, and hours. A separate tab shall be used for the Prime and each Subcontractor. Additionally, any material or travel handling rates proposed for the Material or Travel line items shall be noted as well.

This allowed vendors flexibility but also required them to provide the back-up for that pricing, including applicable Labor Hours, Labor Categories and Mix, and Labor Rates, which allowed VA to conduct a thorough and sufficient price evaluation and allow for the determination of a fair and reasonable price. This approach also worked with comparative analysis as its less important to have an apple-to-apple’s basis for comparison in the selection decision. Additionally, as its often difficult for vendors to determine how much work is involved with agile requirements, as the requirement document speaks more to a requested process versus how much of that process is going to be needed (since the Government cannot specifically determine that either), we released our budget/estimated value from our Independent Government Cost Estimate in our solicitation. We were less interested in who could come in with a lower price as to who could maximize the value of the budget we were provided. This also helped in the best value trade-off determination as it kept pricing somewhat aligned and did not leave us with wildly differing prices that were based on each vendor’s assumptions.

Technical Capabilities Demonstration (TCD)

The last aspect of the Evaluation was the TCD. The goal of the TCD was to create a working prototype in real-time in response to a time-limited scenario. Each vendor was given a scenario detailing a fictional government problem and had four hours to create a working prototype solution in the proposed CMS (must be same as what was included in the Case Study). Two Government employees were provided playing the roles of 1. Business Owner and 2. End User for the scenario. Offerors were allowed up to 6 total attendees for the Demonstration. A whiteboard was made available, but otherwise vendors were required to provide anything necessary to conduct their demonstration (including internet access). This is an important point as if the Government takes responsibility for any other aspect of the set up for the Demo, should there be an issue with the internet in the location or something else the Government is responsible for providing, it can create a big problem for the acquisition that may not be easy to fix (especially if there are multiple vendors and only the last vendor had the issue). The TCD provided an opportunity for the team to demonstrate the team collaboration, agile methods, user-centered design, CMS configuration, content writing, and iterative development skills needed to meet VA’s needs and to support its proposed approach from the Written Technical Solution. Each TCD was held during the morning on a separate day. This allowed for technical evaluations to start directly after the end of each TCD when the details of each Demo were fresh in each evaluators’ mind.

Each vendor was provided a problem to solve in the four-hour TCD period. They were free to move around the room and interact with the two VA staff provided as the user and the stakeholder however there was no interaction allowed between the vendor and any other Government evaluator in the room. It should also be noted that the user/stakeholder were not evaluators and were there simply to play a role from the Use Case each vendor was asked to solve. At the completion of the TCD, each vendor was required to submit the following:

1. A URL to a private git repository on either Github.com or Bitbucket.org which includes the source code. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access the repository;
2. Documentation for setting up the development environment and running the application including a checksum hash pointing to the revision to be evaluated on the master branch;
3. A publicly-accessible URL to the Quoter’s prototype at the top of the README file;
4. A URL to a private administrative panel for the CMS portion of the prototype. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access the administrative panel;
5. Access via URL to any additional tools used to track work (e.g. JIRA, trello) used during the TCD. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access these tools;
6. Documentation, including diagrams, of the overall architecture.

All supporting digital and non-digital artifacts created during the design and development of the prototype were to be uploaded into the repo. Examples of artifacts included user stories, wireframes, and test plans. These artifacts were required to be representative of the vendor’s proposed process for documenting work. Images of non-digital artifacts created during the demonstration (e.g. white board drawings) were also uploaded to the repo.

Initial Best Value Determination

Once the TCDs were complete, the Government created a Comparative Analysis Evaluation document where a comparison was made between each Offeror’s Written Technical Solution and TCD. This document, along the final Price Evaluations, was used to determine the Best Value proposal which was documented in a Comparative Analysis Decision Document (CADD) which was signed and included in the award file.

Negotiation of the final PWS and Schedule of Deliverables

Following the evaluations and the initial Best Value determination, the apparent winner was asked to provide a final PWS and make any Government requested changes to the Schedule of Deliverables (we actually didn’t have to request any pricing changes) based on their technical approach and pricing proposal. Both were then reviewed by the Government and after a short discussion and some minor updates, were incorporated into the award document and the award was signed. Each vendor was notified in the solicitation that should finalizing the PWS or Schedule of Deliverables lead to changes to the Technical Approach or Evaluated Price, the Government reserved the right to revisit its Best Value determination and potentially award to a different vendor. Upon completion of negotiations, a short Addendum was added to the CADD documenting that the decision remained the same and did not change as a result of negotiations.

Lessons Learned

The process was successful and allowed the Government to find a vendor capable and experienced enough to meet its demanding technical requirements on a very important and high-profile project. However, as with any time we try something new, there were several lessons learned that we’ll use to refine our process for next time:

• When the required timeline for an acquisition allows for it (and sometimes it just might not depending on the Government’s specific need(s)) we can allow for a review of the Written Technical Solutions prior to determining who would move on to the TCD. A TCD can be costly and a require significant level of travel and coordination between the prime vendor and its proposed subcontractors (especially for small businesses). However, there may be issues in the Written Technical Solution that would preclude a vendor from winning the award and which may not be easily correctable through the evaluation process and which may render a vendor incapable of winning the award. If this is the case, vendor’s may expend the time and money to support the TCD when they have no reasonable chance of winning the award. The costs associated could be mitigated by notifying them ahead of the TCD whether they had a reasonable chance to win or not, or by setting up a formal cut-down prior to the TCD. In future acquisitions, when its appropriate, we will try to be more careful with the order of steps and notifications to avoid any chance of this occurring. Additionally, FAR processes like multi-step advisory (FAR Part 15.202) can be used in this type of scenario.
• Ensure the evaluation team is experienced and knowledgeable in whatever will be demonstrated in the TCD. We had some extremely experienced and highly technical evaluators that were instrumental to the acquisition’s success.
• When using a Statement of Objective, as was the case for the VA.gov modernization solicitation, ensure evaluation criteria are well understood by both the government evaluators and the vendors proposing the effort. It is difficult to compare technical approaches that can vary greatly from vendor to vendor and its doubly hard to do when there are differing expectations between the participants.
• Evaluation procedures similar to those documented above may be used to specifically entice non-traditional vendors to enter the government as hopefully (and we have a lot of work left to do on this point) it better mirrors commercial practices, at least as compared to standard government processes. However, this brings with it a new set of challenges. New entrants to the market are not familiar with many processes that are standard and well understood in our community (like restrictions on communication with vendors during an evaluation or specific formatting issues) and this can lead them to make mistakes or assumptions a typical government vendor would not, but which can have a serious effect on their ability to win the contract. This requires Government to try as best as possible to state things in plain language or to be extra explicit in its instructions where maybe it wouldn’t be under different circumstances. It is easy to overlook these types of issues as generally they are common knowledge and well understood, and vendors do have the opportunity to ask questions during the Due Diligence step, but this isn’t always enough. Its understandably frustrating for vendors that have spent a lot of time and effort on a solicitation to feel like they lost based on a technicality (even if it’s really not a technicality) because they did not fully understand the solicitation requirements, case law, or standard procedures well enough. If we want them to stay enticed and bring innovation and new technologies into our market we must support them accordingly.
• Although it is enticing to try and make the TCD as complex and thorough as possible to try and get at every bit of expertise a company can bring to the table, this should be balanced with the time and cost involved. Our TCD was successful in indicating which vendors could meet our needs and which could not, but could we have made that decision with less steps or in less time? I am not sure, but we should be asking that each time we develop an innovative evaluation process and I think should limit procedures to only those necessary to properly evaluate their capabilities. For instance, other TCDs have taken the form of the submission of technical prototypes or artifacts from past projects that required significantly less and time and money than an in-person demonstration.
• Consider, after the protest period (if applicable) and formal debriefs (if applicable) are over, reaching out with a short informal survey of the proposing vendors to see what they would have liked to see differently or where things worked well. This in my experience is rarely done and can lead to insights the Government would never benefit from had they not asked. We asked and will be incorporating several of the suggestions into upcoming solicitations.

Bottom Line

Both the Government and vendors felt the approach was positive and suggested we should continue using similar procedures in the future. The Case Studies allowed us to cut down to a manageable number of vendors in a way that was fast for Government evaluators and did not require much from the vendors. Additionally, we were able to request a short Written Approach that was focused on the outcomes and specifics of the approach, without the long and unnecessary marketing pitches vendors feel the need to include to establish their technical capabilities, because we were able to verify those capabilities in a demonstration that we watched live and which was over in four hours. This helped shorten our overall schedule greatly and avoided having to read a fifty-page technical volume that may have been written by a consultant hired solely for that purpose. The Comparative Analysis allowed us to compare differing technical solutions without the burden of developing pre-determined technical discriminators. There are certainly things we will refine in future acquisitions but we felt the acquisition was very successful and did for us exactly what we were hoping to do—find the best possible innovative solution in the shortest and easiest way possible for all parties. It was also fast, from date of the acquisition package being finalized to the date of award was around 6-weeks. Even better, there wasn’t a thing we did that isn’t clearly allowed by the FAR!!

APPENDIX: INSTRUCTIONS, CONDITIONS, AND BASIS FOR AWARD

Step 1: Proposed Content Management System (CMS) and Case Studies

A. Proposed CMS

Each Quoter shall include which CMS it is proposing in response to this solicitation. The CMS shall comply with Statement of Objectives (SOO), specifically paragraph 5.2.1 “Operating Constraints.” Quoter shall provide a brief description of why it chose the proposed CMS and how it satisfies the requirements of the SOO. Each Proposed CMS submission is limited to 2 pages.

B. Case Study Submission

Quoters shall submit three relevant case studies for evaluation. Relevant case studies must demonstrate recent (within the past three years) performance of tasks, detailed in the Statement of Objectives (SOO), related to the proposed CMS, performed by the quoter or any proposed subcontractor who will be responsible for at least 30% of your proposed approach. Quoters are strongly encouraged to submit case studies that demonstrate capability to perform multiple tasks from the SOO and include where those services supported the proposed CMS in circumstances similar to those contained in SOO Section 5.2.1 “Operating Constraints.” The Case Studies shall demonstrate an agile methodology and adherence to practices found within the US Digital Service Playbook. Each case study submission is limited to 3 pages in PDF Format.

Quoters must include the following details for each case study submission:

A. Client organization name
B. Period of performance
C. Quoter’s role
D. Goals and outcomes
E. Technology solution
F. Delivery Methodology

**Please note that Quoters shall ensure that any subcontractor whose relevant past experience is utilized for a Case Study provided in Step 1 is also included as a proposed Subcontractor in each future Step including any resultant award. Failure to ensure this may render an Quoter’s quote unacceptable.

Proposed CMS and Case Studies Evaluation

The Quoters Proposed CMS solution will be evaluated to determine whether it can meet the requirements of the Operational Constraints in Section 5.2.1. The Quoter’s Case Studies will be evaluated to determine the Quoter’s capability and suitability to perform the work required in the Technical Solution. The Quoters determined to be most suitable and capable resulting from Step 1 will be selected for Step 2. The Government will determine the appropriate number of Quoters for Step 2 that is most beneficial to the Government.

Step 2: Due Diligence (NON-EVALUATED STEP)

Date: XXXXXXX

This is a single meeting with each Quoter who has been selected to move forward from Step 1. This meeting is considered the question & answer opportunity and is a non-evaluated step. No comments, information, or questions presented by the vendors will be considered in the evaluation. This time is open for vendors to ask questions in order to limit the amount of assumptions that will be included as part of the Quoter’s technical or price solutions. Further instructions for Step 2 will be provided to all selected Quoters. Please note, when specific information that is determined necessary for the preparation of proposals is disclosed to one or more potential quoters, that information will be made available to all quoters.

Step 3: Submittal of Written Technical Solution, Price, and Technical Demonstration

Only open to Quoters who have been selected to move forward from Step 1. Submit Written Technical Solution and Price Volume to XXXXX no later than XXXXX.

Submit the following:

A. Written Technical Solution limited to 15 pages excluding cover letter and table of contents from page count in PDF Format.
B. Price Volume

A. Written Technical Solution

The Written Technical Solution shall demonstrate the Quoter’s ability and expertise to deliver a solution that meets the established needs and purpose of the RFQ. The Quoter’s proposed solution shall identify how the objectives will be met as stated in the Statement of Objectives. Within the Written Technical Solution, the Quoter shall demonstrate its:

1. Overall methodology and approach to plan, implement, and configure the proposed CMS for VA.gov.
2. Knowledge and approach to Agile software development.
3. Knowledge and approach to User Centered design.
4. Knowledge and approach to Development Operations (DevOps).
5. Knowledge and approach to Content Writing.
6. What the Quoter would need from the Government to ensure success and any barriers that would reduce or delay success.
7. How success and end user satisfaction will be determined and the strategy for capturing both product metrics and process metrics.
8. The proposed Labor Mix and Level of Effort by Iteration supporting the proposed firm-fixed-price. This description shall indicate whether the Labor Category is being proposed for the Prime or a subcontractor including which proposed subcontractor. Please include Labor Category descriptions for each Labor Category proposed including the experience, skill sets, and education for each Category.

Technical Assumptions, Conditions, or Exceptions – The Quoter’s Written Technical Solution shall include all (if any) technical assumptions, conditions, or exceptions related to any of the requirements or terms and conditions of the Statement of Objectives. If not noted in this section of Quoter’s quote, it will be assumed that there are no assumptions, conditions, or exceptions for award, and that the Quoter agrees to comply with all of the terms and conditions set forth in this RFQ. It is not the responsibility of the Government to seek out and identify technical assumptions, conditions, or exceptions buried within the Quoter’s submission. The Government reserves the right to reject any quote that includes any technical assumptions, conditions, or exceptions that impact or affect the Government’s objectives or requirements.

B. Price

Quoters shall submit a price volume which shall include the following:

• Completed Section B and price proposal excel spreadsheet
• Supporting documentation as described below
• Assumptions, conditions, and exceptions related to price

Section B and price proposal excel spreadsheet: Vendors shall fill into the provided Section B document their proposed Contract Line Items (CLINs) and provide fill-ins as included in each CLIN, and a unit price and extended price for each CLIN. Vendors are free to add additional CLINs to support their proposed price. Additionally, a price proposal shall be submitted in Microsoft Excel spreadsheet format. The first tab shall be a summary to include a top-level rollup of the total dollars and percentages by labor, materials, travel, Other Direct Costs, and total Task Order price. Labor shall further be broken out by labor categories, labor rates, and hours. A separate tab shall be used for the Prime and each Subcontractor. Additionally, any material or travel handling rates proposed for the Material or Travel line items shall be noted as well.

Supporting documentation - Documentation is required to support the pricing proposed. This shall demonstrate the correlation between the proposed technical solution and the Section B submitted. The supporting documentation shall also include a Basis of Estimate (BOE) which aligns to how the pricing methodology is applied within each iteration. The BOE shall include, but is not limited to, such things as:

• Number of Teams proposed
• Size of Agile Teams
• User Story sizing methodology
• Any discounts applied

Price assumptions, conditions, or exceptions – Submit all (if any) price assumptions, conditions, or exceptions related to any of the terms and conditions of the Statement of Objectives. If not noted in this section of the Quoter’s quote, it will be assumed that the Quoter proposes no price assumptions, conditions, or exceptions for award, and agrees to comply with all of the terms and conditions set forth in this RFQ. It is not the responsibility of the Government to seek out and identify price assumptions, conditions, or exceptions buried within the Quoter’s quote. The Government reserves the right to reject any quote that includes any price assumptions, conditions, or exceptions that impact or affect the Government’s objectives or requirements.

C. Technical Capabilities Demonstration

Date: XXX - XXXX
Location: 23 Christopher Way, Eatontown, NJ 07724

Only open to Quoters who have been selected to move forward from Step 1.

The goal of the Technical Capabilities Demonstration (TCD) is to create a working prototype in real-time in response to a time-limited scenario. Quoters will be given a scenario detailing a fictional government problem and will have 4 hours to create a working prototype solution in the proposed CMS. Two Government employees will be provided playing the roles of 1. Business Owner and 2. End User for the scenario. Offerors are allowed up to 6 total attendees for the Demonstration. A whiteboard will be available. Please note that there is no guarantee of internet access and Quoters shall plan accordingly.

This is the opportunity for the team to demonstrate team collaboration, agile methods, user-centered design, CMS configuration, content writing, and iterative development skills that will be needed to execute the SOO. The process used to develop the prototype shall demonstrate the same solutions detailed in the Written Technical Solution.

In terms of design, the coding submission shall be accessible (i.e. 508 compliant); usable on both desktop and mobile; and follow the U.S. Web Design Standards (USWDS) where applicable. The coding submission shall render correctly in at least one of the following modern browsers on both desktop and mobile: Chrome, Safari, and/or Firefox. Custom functionality may be added if it is not included in USWDS. An environment may be set-up in advance of the demonstration along with any non-content specific work. Any files not used as part of the prototype shall be removed before the end of the demonstration.

The coding submission shall use the Quoter’s proposed Content Management System technology. At the completion of the demonstration, the vendor shall submit:

1. A URL to a private git repository on either Github.com or Bitbucket.org which includes the source code. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access the repository;
2. Documentation for setting up the development environment and running the application including a checksum hash pointing to the revision to be evaluated on the master branch;
3. A publicly-accessible URL to the Quoter’s prototype at the top of the README file;
4. A URL to a private administrative panel for the CMS portion of the prototype. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access the administrative panel;
5. Access via URL to any additional tools used to track work (e.g. JIRA, trello) used during the TCD. Quoters shall add account “VA.gov Modernization” as a collaborator so that the Government Technical Evaluation team can access these tools;
6. Documentation, including diagrams, of the overall architecture.

All supporting digital and non-digital artifacts created during the design and development of the prototype shall be uploaded into the repo. Examples of artifacts include user stories, wireframes, and test plans. These artifacts shall be representative of the vendor’s proposed process for documenting work. Images of non-digital artifacts created during the demonstration (e.g. white board drawings) should be uploaded to the repo. Evaluators will be present for the entire TCD.

Artifacts, additional technical solution materials, or other non-germane documents not directly related to the design and development of the prototype will not be accepted. The contracting officer has the ability to remove any documentation submitted that does not support the TCD.

The Government will schedule the demonstrations by drawing lots among those Quoters who are selected in Step 1. The Government will advise Quoters of the date and time of their TCD which is anticipated to be held between XXX and XXX.

The Government will have the ability to ask clarifying questions specific to the Quoter’s prototype solution during the time allotted for the TCD. These do not count as discussions, unless otherwise directed by the Contracting Officer. No updates will be allowed for the TCD, however the Government reserves the right to enter negotiations on the Quoter’s Written Technical Solution or Price Volume.

Written Technical Solution, Price, and TCD Evaluation:

The Written Technical Solution will assess the Quoter’s overall approach to the project and what, if anything, it would need from the Government to ensure success as well as identifying any barriers that would reduce or delay success. The TCD will be evaluated to determine the Quoter’s capability and suitability to perform the work as proposed in the Written Technical Solution. The technical capabilities demonstrated will be assessed to determine if the Quoter’s methodologies will result in the continued delivery of high-quality product, and will meet the objectives for digital strategy implementation. The TCD will be evaluated by assessing performance in the following categories:

• Ability to Define the Problem
• Ability to Design a Solution
• Iterative Approach
• Documenting the Work
• Ability to Produce a high-quality Prototype

The Government will evaluate price by adding the total of all line item prices, including all options. The total evaluated price will be that sum. The Government will adjust the Quoter’s proposed Total Evaluated Price if mathematical errors are identified.

Step 4: Negotiation of Performance Work Statement and Quality Assurance Surveillance Plan (QASP)

Following Steps 1, 2, and 3, in consideration of the Basis for Award, the apparent successful Quoter will be chosen to provide a final PWS, QASP, and associated minor price adjustments (if necessary), which will be negotiated and finalized with the Government. If a final PWS cannot be worked out, or fails to provide best value solution award following negotiations, then the Government may select the next highest valued vendor for negotiations of a PWS and QASP.

Basis for Award

The determination of the BEST VALUE quote using comparative analysis in accordance with 13.106-2(b)(3) will be based on:

1. Non-Price:
   a) Proposed CMS and Case Studies
   b) Written Technical Solution
   c) TCD

2. Price:
   a) Evaluated Price

Any award will be made based on the best overall (i.e., best value) quote that is determined to be the most beneficial to the Government, with appropriate consideration given to the Written Technical Solution, TCD, and Price.