Providing Key Usability Testing and Analysis Helped Guide the Texas NIC’s Re-Design of a Free, Online Resource for Texas Vets and Their Families


This endeavor involved working with the Texas NIC* to help them effectively combine a series of extant online resources for Texas Veterans that had been assessed to be ineffective or dysfunctional by the eight different types of former Soldiers, Sailors, Marines, and Air Force and Coast Guard personnel that we worked with as this project progressed. This type of project is heavily informed by a co-participatory approach to designing. This involves allowing those for whom a given interactive experience is being designed to contribute their experiential knowledge to its development as it evolves.

Our faculty-led teams of interaction and user experience designers worked over the course of 16 months with over 25 Texas Veterans and their family members to ensure that their interactions with what would become the new Texas Veterans Portal would meet their needs and aspirations. The Texas Veterans we worked with comprised a diverse group, ranging from those who had served during the Korean conflict to the modern era and who hailed from Austin, Dallas, Fort Worth, Houston, the Red River counties, the Rio Grande Valley and the Panhandle.

*The Texas NIC (Network Information Center) is based in Austin and exists to help organizations that work with or on behalf of Texas-based governmental organizations—or the organizations themselves—design and develop effective technological systems and applications.


This project required that our User Experience research and analysis teams 1) manage and document a series of complex interactions with several different groups of Texas Veterans, and 2) do so in ways that allowed us to effectively determine ”what is was” that caused their greatest frustrations as they attempted to use extant web-based resources to obtain information that was important to them and their families.

This project is a great example of an approach we advocate quite frequently across our project-based courses and research: identify and analyze aspects of situations and sets of circumstances that cause displeasure, frustration, confusion—FAILURE—early on, and then use what we learn to guide future decision-making. Doing this requires that all of the personnel that constitute a given user experience or interaction design research team engage in learning experiences that teach them to empathize with those unlike themselves, and who do NOT or can NOT exclusively draw from their experiential or “book-learned” knowledge.


Observations from Etic Perspectives
Observations from Emic Perspectives
Persona-based Modeling
Scenario Building
Heuristic Analysis of User Scenarios
Low-Fidelity Prototyping

It’s essential to work with people who can operate key “scenarios of use” as the design processes that inform the development of interactive work like the Texas Veterans Portal (TVP) project evolve. Analyzing how “satisfied” or “frustrated” individual Texas Veterans and small groups of them became as they were challenged to use extant web-resources to complete specific challenges was essential to our progress. A large part of the knowledge we were able to gain and share was derived from asking our Texas Veteran collaborators to do things like attempt to secure their GI benefits to fund college tuition, obtain information about getting into Assisted Living facilities throughout the state, and gain the information necessary to apply for a home or property loan.
Primary Objectives

1. To positively contribute to the broad scope of decision-making that was guiding the design of the primary, state-of-Texas-sanctioned online information resource for Texas Veterans.
2. To ensure that the Texas Veterans Portal became a web-based destination from which the incredibly diverse cross-section of over 1,000,000 Texas Veterans could turn desirable information into actionable knowledge.
3. To instigate and sustain more enlightening and fruitful interactions between Texas Veterans and their families and caregivers.

Designing the design of our research: employing user journey maps to guide heuristic analysis

User journey maps? Heuristic analysis?: They’re NOT merely jargon…  The first of these terms refers to a method that afforded our user-experience analysis teams an easy-yet-valuable way to determine whether our Texas Veteran collaborators were 1) actually “satisfied” or “frustrated” as they attempted to get specific info from an extant web resource or another, and 2) what specific aspects of their attempt caused them to experience these feelings. The second of them refers to the approach we took regarding how we asked our Veteran participants to describe their responses to the web-based content they encountered during our testing. Engaging in heuristic analysis entailed their drawing from their own life experiences, worldviews and belief systems as they attempted to operate extant websites to do things like obtain specially designated license plates, bequeath tuition benefits to their children, or learn how to reduce the interest rate on their homes.

A Heuristic Analysis Diagram of User Journey Map 02
Tracking One User’s “Satisaction vs. Frustration Journey” as Her Attempt to Learn About Land Purchase Programs for Texas Vets Evolved

At the outset of this project, our interaction design research teams and our Texas Veteran collaborators identified and assessed the efficacy of over a dozen extant websites that purported to operate as viable information resources. A key aspect of this assessment process involved our student-based teams observing and then interviewing individual Texas Vets or their family members as they attempted to obtain information about specific subject matter from one or more of these resources. The documentation of each Vet’s or Vet family member’s attempt to use these “older” websites to answer a specific question about benefits for Texas Veterans’ is depicted in the three diagrams at left.

A Heuristic Analysis Diagram of User Journey Map 07 (Note: This Map Depicts a Particularly Frustrating User Experience)
Tracking One User’s “Satisaction vs. Frustration Journey” as His Attempt to Learn How to Get His Wife into a Long-Term Care Home Evolved

Operating usability tests that involve UNT Interaction Design faculty and students etically observing and then interviewing actual Texas Veterans yields information that can guide effective design decision-making in ways that methods like eye-tracking CANNOT. All of the Texas Veteran co-participants we worked with during the operation of this set of assessments had accrued experience—much of it negative—with trying to obtain information about their and their families’ benefits from a wide variety of other web-based resources in the past.

Observing an individual’s string of reactions and behaviors as they attempt to operate a given interactive structure or user interface (UI) helped us provide our partners at Texas NIC with information bolstered by understandings of the causal factors that contributed to “what wasn’t working,” or “what was causing frustration.” This information, which is depicted in the three diagrams at left, helped inform the design decision-making that would eventually result in the creation of a Texas Veterans Portal that meets the needs of its diverse group of over 1,000,000 users much more effectively.

A Heuristic Analysis Diagram of User Journey Map 10
Tracking One User’s “Satisaction vs. Frustration Journey” as Her Attempt to Learn How to Get a Handgun Permit at a Reduced Rate Evolved

The kinds of etic observations and subsequent user interviews that transpired during our array of usability tests involving Texas Veterans proved to be an efficient and effective means to provide us and our Texas NIC partners with useful and usable information. Analyzing etic observations of how particular Vets or their family members use specific features and systems embedded in a given interface helped prepare researchers to engage in more broadly informed, deeply plumbed conversations with those individuals later on. This information was then used by design and development teams to ensure that future versions of that interface, or completely new, vastly-different-but-improved interfaces met the REAL needs and aspirations of REAL users.

The three diagrams depicted at left reflect only a small portion of the more than 30 usability tests that were conducted with a broad cross-section of Texas Veterans over a 7-month period. With that stated, it should be understood that, with effective preparation and planning, this type of usability testing can be designed, facilitated, analyzed and documented in a matter of two to five weeks.

The Re-Design of the Texas Veterans’ Portal Allows Texas Veterans to “One-Stop-Shop” for Information

Spending even a small amount of time and money to formulate, operate and analyze data from this type of usability testing early in the interaction design process can (and has been shown, repeatedly…) to save large amounts of time and money later on. This is especially true if significant portions of a user interface, or an entire array of features, or even a whole system, has to be modified or even scrapped once the design and development processes have evolved to the point where even the first, mid-level-fidelity prototype has been created. The kind of usability testing that guided the early stages of the re-design of the Texas Veterans Portal also helped ensure that the Texas NIC teams didn’t squander precious time and monetary resources “trying to solve the WRONG problems.”

Takeaways: what we learned that will guide our future efforts

The type of usability testing that guided the early decision-making during our consulting on the Texas Veterans Portal re-design project exemplifies the value of approaching interaction design projects as co-participatory, human-centered endeavors.

This project was co-participatory in that it could not have progressed effectively without an ongoing collaboration between our teams of designers and design researchers and over 25 individual Texas Veterans and their family members during several phases of its development. It was human-centered in that the entire project was planned, organized and executed so that the criteria for measuring its success was guided by whether the design and functionality of the site met needs and aspirations of its intended users: Texas Veterans.

These types of approaches also heavily inform how we think about and engage in interaction and user experience design. Our processes often begin by working with people who are having or who have had less-than-desirable experiences as a result of interacting with a some portion of a particular interface, or communications system, or product, or network. We then work with them—rather than for them—to empathetically arrive at one or more means to evolve less-than-desirable situations into ones that are more desirable. Our processes are cyclic, as well as flexible and dynamic, and guide the development and implemenation of innovative and sometimes even inventive outcomes.