Overview of Possible Methods for Collecting Outcome Data on Limited Services

Limited service cases make up approximately three-quarters of the cases that Legal Services Corporation (LSC) grantees close in a given year.1 Such cases are closed with Case Service Report (CSR) Handbook closure codes A (“counsel and advice”)2 and B (“limited action”).3

Despite their prevalence, little is currently known about the outcomes of these types of cases. In extended service cases, grantees usually know what the outcome of the case is because they are involved in the case and have contact with the client for an extended period of time. By contrast, in limited service cases, the interaction with clients is, by definition, limited, involving the provision of legal advice or taking some limited action on behalf of a client in a single interaction. Thus, grantees do not have the opportunity to learn about what happens with the client’s case.

LSC is committed to helping grantees think through ways they can address the challenge of collecting data on the outcomes of limited service cases. This document provides a high-level discussion of some potential methods for collecting outcomes data based on a scan of strategies being employed in the legal services community and in other industries.

Four methods are discussed:

  • Follow-up phone calls
  • Text message surveys
  • Online surveys
  • Tracking administrative records

Follow-Up Phone Calls

How it works

  • A grantee representative can call clients, ask how things turned out with their case, and record the data in some specified format or platform (e.g., in a case management system).
  • The caller could read from a brief phone survey script with standardized questions or follow a semi-structured interview protocol, giving prompts to guide the conversation and learn about the outcome of the case.

Pros

  • Follow-up calls can generate specific and detailed information about outcomes because the caller can ask questions specific to the client’s case.
  • Follow-up calls can provide an opportunity for learning about why the client was or was not successful.

Cons

  • Follow-up calls are time-consuming and resource-intensive, which can make it difficult for grantees to do.
  • Unless clients are told in advance that they might receive a follow-up call, they might be confused by the call or decline to answer it.
  • It is also possible that the client will ask questions or seek additional help, so the caller needs to know how to respond appropriately.

Notes from the field

  • To overcome the challenge related to time and resources, volunteers can be leveraged to conduct the follow-up calls. Atlanta Legal Aid Society and the Boston Bar Association Volunteer Lawyers Project use volunteers to conduct follow-up calls.
  • Atlanta Legal Aid Society uses follow-up calls as an opportunity to provide additional support to clients (not only to collect outcomes data). Their legal aid staff have found this to be a respectful way to learn about outcomes because its representatives are providing additional support and not simply asking more from clients who might feel frustrated that they only received limited service in the first place. Atlanta Legal Aid Society also finds that the provision of additional, follow-up support improves the likelihood that clients will be successful.4

Text Message Surveys (SMS Surveys)

How it works

  • Using an SMS survey platform such as Twilio, Mosio, or Qualtrics, among others, grantees can send automated text message surveys to clients.
  • The text message surveys involve questions with simple answers (e.g., press 1 for “yes,” 2 for “no”) or numeric responses (e.g., satisfaction on a 1-10 scale), making it possible for clients to respond easily via text and for the responses to be coded automatically.

Pros

  • This approach is relatively low-cost.
  • Text message surveys can be distributed automatically and the data are stored automatically, requiring minimal time and resources after the initial setup.

Cons

  • Text message surveys are often ignored by recipients, making response rates low.
  • It would be difficult to program text message surveys to collect data on the range of specific outcomes possible. They might be better used for measuring clients’ satisfaction with the assistance they received or generic outcomes such as whether the issue for which they sought assistance is resolved.

Notes from the field

  • Supported by a Technology Initiative Grant (TIG) from LSC, the Northwest Justice Project (NJP) piloted a project called “Texting for Outcomes” through which they sent text messages to clients receiving limited service assistance to see if the clients were successful in attaining their objectives. Response rates were very low and NJP concluded that texting was not a viable method for them to collect outcomes data.
  • The Legal Aid Society of Cleveland has also received a TIG and is currently developing a system to collect outcomes data via text message surveys.
  • Atlanta Legal Aid Society has a texting system for collecting client feedback. It is used to collect client feedback on various dimensions of satisfaction with their experience. It is not used to collect outcomes data.

Online Surveys

How it works

  • Using an online survey tool such as SurveyMonkey or Qualtrics, among others, grantees can send clients a link to an online survey about the outcomes of their cases.
  • The link can be distributed by email or by text message. Distribution via text message is likely a more effective approach to reaching legal aid clients who very likely have smartphones but some of whom may have limited access to the internet outside of their phone.
  • Questions can be closed-ended (i.e., with specified response options) or open-ended (with text boxes for respondents to provide responses in their own words).

Pros

  • Online surveys offer the potential for collecting more data and more specific data related to outcomes compared to text message surveys.
  • This approach is relatively low-cost and does not require many resources once the surveys are set up.
  • Responses to closed-ended questions are automatically coded and recorded.
  • Responses to open-ended questions can provide additional detail and context to clients’ experiences.

Cons

  • Invitations to online surveys are often ignored by recipients, making response rates low.
  • To increase data accuracy and client response, online surveys need to be designed well, which requires working with a professional or someone with experience designing questionnaires.
  • Making use of information from open-ended questions requires time and resources to read, code, and analyze the responses.

Notes from the field

  • The health care field has successfully used online surveys to track patient outcomes when patients are unable to return for follow-up visits during which outcomes would otherwise be recorded. Examples of survey platforms used in health care include the Socrates Orthopaedic Outcome Software, Ortech Software, and the Surgical Outcomes System, among others.5
  • If using online surveys to track outcomes, it will be important for grantees to carefully construct clear-cut questions that ask about specific outcomes experienced—rather than about clients’ attitudes or opinions about their experience.

Use of Administrative Records

How it works

  • For limited service cases involving the court system, grantees can review court records to track and record the outcome of a client’s specific case.
  • Depending on the state and the court’s system for keeping records and making them publicly available, this process could involve searching an online database or sifting through hard-copy files.

Pros

  • This approach records and captures actual outcomes, not clients’ self-reported perceptions of outcomes.
  • In states where court records are available in digital databases, accessing court records can be a relatively easy way to track the outcomes of cases.

Cons

  • In states where court records are not available in digital databases, use of administrative records can be a very time- and resource-intensive endeavor.
  • This approach can only be used on the subset of limited service cases that involve the court system and excludes many other types of cases such as those relating to public benefits.
  • Using administrative records to track outcomes and link them to the legal assistance they received through extended services is rather straightforward because grantees are involved in the case and know what assistance they provided to achieve the outcome. This can be more challenging with limited service cases because it might not be clear to what extent—if at all—a client’s limited interaction with a legal aid attorney contributed to the ultimate outcome of their case.

Notes from the field

  • Several legal aid organizations use administrative records to track outcomes from limited services, including:
    • Legal Aid of Western Ohio’s Access to Justice Group (on housing cases)
    • Land of Lincoln Legal Assistance Foundation’s Eastern Regional Office (on divorce cases)
    • Mid-Missouri Legal Services Corporation (on cases receiving pro se advice)
    • Community Legal Services, Inc. (on family cases)
    • Boston Bar Association’s Volunteer Lawyers Project (on guardianship cases)
  • 1. In 2016, limited service cases made up 76 percent of all cases served. See: Oh, M. L. Lee, S. John. 2017. LSC by the Numbers: The Data Underlying Legal Aid Programs (2016). Legal Services Corporation, Washington, D.C.
  • 2. In the 2017 CSR Handbook, case closure A corresponds to “Counsel and Advice” cases: “a case closed in which the program provided legal advice to an eligible client should be closed as Counsel and Advice (e.g., the advocate ascertained and reviewed relevant facts, exercised judgment in interpreting the particular facts presented by the client and in applying the relevant law to the facts presented, and counseled the client concerning his or her legal problem).”
  • 3. In the 2017 CSR Handbook, case closure B corresponds to “Limited Action” cases: “a case closed in which the program took limited action(s) on behalf of an eligible client that addressed the client’s legal problem that is not so complex or extended as to meet the requirements for CSR Category L should be closed as Limited Action. Examples include, communications by letter, telephone or other means to a third party; preparation of a simple legal document such as a routine will or power of attorney; or legal assistance to a pro se client that involves assistance with preparation of court or other legal documents.”
  • 4. To learn more about Atlanta Legal Aid Society’s follow-up program, Enhanced Services Project (ESP), click here.
  • 5. Socrates: http://socratesortho.com/; Ortech: http://ortechsystems.com/how-it-works/input/; Surgical Outcomes System: https://surgicaloutcomesystem.com/.