BoC-Information Gathering, Research and Evaluation

The Board of Corrections' Approach to Information Gathering, Research and Evaluation

The Strategic Plan for the Board of Corrections (BOC) articulates an ongoing commitment to efforts that build high quality information for use in strategic decision making by the local corrections community.  To be effective in its information gathering, research and evaluation activities, the BOC's Research Team follows the five fundamental steps discussed below.

Step 1: Defining the question

Getting the right information has become an exacting science. It starts with asking the right question. For example, the BOC is often asked the question, "How many inmates are incarcerated in local jails?" Such a question is a starting place; however, to ensure the utility of information gathering, the question needs to be refined. Clarifying questions need to be asked. For example, do we want to know:

• The number of inmates by facility? By jurisdiction? Only the statewide summary?
• The inmate characteristics such as gender, ethnicity, offense, current status?
• The total number of inmates on a particular day? Or the average number of inmates per day across a span of time (such as a month)? Or the highest one-day number across a span of time?
• How the number of inmates for one span of time compares to the number for another span of time (for example, comparing last year with this year)?
• The typical length of time that inmates stay in a facility?

Refining the question requires consultation with all of the groups and individuals who have a stake in the outcome of the information gathering process: the Legislature, the BOC, the field, and others. Not only does the question have to satisfy the needs of the stakeholders, but it must also be stated in a manner that can be answered by those who supply the data. For example, in gathering information for the Jail Profile Survey, the question, "Who is currently in jail?" might imply that we would try to obtain names and addresses of inmates, which is both infeasible and unnecessary.  A more precise question would be, "On the average day in November 1999, how many inmates were there in local California jails?"

Step 2: Determining the method for answering the question

Most of the questions presented to the Research Team concern the entire state of California (i.e., 58 counties, each with separate corrections and probation departments and many with multiple detention facilities, as well as cities with adult detention facilities). These local systems employ tens of thousands of people.  The sheer number of counties and cities and the variations in their size and demographics present the Research Team with interesting challenges in this step, which focuses on how the information should be gathered.  Three questions must be addressed in this step:

1.  What data gathering process should be used?

There are many possibilities including: telephone surveys, paper and pencil surveys, web-based surveys, attitude surveys, observation of corrections jobs being performed, subject-matter-expert meetings, literature reviews, field tests with experimental designs, trend analyses and projections.  For the Jail Profile Survey, the chosen method included two surveys, one monthly and the other quarterly, of every local adult detention facility in the state. Many issues are considered in completing this step. For the Jail Profile Survey, the most important factors determining the data-gathering method were that we needed: a) up-to-date information from local facility administrators, b) the ability to quickly assess inmate-population changes, and c) complete data from all jurisdictions and facilities.

2. What approach to "measurement" should be used?

The specific question that needs to be answered in this step is, "What instrument (e.g., paper-and-pencil survey) will be used to gather the data?" In the case of the Jail Profile Survey, a fill-in-the-blanks survey document was developed. Detailed instructions were written for those responsible for completing the survey in local jurisdictions. The survey was field-tested before it became operational. Training classes were held to instruct facility personnel regarding completing the survey. "Measurement" is discussed in more detail below.

3. How often should the data be gathered?

The answer to this question depends on the nature of the information gathering effort.  For some topics, we gather information only once (for example, if we wanted to know the immediate impact of new legislation). However, in the case of the Jail Profile Survey, we need up-to-date descriptive and trend information. Therefore, we gather Jail Profile data every month.

Step 3: Measurement

It is useful to think of information gathering as measurement. "Measurement" is the assigning of values (numbers or assessments) to "things" (such as attitudes, counts of people or objects, employee skills and abilities, training content and training presentations).

The science of measurement has been one of the great advances of the 20th century. In our quest for knowledge, measurement has played a central role. How do we know a phenomenon really exists and has predictable relationships with other phenomena? The answer is measurement -- and for the type of measurement the BOC typically uses, two concepts are critical: reliability and validity.

A measurement is reliable if it is consistent. For example, if you measured the width of a window with a ruler, hopefully you would get a consistent or reliable result. Also, if we asked students to rate the content of a course, hopefully their ratings would be in a fairly narrow range. Otherwise, the result of the measurement wouldn't be very useful. A measurement is valid if it measures what it's supposed to measure. For example, the sales figures for brands of automobile are not valid measures of the quality of the automobile. Price and other issues have to be taken into account. Similarly, a math test composed of difficult calculus problems is not a valid test of arithmetic skill.  In both these examples, the automobile sales figures and arithmetic test scores will be very reliable; thus, it is clear that reliability without validity doesn't count for much.

Most of the information that we collect at the BOC comes from the field. In other words, we most often ask local corrections and probation personnel to make the measurements for us. Therefore, we must take great care to ensure that our instructions are clear, that the data gathering instruments are well designed, and that the data are relevant to what we want to know.

Step 4: Research Design

The BOC's Research Team also gets involved in a special form of data gathering called "research." Although this word means different things to different people, the BOC defines research as follows:  research involves conducting an experiment where one approach (or treatment or intervention) is predicted (using very specific language) to have a better outcome than a comparison approach, or a better outcome than we could expect by chance.

The BOC funds a number of research projects through its Juvenile Challenge Grant, Adult Mentally Ill Offender Grant and Juvenile Repeat Offender Prevention programs. The BOC also gathers common data from all the grant projects in order to conduct statewide research by combining local samples.  While an important undertaking, the BOC recognizes that this latter activity is just one source of information for judging the merits of a particular program.

The BOC has built certain principles into the research that it sponsors and conducts, one of which is that we award grants based our preferences for the following order of research designs:

• true experimental designs, with random assignment of subjects to the new program and comparison groups, is the preferred design;
• quasi-experimental designs, with the new program group matched to a concurrent comparison group;
• historical quasi-experimental designs with the new program group matched to an historical comparison group; and,
• quasi-experimental designs where the new program is applied to an entire geographical area and outcome summary statistics from that geographical area are compared with outcome summary statistics from a matched geographical area.

In 1999, in response to questions raised about the BOC's preferred research design for the Adult Mentally Ill Offender Crime Reduction Grant Program, the BOC issued a summary to clarify its approach to the evaluation component of projects and provide additional information sources on this issue to interested parties (click here to review that summary).

In addition to its research design preference, the BOC requires thorough experimental designs in which the following topics are addressed: the rationale for the hypotheses, the statement of the hypotheses, the criteria for selecting research subjects into the research, the method of choosing the research subjects, the method of establishing the comparability of the treatment and comparison subjects, the sample sizes, the approaches to measurement of the independent and dependent variables, the statistical approaches to hypothesis testing, the data-gathering methods, and the approach to documenting and reporting results.

The BOC understands that most grantee agencies lack the in-house expertise necessary to design and implement the types of evaluations currently required by the Board and, in all likelihood, will need to outsource this activity to external professionals.  The BOC also recognizes that in-depth evaluations require a major investment of resources and remains committed to providing technical assistance to counties, as needed, to facilitate these efforts.

Step 5: Interpreting, Documenting and Reporting Information

If a tree falls in the forest and there is no one around to hear it, does it make a sound? Is information that has no influence on any outcome meaningful? In the information age, what do all these data mean? Those who gather data in an applied research setting for a statewide governmental agency have the following responsibilities:

• The reason for gathering the data must be clear and the data gathering method well designed;
• The data gathering method must be consistent with the reason for gathering the data;
• The data collected must be appropriate for its intended use (e.g., decision making, program evaluation, attitude assessment); and
• The data must be analyzed in a manner that will most accurately reflect the findings and be most useful to the decision makers.

The BOC's approach to information gathering is to report the results as objectively as possible. Care is taken to avoid "going beyond the data" in discussing the meaning of the results and to ensure that the data are translated into an understandable, useful tool for constituents.  Fortunately, the BOC has three researchers on staff with a combined experience of over 60 years in applied (versus academic) research (click here to meet the members of the BOC's Research Team).