One of the most important tests of the value of a survey is the “sniff test.” We all learned this test from our mothers. Mothers have a highly developed sense of smell and could tell with one whiff when we hadn’t taken a bath, been around someone who had been smoking, and even know what we had been eating.
In the same way research must pass the sniff test. Both the results and how the research was conducted need to make sense and not hint at any bias when the reader gets a whiff. From the beginning we resolved to be transparent about this study’s methodology. So at several stages along the way, we sought out expert advice to test our process. Providing an open process not only made for a better study; it also should allow participants and you the reader to have confidence in the work throughout every phase. Keep in mind – if someone won’t tell you how they know, you might wonder if they really do know.
Here is the process we used to develop and test the Transformational Church research and assessment tool.
Phase 1: Identify Churches for Deeper Learning
The first step of the research was a quantitative research study of Protestant churches. Among the first seven thousand pastors who completed the quantitative survey, their churches were from 123 different denominations as indicated from our sample. Within the survey we had a shorter list to choose from which showed that 6 percent of participants were non-denominational (or had no connection with a denomination-like affinity group, such as Independent Fundamentalist networks). The goal was to establish benchmarks of key metrics, but more importantly to identify the best churches. These churches would be the source of additional insights in Phase 2.
A calling list was randomly drawn from a list of all Protestant churches in the United States of America. Up to six calls were made to reach a sampled phone number. Each interview was conducted with the senior pastor, minister, or priest of the church called. The calling was done in waves of one thousand completed interviews working through a portion of the sample in each wave systematically with quotas for church size (sizes of the churches were proportional to the normal size of U.S. churches). The calling was done by a professional call center with experienced supervision.
After analyzing survey responses, we used the following criteria to determine the most effective churches among the 4,006 churches that were interviewed in the first four waves of Phase 1. To aid in recruiting churches for Phase 3, a total of seven thousand pastors ultimately completed Phase 1 questions.
- The pastor must strongly agree that “our church considers Scripture to be the authority for our church and our lives”
- The church must have grown at least 10 percent comparing the churches current worship attendance to five years prior.
- Because of the volatility in church growth numbers among small churches (e.g., one family coming or going has a large impact on percentage growth), the church must have a minimum of fifty in current worship attendance.
- The church could not have more than one missing response among the ten questions in the survey.
- The church gave us permission to contact them for further research on church health.
- The remaining criterion was a score based on rankings of each of the following seven areas:
- Percent growth in worship attendance (current vs. five years prior)
- Percent of worship attendance involved in some small group, Sunday School class, or similar group
- New commitments per attendee (new commitments to Jesus Christ as Savior through your church in the past twelve months divided by current average worship attendance)
- Percent of new commitments to Jesus Christ who have also become active in the life of your church (last twelve months)
- Percent of adults who attend your church at least once a month who have regular responsibilities at your church
- Percent of adults who attend your church at least once a month who are involved in ministries or projects that serve people in your community not affiliated with your church
- Level of agreement with the statement, “We are consistently hearing reports of changed lives at our church”
As you can tell, we were looking to change the scorecard. We were not saying that things like attendance and conversions don’t matter – they do! But they are not the only things that matter. For too long we have focused only on nickels, noses, and numbers. We wanted to look at lives changed, churches growing, and communities being changed and served. Hence, we started with some expanded metrics.
Phase 2: Qualitative Research to Obtain In-depth Insights
We then contacted the 298 churches with the top 10 percent responses based on the Phase 1 criteria who also permitted us to contact them. We requested to interview the senior pastor, in person if at all possible. With much persistence by LifeWay church consultants, more than 250 interviews were completed, as they traveled to meet pastors and church leaders from Maine to California, from Washington to Florida.
The basis of the interview was a pair of topic guides that the interviewer used to encourage each pastor to describe the activities of the church, why it is working, and how they know it is effective. Because of the depth required in the interviews, the nine general subject areas were split between the two topic guides. Most interviews used only one, but a few generous pastors were interviewed using both.
Each pastor was given the opportunity to provide additional facets of ministry they considered vital to an effective ministry. The additional information provided learning beyond the general areas of inquiry that included worship, evangelism, discipleship, small groups, prayer, leadership, how new people are connected, the quality of life within a congregation, and level of engaging a lost and hurting world.
The interviewers employed by LifeWay in the process were church consultants that had extensive ministry experience. To prepare each consultant and to ensure that each interview was carried out in a consistent manner, two training sessions were provided for conducting interviews.
Each interview was recorded and transcribed. Each church consultant completed a summary immediately following the interview highlighting the most important things they heard. We were able to enhance interview quality by sharing the early transcripts among church consultants for them to hear other interviewers and to observe when useful information was obtained and when it was missed.
Interviewer observations and analysis of the transcripts yielded principles that led to the framework for the Transformational Loop and provided many practical examples that Ed Stetzer and Thom Rainer included in the Transformational Church book.
Phase 3: Church Assessment on the Layperson Level
Principles that were common among the top 10 percent churches were used to develop questions for the church assessment tool. Thom Rainer and Ed Stetzer and our LifeWay Research Team (particularly Scott McConnell and Lizette Beard) looked at them extensively. The questions were reviewed by a panel of experts that included:
- Alan Hirsch, founding director of Forge Mission Training Network and author of The Forgotten Ways and coauthor of Untamed with his wife Debra.
- Chuck Lawless, Global Theological Education Consultant for the International Mission Board as well as Professor of Evangelism and Missions, and Dean of the Graduate School at Southeastern Seminary.
- Elmer Towns, cofounder of Liberty University, author of more than one hundred titles listed in the Library of Congress including The Names of the Holy Spirit and 11 Innovations in the Local Church.
- Bob Whitesel, board member and president of the Great Commission Research Network.
- Bill Easum, senior consultant of 21st Century Strategies and author of A Second Resurrection and Go Big with Small Groups among many other books.
- Gary McIntosh, professor of Christian Ministry and Leadership at Talbot School of Theology and author of thirteen books including Here Today, There Tomorrow: Unleashing Your Church’s Potential and Beyond the First Visit: The Complete Guide to Connecting Guests to Your Church.
- Eric Geiger, vice president of LifeWay Christian Resources and author of Identity and coauthor of Simple Church, Transformational Discipleship, and Transformational Groups.
- Dallas Anderson, national facilitator of evangelism for the Mission America Coalition and director of proclamation ministries for the Billy Graham Center.
Each was asked to comment, edit, and add to the list of questions. An extensive list of questions was chosen initially– twenty questions each for the seven elements of the Transformational Loop – 140 questions in all. The lengthy survey was used so that the paring down of the survey would be based on objective criteria rather than the opinions of a few.
A random set of Protestant churches was then invited to have their pastor and adult members use the Transformational Church assessment tool.
Analysis was done on 4,077 members who completed the initial version of the survey. The analysis validated the scales (sets of questions in each Transformational Church element that have a similar underlying pattern of response). In each of the elements, outliers constituted 1 percent or less of the values.
Factor analysis was run using the questions for each Transformational Church element in the survey. Cronbach’s Alpha for each factor proved very reliable with the lowest being .826. This internal consistency establishes the reliability of the questions in the assessment tool. Individual questions that did not have at least 10 percent of values on each half of the Likert scale were eliminated as were questions that did not load with a factor in their section. The final set of 82 questions comprised the assessment tool.
Phase 4: Church Assessment at the Congregation Level
Additional randomly selected Protestant churches were recruited to test the validity of the final assessment tool. Of the 280 Protestant churches that were randomly selected for this test, 94 obtained enough responses from their attendees to provide results representative of their church with a confidence interval within +/-10 percent. This constituted a 34 percent success rate.
The 82 questions were reassessed for validity and reliability to doubly insure the accuracy of the survey method. With a total of more than 9,000 church-attendee responses to compare to the 4,077 in Phase 3, frequency distributions were found to possess negligible differences. Validity of the final method was supported by this consistency between Phase 3 and Phase 4.
Internal consistency estimates were run to test the reliability of the survey method. Cronbach’s alphas for Phase 4 were all between .87 and .94, and Spearman-Brown coefficients for the survey were all between .85 and .95. These results support strong internal reliability for the survey method. This effectively supports the reliability and validity of the final survey form. Each Transformational Church element possessed statistical properties to support the creation of an “additive scale.” Factor analysis supported the creation of 2 additive scales in the Community element. As a result, an additive sum was calculated for each church attendee that took the survey. Then, a mean value was calculated for each additive scale by congregation. This, in effect, assigned a relative value for a church’s attendees’ agreement with each given section of questions on the survey. The average level of income, education, and age of the church attendees was also calculated and joined to the church-level data that was provided by the pastor in order to account for demographic effects in multiple regression analyses.
The resultant Transformational Church element sums were all highly correlated to each other (above .69). This confirms statistically the convergence of the Transformational Church elements mentioned in chapter 2 of the Transformational Church book.
Separate multiple regression analyses for each sum revealed that one or more of the Transformational Church elements are predictive of each of the following church outcomes:
- 1 Year Worship Attendance Growth
- 5 year Worship Attendance Growth
- New Commitments to Jesus Christ per Attendee
- Guests per Attendee
- Percent of Adult Attendees with Regular Responsibilities
- Percent of Adult Attendees Serving in the Community Outside the Church
In the end, we are confident that the principles were discovered among effective churches. These Transformational Church elements were assembled in a reliable survey that will consistently measure these principles in different churches. The Transformational Church Assessment Tool (TCAT) questions have been and continue to be rigorously tested. The assessment tool produced by this study is both internally and externally valid. And, the actionable Transformational Church principles predict desirable church outcomes such as attendance growth, new commitments to Jesus Christ, and service.