Guide to phone survey data quality control

This article supports the computer-assisted telephone interviewing (CATI) starter kit guide.

Monitoring the quality of phone survey data isn't very different from face-to-face survey data, so if you have experience monitoring survey data in general, you should cope well. However, there are some phone survey-specific data quality features and some general advice we can offer about phone survey data quality, which we'll explore in this article.

This article will be structured as follows:

1. Useful features from face-to-face survey (CAPI) quality control
2. New features specific to phone survey (CATI) quality control
3. Recording phone calls
4. Supervising phone surveys (CATI)
5. Monitoring your project's progress
6. Summary

1. Useful features from face-to-face survey (CAPI) quality control

• As you may be aware, SurveyCTO has been a popular choice for traditional face-to-face surveys in hard-to-reach places, given that SurveyCTO Collect works 100% offline. Not only that, a range of form and server-side features built into SurveyCTO Collect and the server console enable the collection of high-quality data. These are well-documented, but briefly, these features are:
• Form designs, including relevance, constraints, which are well-tested.
• Metadata like the start, end, and duration values. These can help you understand how long the form was open, summarizing the time window that data capture took place.
• Text audits which time stamp the first arrival time, and time spent value for each field in seconds.
• Time stamps which you can add using the calculate_here function in combination with once() and/or duration(), or other time-related functions to mark the time a particular point in the form was passed. See this sample form.
• Audio audits which provide randomly recorded, strategic audio recordings of interviews (see this blog post for advice on audio audits). These work with phone calls too, on Android versions 4.4 through 7 (more on this below).
• Speed violations for flagging signs of rushing through forms that could be suspicious.
• Sensor metadata for monitoring how the device is being used, and its environment. Audio metadata works on phone calls too (when they're detectable). Less useful than in the context of face-to-face interviews though.
• Data monitoring using the Data Explorer, or another visualization tool, via data publishing.
• Automated quality checks can be configured to automatically flag records for review. This could be based on explicit questions, the above metadata, and some of the new data quality features below (see our videos on data monitoring).

2. Features specific to phone survey (CATI) quality control

We've developed a number of phone survey-specific quality features that we strongly recommend you integrate into your phone survey form designs. These are complements to the above already-rich set of data quality features, not alternatives.

You must use SurveyCTO Collect for Android version 2.70.6+ to benefit from these new phone data quality features. These new CATI quality control features aren't supported by iOS.

Using SurveyCTO Collect's phone call management features

The SurveyCTO Collect app for Android sports in-app phone call management features which are a great convenience, but also an enabler for phone survey data quality. While SurveyCTO Collect acts as the phone app on Android, this affords greater access to very useful data quality indicators, such as the duration of phone calls, and the most detail-rich phone call log (see below for these).

With this in mind we recommend that you do the following:

1. Tap on the three-dot menu in the upper-right of SurveyCTO Collect, open Admin Settings, find Set as default phone app to the most demanding setting appropriate (might vary if enumerators are using their own phones).
2. Use the collect-is-phone-app() function in the form to record whether calls were made with SurveyCTO Collect, so you can understand the data better.

Remotely configure identical SurveyCTO Collect settings (including Set as default phone app) on any number of Android devices using quick setup.

Getting the phone call duration

The most basic data quality question with a phone survey is "did a phone call take place?". You can answer this question and understand very quickly on a basic level, whether the data submitted in the form is credible but recording the phone call duration. Do this with the phone-call-duration() function. 

The phone-call-duration() function stores the time in seconds a phone call was active while a form was open. If there were multiple calls, this function stores the duration of all calls combined. Just like with the duration() value (the time a form was open), you can compare this value, aggregate it, and use it to compare enumerator performance. Comparing the two values will give you the percentage of time that the form was open where the call was active. 

Very brief phone call durations will lack credibility, but try to understand call duration in the context of the distribution of your call data. The best thing about phone call duration is that it is an immediately understandable, simple number that tells you something crucial about a phone interview.

Reviewing phone call logs

Beyond the question "did a phone call take place?", you will be concerned whether the relevant phone call took place, i.e., which phone number was called and at which point of the survey the call was conducted. The phone-call-log() function can help you do this. Every event (such as calls starting or ending) is logged with a timestamp, so you can establish a timeline of contact attempts that happen while a form is open. See an example of the log content below.

Form started||0
Call started|5555555555|12
Form exited (call in progress)|5555555555|25
Form resumed (call in progress)|5555555555|26
Call ended|5555555555|45
Form exited||77

Phone call logs are more work to immediately understand compared to phone call duration (a single integer value), as they're a set of repeated events in a list. 

To get the most out of phone-call-log(), see this guide. Take a look for ideas on how to summarize the phone call log, and clean it up to be more immediately readable.

Note that depending on the make of your device and its Android version, the phone numbers you dial may not be available to SurveyCTO Collect, so might be missing (for security reasons). However, as a fall-back, the phone numbers you dial should originate from pre-load data stored in the form, and the method you use for starting the phone call should store details about the call including the phone number. However, populating the dialer and sending a phone number to another app to complete a call is not necessarily proof that a call took place.

3. Recording phone calls

Recording phone calls (or strategic sections thereof) with audio audits can offer great insight into the quality of data. Depending on context, a different approach to recording calls may be required.  See your options in the table below:

Survey context	Available options	Notes about version compatibility
Enumerator is filling out the form on an Android device with an active SIM card.	Launch the call using SurveyCTO Collect and use Collect's built-in audio audit feature.	In Android versions 4.4 - 8, this should work fine, capturing both sides of the conversation. Support for recording phone calls is limited in Android 9, and will not work in Android 10+. If you're having trouble recording the other side of the conversation in Android 9 or 10+, you can try turning on the Admin Setting Use VOICE_CALL instead of MIC for audio during calls, which might help. To see a list of devices that have been tested for this functionality, see here.
Enumerator is filling out the form on an Android device without an active SIM card, but has access to a separate device with an active SIM card.	Use speakerphone on a separate device to make the call, and record the audio using Collect's built-in audio audit feature. Be mindful of respondent privacy when using speakerphone.
Enumerator is filling out a web form, but can receive calls on a different device.	Use a cloud communication service like Twilio/Exotel. Most of these support recording the audio from the call. The drawback here is that the audio will not be mapped to the questions being asked. So you might have a difficult time finding the audio during a specific question.

Listening to call recordings can be fruitful, but is also very labor-intensive, so consider strategies for which calls you listen to, and which questions you focus on (somewhat like the random recording strategies we suggest with audio audit recordings).

We have tools available to make Twilio call recording much easier. You can use them to record calls, stop or delete the recording during a call, and download the recordings when you are ready. To learn more, check out our support article on using Twilio.

4. Supervising phone surveys (CATI)

Much of what you did before in terms of supervising a survey can work for phone surveys too. For example:

1. The role of the supervisor. Supervisors can still remotely accompany enumerators while they make calls, helping to support the team, while promoting high quality data.
2. Back-check components to phone surveys can also work. Just like with face-to-face surveys, draw a random sample of successful interviews and follow-up with the respondents via phone to confirm that the original interview took place.

Besides the above, there are a few additional actions you can take to improve your CATI data quality:

1. Think about following up via bulk SMS to confirm the interviews you have on record, or simply to remind respondents about the interview
2. Consider building interview quality and training reinforcement straight into the survey by covertly including phone numbers of supervisors/back-checkers, who will be trained to act as respondents when enumerators call them. If you use SurveyCTO Collect's call management features or voice services like Twilio or Exotel, the respondent phone numbers and the numbers of back-checkers can be hidden, so the same numbers could be used again and again.
3. Finally, something that can help a lot when managing a phone survey team (especially if they're working remotely) is communication. Team meetings for remote teams can be incredibly helpful for ironing out problems early in a project, but can also serve to reinforce a personal sense of ownership of project outcomes.

5. Monitoring your project's progress

Visualizing your data through dashboards is a great way to exercise quality control. To help get you started, we developed an example Google Sheets CATI dashboard for you, to illustrate one approach to maintaining oversight of survey progress and adherence to protocol. Google Sheets is one of many tools you can use to make a dashboard, and it is a good choice, given SurveyCTO's support for publishing to Google Sheets. 

For more resources on dashboards, check out our blog posts on Google Sheet dashboards here and here, also our guide to integration with Microsoft Power BI.

6. Summary

All projects are a little different, so we suggest that you design a monitoring and quality-checking process using the above features that's a good fit for your project. "Good fit" means that it focuses on what's most important, and that it is manageable given your resources. Some general suggestions toward an overall strategy are:

• Include the metadata that makes sense for your project.
• Make sure it is quick and simple to confirm whether a call happened, and how long it was (the most basic indicators of what happened).
• Consider flagging submissions with basic problems automatically using automated quality checks.
• Try to make your data visible and easy to digest for the people who will be monitoring it.
• Be strategic about recording calls and auditing those recordings. Try to record and listen to only what you need.
• Communicate with your team to provide them with feedback!

Do you have thoughts on this support article? We'd love to hear them! Feel free to fill out this feedback form.