Monitoring & evaluation of program beneficiary confidence

This article is accompanied by sample forms, dataset definitions and sample data saved in this folder.

1. Introduction

In the context of monitoring and evaluating an education and training program, data collection plays a crucial role in tracking the progress and outcomes of beneficiaries. This article presents an overview of a sample workflow designed to measure changes in participants' confidence levels in various business skills at baseline, midline, and endline observation points. By implementing this monitoring and evaluation (M&E) use case, we gain valuable insights into the program's effectiveness and its impact on participants' skill development. The same tracking mechanism could be used to track other key program indicators.

This use case covers the following sections:

2. Deploying the workflow
3. Overview
4. Workflow components
5. Customization and improvements

In this workflow, you will be able to:

2. Deploy the workflow now!

Click below to view this workflow in the Hub and install it on your server.

Install Workflow

3. Overview

In the sample workflow, beneficiaries complete the same form three times to generate baseline, midline, and endline observations. Each time, confidence ratings on different skill areas are published to a server dataset in the workflow, and the form is pre-populated with data from the last round using pre-loading. This consolidated data in the server dataset can be used later during the evaluation phase, where the data is consolidated and ready for pre- and post-test analysis, longitudinal analysis, and more.

The person who fills out the form is called the "beneficiary", since they are the one benefitting from the training. They can also be called "participants" or "trainees".

Here is what you can expect from this workflow:

	Track the beneficiary using unique web form links.
	Track the beneficiary's changes over the course of the study.
	Prevent the beneficiary from submitting the next form too soon, to make sure data is gathered at regular, spaced intervals.
	Prevent the beneficiary from submitting the form again after they have already completed it three times.
	Collate multiple form submissions by the beneficiary so that it is easy to track and compare self-confidence ratings.

Note that SurveyCTO example workflows are highly customizable according to your project’s requirements. This workflow can be adapted to serve more specific needs.

To try the workflow, deploy it to your server, and go to this URL:

https://SERVERNAME.surveycto.com/collect/sample_me?caseid=4

Be sure to replace "SERVERNAME" with the name of your server. That URL is for a case where the form has not yet been completed. Take a look at the server dataset data before filling out the form (to view dataset data, on the Design tab of your server console, go to the server dataset, and click Edit). Then, try filling out and submitting the form three times (wait a few minutes after each submission), and then take another look at the server dataset, and see how it has been updated.

Important: The workflow has been set up so the beneficiary must wait 48 hours between submissions. If you would like to fill out the form three times in a row to test it, go to the field "hours_to_wait", and change its calculation from 48 to 0.

3.1 Understanding the workflow

To learn about server datasets, check out our support articles How to create a server dataset and How to set up dataset publishing.

The sample workflow has two parts: a form, and a server dataset. The form pulls from and publishes to the server dataset. The form will ask the beneficiary's confidence in five skills. The form will be completed three times, each time sending the data to the server dataset, meaning there will be 15 columns for skill ratings, since there are five skills that will each be rated three times.

With each beneficiary, follow this workflow:

1. Send a unique form link to each beneficiary (such as by using mail merge), asking them to complete it before they begin the training program. To learn more, check out our Guide to unique links for web forms. Beneficiaries use the link they were sent to fill out and submit the form before the training program starts.
2. Beneficiaries start the training program (outside of SurveyCTO).
3. Halfway through the training program, beneficiaries complete the form for the second time. They should use the same link they used to access the form the first time. You can also re-send the link to each beneficiary, but make sure it is the same URL, so it is always published to the same row of the server dataset.
4. Beneficiaries complete the training program (outside of SurveyCTO).
5. After the training program is complete, the beneficiaries complete the form for the third and final time. They should use the same link they used to access the form the first and second times.

Here are other traits of the workflow:

• Unique links only: The form can only be opened from a unique link. Meaning, if the "caseid" field has no value, then the form cannot be completed.
• Wait 48 hours (number of hours can be changed): After the first time the form is completed, the beneficiary cannot complete the form again until at least 48 hours have passed since the last time it was completed. This is so the beneficiary cannot just complete it three times in a row, instead of completing it after each part of the training program like they are supposed to.
  You can change the number of hours the beneficiaries have to wait by changing the calculation of the field "hours_to_wait". It currently has a value of 48, but you can make this 1 if the beneficiaries should wait one hour before completing the next form instance, 0 if they do not have to wait at all, and so on.
• Limit number of completions: The form cannot be completed more than three times.
• Before and after ratings: The second and third time the form is completed, the end of the form will display confidence ratings in the skills, comparing the last time the form was completed to the current time the form is being completed, so beneficiaries can see if their skill-confidence has improved or not.
• Ask for experience rating and comments: The third time the form is completed, it will ask the beneficiary to rate and give comments about their experience, with that information published to the server dataset.

You can send links in bulk, and also brand them with your own URL. To learn more, check out our support articles on Mail merge and Branding and redirecting web form links, respectively.

3.2 Detailed explanation

We will go into detail on how the workflow works, but we recommend that you deploy it to your own server, and see for yourself how it works in practice.

3.2.1 Form definition

For details on how each calculate field in the form definition works and what it does, check out each calculate field's label. Since calculate fields are hidden fields, these labels will not be shown to the beneficiaries, so they can be used to explain each field.

3.2.2 Server dataset

The server dataset tracks various pieces of data. The first three columns are fairly straightforward:

• id_key: Unique identifier of the beneficiary. This is used by the form to identify the beneficiary's server dataset row, so information about the beneficiary is pulled from and published to the correct row of the server dataset. When generating the form URLs you send to beneficiaries, use these values as the part that comes after the caseid=. For example, if the beneficiary's id_key value is 4, then the URL they will use to fill out the form will be:

  https://SERVERNAME.surveycto.com/collect/sample_me?caseid=4

• name: Name of the beneficiary (not pre-loaded by the form).
• phone: Phone number of beneficiary (not pre-loaded by the form).

Next, there are two columns that will be changed for each submission:

• received: How many form instances the beneficiary has completed so far. When there have been no submissions so far, this will either be blank, or have a value of 0. When the beneficiary completes and submits the form before the training program starts, this will have a value of 1. When it is submitted for the second time halfway through the program, this will have a value of 2. When it is submitted for a third and final time at the program’s end, it will have a value of 3. The sample form has a field called "training_complete" that warns the beneficiary if the form has already been submitted three times. This is retrieved by the form field "received", and updated in the form field "received_new", which is published to this column of the server dataset.
• last_sub_date: This tracks the last time the form was submitted. This is used by the fields "last_sub_date_disp", "time_to_next", and "too_early" to ensure the beneficiary does not complete the form more than once in a 48 hour period. The form field "endtime" is published to this column.

Then, there are fifteen columns that track the beneficiary's confidence ratings in the skills before, during, and after the training program.

The final two columns will be blank until the beneficiary completes the form for the third time:

• course_rating: Stores the beneficiary's rating of the course
• course_comments: Stores the beneficiary's comments on the course.

The form fields of the same name publish to those columns (e.g. the form field "course_rating" publishes to the dataset column 'course_rating'), but those form fields are only relevant the third time the form is completed. Until then, the columns will remain blank, so they do not publish to the server dataset until ready.

3.3 Next steps

Once you understand the workflow, you can customize it to fit your own needs to ask about business skills, the form can ask about other traits, such as household income, food insecurity, and so much more.

Data can be analyzed by downloading the dataset data.

You can also publish the cases dataset to Google Sheets for dashboards, or use the REST API to download the dataset data and integrate it with your downstream tracking system.

4. Workflow components

This workflow has one form and one server dataset. The form both retrieves data from and publishes data to the server dataset.

Workflow Component Files Click on the section to access the files.

Forms Name Monitoring & evaluation use case File(s) Definition: Form definition - Monitoring & evaluation use case Description Beneficiaries fill out this form to self-assess their confidence in various business skills. Cases Datasets Name M&E cases Files(s) Definition: Dataset definition - M&E cases.xml Data: Dataset data - M&E cases Description Stores M&E data. Data managers will export and use this data for the "evaluation" process of the monitoring and evaluation process.

Note: The dataset data we provide is optional. Instead of using the data we provide, you can just open a unique URL to the form with any "caseid" value, and a new dataset row will be created when the form instance is submitted. To simulate the same beneficiary filling out the form again, make sure you use the same URL with the same "caseid" value.

5. Customization and improvements

SurveyCTO has endless possibilities, so this workflow is only the beginning. Feel free to modify and expand on this workflow to fit your own needs. Here are some tips on what you can add:

• Add a text field after each evaluation field where beneficiaries can describe why they chose that rating.
• Add fields that ask why the rating of a skill has gone up or down, with a dynamic field label that asks why the beneficiary's confidence in that skill "increased", "decreased", or "stayed the same", depending on how the beneficiary responded in the current instance of the form compared to the previous instance. Use calculate fields to generate the field labels, and use field references to display them in the field labels.
• Add other fields that ask basic questions such as the beneficiary's name, address, etc, and have them only be relevant the first time the form is filled out. Publish those fields to the server dataset.
• Related to the last improvement, pre-load information about the beneficiary, and confirm in a select_one field that the correct beneficiary opened the link to that form.
• Add timed fields with field plug-ins so beneficiaries must answer with their unconscious belief, instead of their self-rating if they have time to think about it.
• Publish in long format instead of wide format, so there is a row of data for each time the form has been filled out, instead of all data being published to the same row. This can also make the form more flexible, since instead of needing to add a calculate field for each evaluation field (the select_one fields in the form) for each time you would like the data to be published (see rows 47-57 and rows 62-66 of the sample form), you can just publish the evaluation fields directly to the server dataset, no matter how many times the form is filled out.

Do you have thoughts on this guide? We'd love to hear them! Feel free to fill out this feedback form.