Stata command to download data using the API

Our new Stata command (sctoapi) allows you to download your form data and media files at any time, without needing to use the server console Export tab or SurveyCTO Desktop. Combined with our .do file templates, this Stata resource offers you a great tool to automate monitoring and analysis processes of your data collection projects.

This SurveyCTO resource for Stata users is significantly based on the work of a pre-existing resource for Stata users created by Innovations for Poverty Action (IPA). Both resources achieve the same aims, but the SurveyCTO version has been tailored for more general use, compared to the specific requirements of IPA projects.

Using this Stata command, you will be able to:

• Fetch data from one or more forms, submitted after a specific date in WIDE format. This includes encrypted forms.
• Save data in a specified folder in JSON, CSV, and DTA (optional) formats.
• Optionally, download media files and save them in a subfolder named ‘media’.

Tip: If you use our .do file templates, you'll probably just want the CSV file generated by this command. You might also want to edit our .do file template to include an API download step.

Getting started

The sctoapi command is included in the scto Stata package, where you can find other Stata commands to make the most of data collected using SurveyCTO. All information and code is available on https://github.com/surveycto/stata-scto.

If you’ve NEVER installed our scto package...

To install, type ssc install scto in the Stata command window.

If you’ve ALREADY installed our scto package...

To update, type ado update scto, update in the Stata command window.

To read associated documentation, type help sctoapi into the Stata command window.

Syntax

To run the command, follow the syntax below and adjust it according to your dataset:

sctoapi formids [if] [in], server(server name) username(username) password(password) date(unixtimestamp) outputfolder(folder_path) [key(file_path)] [media(varlist)] [dtafile]

For more information on the parameters, particularly the date parameter, please read our comprehensive REST API documentation.

Examples

sctoapi testform, server(scto) username(support@surveycto.com) password(1234) date(1546344000) output(`destination') media(textaudit) DTAFILE

In this example, the sctoapi command is being run to download the data from the form with ID 'testform', deployed on the server 'scto'. We're using the credentials of the user 'support@surveycto.com' with password '1234', which must have the permission to 'Download Form data'. This code will trigger the following actions:

• Fetch data from the form testform submitted after Jan 1, 2019 in WIDE format
• Save the data in `destination' in JSON, CSV and DTA formats
• Fetch media files collected in the field named 'textaudit' and save them in `destination'/media

 

If you're running into errors with the sctoapi command, follow these troubleshooting steps.

How to get the most out of sctoapi command?

The great thing about our Stata supporting materials is that they complement each other. As mentioned above, this Stata command can be combined with other supporting materials to streamline your monitoring and analysis processes. In a single script, you’re able to:

1. Generate and save a .dta file that is formatted and labeled. While the sctoapi command can only save a CSV or DTA file with raw data, you can easily label this data by following these steps:
• In the Design tab, go to the form you’re downloading data from, and click on ‘Download’ -> ‘Stata .do template’.
• Choose WIDE for the format of your data exports.
• In the CSV directory, insert the same directory used in the outputfolder() parameter.
• In the Stata directory, insert the directory where you want to save your final .dta file.
• Download your .do file and open it in the Do-file Editor in Stata.
• In your .do file, before the initialize Stata section (row 10), insert the correct sctoapi command for your use case. Additionally, when creating the local macro for the csvfile (row 24), change the file name to match your form ID, not the form Title. Note that there is no need to specify dtafile, since the .do file will import the CSV instead.
• By running the whole .do file, you’ll be able to download, format, and label your data all at once.
2. Rename and relocate your media files according to other field’s values. While the sctoapi command can only download and save your media files in outputfolder/media, use our sctomedia command afterwards to reorganize these files if needed. If you’re implementing the steps above, insert your sctomedia command at the end of the .do file to add this functionality.

We hope that you find this command useful! Please consider sharing your experiences in our user forum.

Do you have thoughts on this support article? We'd love to hear them! Feel free to fill out this feedback form.