If you were to collect data using a form, and then delete or rename a field in the form definition, then that field's data will no longer appear in the main data exports: it will be considered missing data. If you still need that missing data, you can use SurveyCTO Desktop to export it, even if their fields are no longer in the deployed version of the form.
|
SurveyCTO Desktop 2.71.3 or later is required. To check the version of SurveyCTO Desktop you are using, in the top-left, go to SurveyCTO Desktop > About SurveyCTO Desktop. You can install the latest version here. If you have never used SurveyCTO Desktop before, check out our quick start guide, section 3.2 Downloading data using SurveyCTO Desktop. |
Exporting missing fields
By default, fresh installations of SurveyCTO Desktop will automatically export all missing data. If you previously installed SurveyCTO Desktop before November 12, 2021, you will need to turn on the Export missing data? setting, which you can learn how to do in Missing data not exporting below. Once the needed setting is turned on, missing data will automatically export with the main data.
After the data export is complete, open the data export file: there will be additional columns to the right of the main data (to the right of the KEY column) with the missing data.
For example, let's say there is a form where an older version had fields called "field1" and "field2", but the currently deployed form definition does not have those fields. Take a look at this data export:
The first two form instances were completed using the older version of the form, which still had the fields "field1" and "field2", while the next two form instances were completed using the newer version of the form, which does not have those fields. Since those two fields are now considered "missing data" fields, they appear to the right of the main data and the KEY column, instead of to the left of the KEY column with the main data.
| SPSS users: If you export your data in SAV format, then the missing data will always be considered text-type data, regardless of the original data type. |
To learn how to update your form so there is never missing data, check out our support article on how to safely update a form mid-survey.
For more tips on finding missing data, check out our support article on finding missing data.
Sync logs and messages
After running the Desktop Sync, if your form has missing fields, then it will say either COMPLETED WITH WARNINGS, or COMPLETED WITH ERRORS.
Both of these messages could mean a number of things, so it is a good idea to click VIEW DETAILS, and make sure the missing data is the only issue. Here is an example:
12:00:00: Warning: missing data from [x] submission(s) was found. The following field(s) no longer exist in the current form definition, but have been appended to the exported data anyway:
field1
field2
group1[1]/field3
group1[1]/field4
Learn more about missing data by visiting: https://support.surveycto.com/hc/en-us/articles/4406695685395
If that is the only Warning, and it says, "but have been appended to the exported data anyway", then you are all set, since the missing data has been successfully exported, and it will appear to the right of the main data.
However, if the data export completes, and it says COMPLETED WITH ERRORS, check the logs for a warning like this:
12:00:00: Warning: data is missing from [X] submission(s) because one or more fields no longer exist in the current form definition.
Try enabling the “Export missing data?” feature in your export options.
You can use the "Ignore groups..." setting under "Treatment of enclosing groups in exports" to automatically merge fields with the same name.
Learn more about missing data by visiting: https://support.surveycto.com/hc/en-us/articles/4406695685395
If it does not say, "but have been appended to the exported data anyway", and it instead says, "Try enabling the “Export missing data?” feature in your export options", then the Export missing data? setting is not turned on. If you need the missing data, turn ON that setting from the EXPORT OPTIONS, and run the data export again.
| The part about "Ignore groups..." will only appear if that setting is not already turned on, which we discuss more in Adjusting group settings below. |
Missing data not exporting
If you don't see the missing data to the right of the main data, and you're using SurveyCTO Desktop 2.71.1+, then you will need to turn on the needed settings.
In SurveyCTO Desktop, in the lower-left, click Workspace settings, then EXPORT OPTIONS.
In that list, there is an option to Export missing data (you can use Ctrl + F on Windows or Cmd + F on Mac to find the setting).
Once that option is selected, run the data export again, and it should export the missing data.
| Keep in mind that enabling this export option might slow down your exports considerably, based on the amount of missing data available for export. |
Adjusting group settings
This section is here in order to give comprehensive details on how exporting missing data works, but most users can skip this section without issue.
In the EXPORT OPTIONS, there is a setting that lets you adjust the Treatment of enclosing groups in exports, which you can learn more about in our documentation on data export options.
If you set it to Ignore groups, then, as the name implies, the group path of the fields will be ignored, and only the name of the field will be considered.
But, if it is set to Add groups or Exclude groups but still use them internally, then the export will behave a bit differently. If a field used to be in a group, and you have since deleted or renamed its group while keeping the field, then the field will be considered missing data, since its group path has been changed. A field will also be considered missing data if it used to have no group, but it has since been put into a group. Since that field data is considered missing data, then in the data export, that missing data will appear to the right of the main data.
For example, let's say you have a group called "group1", and in that group, there is a field called "field1". An enumerator collects data using that form, but then you then change the name of the group to "my_gr". If the group setting is set to Add groups or Exclude groups, then the data for the field "field1" would be considered missing data.
Let's take a look at a more practical example. Let's say you start by creating this form (which has been simplified for demonstration purposes), where none of the fields are in a group. Enumerators then collect data using that form. Then, you decide to put the field "resp_name" into a group called "gave_consent", as demonstrated in this form. Take a look at the exported data in this spreadsheet. The first two submissions were completed using the first form, and the second two submissions were completed using the second form, which is the currently deployed form. You can switch between the different sheets at the bottom for examples of each Treatment of enclosing groups in exports setting. On the Add groups and Exclude groups sheets, the "resp_name" data for the first two submissions is in a "missing data'' column, since that data was collected when the field had no group. Meanwhile, on the Ignore groups sheet, since the groups of the fields do not matter, all of the data for that field is in the same "resp_name" column.
Do you have thoughts on this support article? We'd love to hear them! Feel free to fill out this feedback form.
0 Comments