Finding Missing Data: Missing values/variables

This article is part of a sequence of articles on troubleshooting missing data. If you arrived here directly, please start from the first article here.

Form design updates

Briefly, a change to the name of a field or group effectively creates a new field where new data will be stored. The data previously collected before this form revision is still there but only fields defined in the current form version are exported. You can confirm evidence of this by comparing sections of missing data with the formdef_version metadata variable that is exported in your data (formdef_version will tell you which version of your form each submission was made with).

Values and fields which go missing in this way can be restored by re-adding the original field to the form design. Please read the help topic on updating an existing form for a full explanation.

When you update a form on your server, it stores a copy of all your previous form versions. To download a copy of an older form version, scroll to the title of a form on your Design tab, click on the Download icon to the right side of the form title, and then click Form files from the expanded list of icons that opens up.

You may wish to preserve changes you have made which don’t affect the export of data, so compare your current form version and the version that preceded the version where values or fields stopped exporting. Recreate the missing fields with the same name, nested in the same group/s of the same name/s. If you don’t want fields added back to appear in the form, enter 0 in relevance. Update your form with your new version with restored fields and export your data again, and see if you were able to restore your missing values or fields.

Missing because of encryption

It is also worth checking whether the form you are trying to download data from is encrypted. Only those fields that are specified as publishable in an encrypted form will appear in an export from the Export tab of your server console (all other fields will be missing). SurveyCTO Desktop will not export data from an encrypted form without a private decryption key. Read the help topic on encryption for details.

Common form design problems

Sometimes form programming errors can lead to missing values. Here are some mistakes that can lead to missing values:

(A) In order to guarantee that fields are not left blank by mistake, fields should be made required. You cannot expect that any field exports values if the field is not required as surveyors may have simply skipped those fields and not filled them in.

(B) Perhaps a calculate expression produces values some of the time but not at other times - which could explain cases of missing values in calculate fields. Did you thoroughly test your form definition to be sure the calculate field works as intended, in all scenarios?

    1. Calculate fields should not have a relevance expression.
    2. You will also need to plan for when a field that is used in a calculate is blank, as in this tips and tricks help topic.

(C) When a relevance condition for a field re-evaluates, so that the field becomes not relevant after it has been completed, such a field can appear blank in exported data (even though it appeared as relevant while completing the form). For this reason, relevance conditions need to be stable. A stable relevance condition is one that depends only on earlier questions the user has already answered and does not depend on any questions that come after the current question (as a relevance condition that depends on future questions could change as the user gets to those later questions and make the earlier question irrelevant, and hence erase the value stored in it).

Getting help

If you have eliminated the above as causes of missing values or fields, open a support ticket to get direct support from our professional team if you are a paying subscriber by clicking the My requests link in the top right. If you are a paying user but still not able to submit a request, please check with your account's administrator to see if you are a supported user or whether they will need to submit the request for you.

If you are a free community user, you can post a request for other users to help you in the community forum.

Do you have thoughts on this support article? We'd love to hear them! Feel free to fill out this feedback form.


Article is closed for comments.