What Questions should we ask when Testing a Mobile Application?
In developing an application that connects providers of content to a captive audience, we have a dual aim. We need to wrap the content in an easy and attractive to use application and we need to keep the application fresh in order to keep the subscribers (the content providers) onside rather than review this need or be enticed to someone else’s offering.
By surveying the users of the application and the subscribers to our service we can get useful information about what is good and bad about the evolution of the application but also we can control the interaction.
Surveying the users is another service to the subscriber.
We have produced and tested a mininum viable product. The testing of that has led to the development of the first version of our application. At the moment, we are in the mode of developing and testing instead of testing and developing. We must test the first version of the product but, also, we need to switch to testing ideas before we develop them.
Developing ideas and then testing them is expensive. The first version of the application has been developed for Android. Before we invest in the iOS (Apple) version we should validate the Android version.
We should, therefore, develop a survey that validates the first version of the application and test ideas for the evolution of the application. As part of this twofold test, we should keep in mind the original survey and that the new survey should not take too long for users to engage with.
The Original Survey
The survey was built up from WordPress, the plugin Contact Form 7, the plugin Contact Form 7 Conditional Fields and CF7 Google Sheet Connector. When the survey form was submitted the results were written to a Google spreadsheet. The questions were similar to the following:
“a user” | “a staff member”
“female” | “male” | “other” | “rather not say”]
If “other”, please say (gender):
“Under 12” | “12-17” | “18-24” | “25-34” | “35-44” | “45-54” | “55-64” | “65-74” | “75 years or over” | “rather not say”
Did you find what you were looking for?
“yes” | “no” | “just browsing”
How easy was it to find what you were looking for?
“easy” | “difficult”
If “difficult”, would you explain, please?
Is this because of the app or because of the content?
“app” | “missing content”
If “app”, would you explain, please?
If “missing content”, what content were you looking for?
Would you prefer the information in an alternative format?
“no” | “video” | “paper” | “other”
If “other”, what
Would you prefer this information in another language?
“no” | “yes”
If “yes”, what
Do you feel this format of information for visitors will save you time?
“yes” | “no”
If “yes”, how much time?
“under 5 minutes” | “6-10 minutes” | “more than 10 minutes”
Is there anything else you would like to see included in this app?
Features in the First Version of the Application
The MVP tested the idea of replacing a paper guide with an electronic version. The response to using the MVP ranged from inevitable to very useful. The first version of the product extends the electronic guide with a social network function and a diary. The design of the Android application follows Google’s Material design guidelines. Google has researched and developed libraries that support the guidelines. We don’t need to second guess this work. In terms of the user interface, the application uses four major areas of the libraries:
These are UI features familiar to users that use applications e.g. Twitter. The navigation drawer is used to change settings in the application’s environment i.e. such as location. The Bottom Navigation View
While these are standard features of Material design we should validate if we are using them in a way the makes sense to the user. Questions might be:
- did you find the About section of the application?
- were you able to set your location?
- did you find the opportunity to give feedback about the application?
- were you able to switch between the guide and the diary easily?
- were you able to find information easily in the guide?
- were you able to change the size of the text in the guide’s sections?
- were you able to add an event to the diary?
- were you able to share an event with friends?
- were you able to delete an event in the diary?
- did the application download quickly?
- was the application quick to install?
- were you able to move through parts of the application quickly?
- was it quick to share an event from the diary?
- do like the experience?
- do you like to be able to share the diary events with others?
- would you tell others that the application exists?
Rather than just looking for validation we should also look for a better suggestion. Maybe there are other channels for disseminating the content that
- do you prefer the paper guide?
- would you rather see the content in another form, e.g. video, tv, or radio
- would you prefer another way another way of sharing your experience with friends and family? If so, what?
Testing Possible Future F
This is where the opportunity exists to switch around development and test to test and development. In the previous section, we have described the opportunity to validate the app. If the feedback is very negative then we have to look at ending the development or pivoting to something else. If the application gets positive feedback then we might be able to sell subscriptions. However, over a longer period of time, we have to look at ways to keep the application fresh or, like Twitter and Facebook seems to have done, add a feature users cannot live without. We don’t do this because we have a captive audience, we do it because we need to continue the conversation with the subscribers. Individuals in the changing captive audience may not need our application very often. If a large proportion of users do need the application on a regular basis then changes should not be jarring. We have discussed possible changes to the application. These could lead at least to the questions:
- would you like to see the content in the guide in another language?
- do you think that you would use the application more if you were given awards for completing simple tasks?
- would you like to give direct feedback to the organisation through this application? e.g. through surveys
- do you think that the application would be better if it could be used to get cheaper food and drink through a voucher scheme?
We will develop a survey that has four sections:
- establishing the demographic mix of our users
- validating our application
- anti-validating our application
- the application’s evolution.
At every iteration, we will validate, anti-validate and look at the evolution of this application.