What Questions should we ask when Testing a Mobile Application?

In developing an application that connects providers of content to a captive audience, we have a dual aim. We need to wrap the content in an easy and attractive to use application and we need to keep the application fresh in order to keep the subscribers (the content providers) onside rather than review this need or be enticed to someone else’s offering.

By surveying the users of the application and the subscribers to our service we can get useful information about what is good and bad about the evolution of the application but also we can control the interaction.

Surveying the users is another service to the subscriber.

We have produced and tested a mininum viable product. The testing of that has led to the development of the first version of our application. At the moment, we are in the mode of developing and testing instead of testing and developing. We must test the first version of the product but, also, we need to switch to testing ideas before we develop them.

Developing ideas and then testing them is expensive. The first version of the application has been developed for Android. Before we invest in the iOS (Apple) version we should validate the Android version.

We should, therefore, develop a survey that validates the first version of the application and test ideas for the evolution of the application. As part of this twofold test, we should keep in mind the original survey and that the new survey should not take too long for users to engage with.

The Original Survey

The survey was built up from WordPress, the plugin Contact Form 7, the plugin Contact Form 7 Conditional Fields and CF7 Google Sheet Connector. When the survey form was submitted the results were written to a Google spreadsheet. The questions were similar to the following:

Are you?
“a user” | “a staff member”
Your gender?
“female” | “male” | “other” | “rather not say”]
If “other”, please say (gender):
text field
Your age?
“Under 12” | “12-17” | “18-24” | “25-34” | “35-44” | “45-54” | “55-64” | “65-74” | “75 years or over” | “rather not say”
Did you find what you were looking for?
“yes” | “no” | “just browsing”
How easy was it to find what you were looking for?
“easy” | “difficult”
If “difficult”, would you explain, please?
text area
Is this because of the app or because of the content?
“app” | “missing content”
If “app”, would you explain, please?
text area
If “missing content”, what content were you looking for?
text area
Would you prefer the information in an alternative format?
“no” | “video” | “paper” | “other”
If “other”, what other format?
text area
Would you prefer this information in another language?
“no” | “yes”
If “yes”, what other language?
text area
Do you feel this format of information for visitors will save you time?
“yes” | “no”
If “yes”, how much time?
“under 5 minutes” | “6-10 minutes” | “more than 10 minutes”
Is there anything else you would like to see included in this app?
text area

Features in the First Version of the Application

The MVP tested the idea of replacing a paper guide with an electronic version. The response to using the MVP ranged from inevitable to very useful. The first version of the product extends the electronic guide with a social network function and a diary. The design of the Android application follows Google’s Material design guidelines. Google has researched and developed libraries that support the guidelines. We don’t need to second guess this work. In terms of the user interface, the application uses four major areas of the libraries:

These are UI features familiar to users that use applications e.g. Twitter. The navigation drawer is used to change settings in the application’s environment i.e. such as location. The Bottom Navigation View is use to show and change the context of the application i.e. guide or diary. The Floating Button is used create an event related to the context that the application is in. The RecyclerView swipe allows users to delete or share events from the diary.

While these are standard features of Material design we should validate if we are using them in a way the makes sense to the user. Questions might be:

  • did you find the About section of the application?
  • were you able to set your location?
  • did you find the opportunity to give feedback about the application?
  • were you able to switch between the guide and the diary easily?
  • were you able to find information easily in the guide?
  • were you able to change the size of the text in the guide’s sections?
  • were you able to add an event to the diary?
  • were you able to share an event with friends?
  • were you able to delete an event in the diary?
  • did the application download quickly?
  • was the application quick to install?
  • were you able to move through parts of the application quickly?
  • was it quick to share an event from the diary?
  • do like the experience?
  • do you like to be able to share the diary events with others?
  • would you tell others that the application exists?

Anti-validation

Rather than just looking for validation we should also look for a better suggestion. Maybe there are other channels for disseminating the content that out weigh the features of our application: content and socialising. Questions could be:

  • do you prefer the paper guide?
  • would you rather see the content in another form, e.g. video, tv, or radio
  • would you prefer another way another way of sharing your experience with friends and family? If so, what?

Testing Possible Future Features

This is where the opportunity exists to switch around development and test to test and development. In the previous section, we have described the opportunity to validate the app. If the feedback is very negative then we have to look at ending the development or pivoting to something else. If the application gets positive feedback then we might be able to sell subscriptions. However, over a longer period of time, we have to look at ways to keep the application fresh or, like Twitter and Facebook seems to have done, add a feature users cannot live without. We don’t do this because we have a captive audience, we do it because we need to continue the conversation with the subscribers. Individuals in the changing captive audience may not need our application very often. If a large proportion of users do need the application on a regular basis then changes should not be jarring. We have discussed possible changes to the application. These could lead at least to the questions:

  • would you like to see the content in the guide in another language?
  • do you think that you would use the application more if you were given awards for completing simple tasks?
  • would you like to give direct feedback to the organisation through this application? e.g. through surveys
  • do you think that the application would be better if it could be used to get cheaper food and drink through a voucher scheme?

Conclusion

We will develop a survey that has four sections:

  • establishing the demographic mix of our users
  • validating our application
  • anti-validating our application
  • the application’s evolution.

At every iteration, we will validate, anti-validate and look at the evolution of this application.

Leave a comment

Your e-mail address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.