They created an app, Testeum tested it.
In 2023, we decided to offer a test campaign every month to an innovative project in New Caledonia, the country where our platform was first developed, and to which we are eager to give back.
In this article, we will share the testimony of the team behind the first project to use Testeum in this context, namely the “Domaine NC” mobile application.
Who are you ?
I am Adrien Sales, I’m the product manager of “Domaine NC”, and I am Laurent Schaeffer, the developer of this mobile app.
What is your app for ?
LS : Simply put, this mobile application is used to view the domain names in New Caledonia in a mobile-friendly and compact manner. It allows users to view information about a domain name and receive notifications when it’s about to expire.
AS : We have created a disruptive user experience to facilitate a management process that was previously done manually… with the risk of forgetting to renew the purchase of the domain name. This can have a significant impact on our digital business because if your domain name expires, you disappear from the web and lose visibility, or even worse: transactions. We wanted to highlight the value of this data and its’ potential.
What are the quality risks of this application ? What could go wrong ?
AS : One of the risks is having a suboptimal or overly complicated user experience. We were also worried about having poor response times… or even worse : no response (for example, a nasty 500 error) !
What questions did you want the crowdtesting campaign to answer ?
LS : We needed to make sure that each user could easily access the information they were interested in.
AS : Indeed, we wanted to ensure ease of use for a novice user. We also wanted to know if our performance was passable, good, or excellent (we aimed for instant search and display).
How did you organize your test campaigns ?
AS : First, we defined a very limited scope of features based on our MVP through a dedicated milestone on GitHub. Then, we organized a very simple first test campaign to get feedback from testers on this perimeter.
LS : After fixing bugs reported during this first testing session, we launched additional test campaigns: one for each feature.
What did you think of the crowdtester's feedback ?
LS : We discovered a different perspective from ours, which allowed us to improve the application on a UX and functional level.
AS : We discovered very relevant test scenarios. For example, one person searched for the maximum size accepted by our domain name input field. Some tests even allowed us to improve the security of our API! We had new ideas to simplify usage, such as adding placeholders to guide users.
We also received feedback on a pure functional bug : the automatic search was supposed to happen only after 3 characters were entered, but that was not the case. We therefore improved the app’s speed and reduced the number of API calls.
How did you find the platform itself ?
AS : The dashboards are beautiful and very easy to understand! Everything is ready to use. We were able to copy and paste them as is to organize our teamwork with very little effort.
LS : The platform is easy to use and allows for optimal organization of campaigns.
In your opinion, how should a crowdtesting approach be articulated with other testing practices ?
AS : From what I’ve seen, I think crowdtesting allows you, for example, to test release candidates or betas in real-world conditions (on a target infrastructure). For instance, I was able to measure the impact on the performance of the API and see if the entire chain was holding up well: it was really end-to-end! Awesome and tremendously effective.
I would like to see an organizational description that integrates crowdtesting with all other testing phases, and appear in the release planning.
I think that running a crowdtesting campaign with Left Shifting would make sense in a LEAN approach, which prioritizes Time To Market, quality, and more generally the entire delivery chain. We implemented a complete CI chain that allowed us to deploy the version on stores with a simple button press, literally. Thus we can focus on compact perimeters and quickly close loops with high software quality. We did everything via GitHub/Fastlane cloud tools.
What advice would you give to someone considering a crowdtesting approach to test their product ?
AS : I would advise them to :
- Raise awareness with their team : present the key concepts of the platform to the team and sensitize them to Left-Shifting. They can show examples of crowdtesting campaigns : wins, failures, very concrete examples. They can also present approaches (one big test campaign vs. several small ones) and share testimonials.
- Sensitize the team to the fact that testing a feature is part of the development costs… and therefore must be taken into account from the beginning.
- When designing the test campaign, propose a complete journey to make the most of the platform (ROI).
- Have a Product Owner (PO) with a very clear product vision.
- Involve the team (Devs, Scrum Master) so that crowdtesting is a project phase like any other… If possible, include complete newbies to the team who know nothing about the business domain.
- Bonus : create a user club for those who have used the platform and create a guide of best practices together.
Oh, what is shining on the ground ? A lamp ? HA, here is the Testeum genie, you can make 3 wishes for new features on the platform !
LS : Do we have 3 each ? I’ll make 2 for myself:
1) The ability to export test data to other platforms (like GitHub for example) or in JSON format, and
2) Provide an API to retrieve test data and/or other data from the platform.
AS : I’ll make 3 wishes !
1) Being able to trigger webhooks to create issues automatically (GitHub issues for example) or trigger workflows (IFTTT, Zapier, Power Automate, etc.), in an event-driven mode, or as a part of CI chains.
2) Provide an API to consume feedback and campaign data for B2B integration or reporting purposes.
3) Being able to target professional QA profiles (in addition to current criteria : age, gender, country, and type of owned equipment).
