In the feature video Dave Pier, Product Manager at Skyscanner shares their real-world experiences of mastering data insights to achieve their phenomenal success.
(You can find the slides here.)
He brings out the practical difficulties in handling the data, the scenarios he came across, his experience and the huge role the data plays in the field of product development.
The objective is to share learning experiences from the mistakes they made so that others can avoid doing so.
At 0:35 seconds he shows the look of the desktop application of the Skyscanner 13 years ago. He further emphasizes the MVP version of the Skyscanner application and the looks of the app when Scouts and three other guys developed it. The app was developed initially developed in order to find the cheap flights to go on a ski holiday.
Now the application has developed drastically in terms of the number of users registered as well as the functionalities provided by the application. Dave Pier also brings in statistical data at 1:03 stating that around 800,000 employees work for the Skyscanner at 10 different offices. He further continues that all these statistical data are more like vanity matrices.
He also expresses that the above statistical information is not that informative or interesting. Data that helps in interpreting results, studying the workflow, the trends as well as people’s opinions.
Through the concept of highest paid person’s opinion at 2:21, David Pier explains how the initial look of the Skyscanner desktop application had scrolling functionality on either side. Though it was liked by few in the company, many did not. Since the results and number of people using the application data was less, this functionality was removed.
He also states that every time the experiment was run on the application, the users using Chrome were found to be significant. He says that initially his company was looking at the data in a wrong manner and that’s the reason for it.
At 5:12 he talks more about the false negatives and the way the bad decisions can be made.
At 7:59 he brings out the idea is that the data is generally totally systematically biased. He stresses that people shouldn’t go after data as it is not a long-term strategy and that the design should always be made from the user experience you desire. The biggest learning point of the talk was that the quantitative data is never a substitute for the qualitative insights.
At 12:17 he says that never to trust key performance index, instead trust on the improvement process of the data and the application. He also says that that the data must not be trusted blindly without thinking of its causes as well as its long-term progress.
At 14:53 he further emphasizes on the list of mistakes he did that included considering the quantitative data more, being reluctant to work to change the behavior of the customer or user thinking.
At 16:34, a person from audience questioned Dave Pier on how important measurement or gauging data is, how tedious and accurate is the process when compared to the other aspects of the business growth. To this, he answered that wrong data is worse than less or nil data. As wrong data can lead to false interpretations and create problems, it is better to go for the maximum possible perfect data and then do the approximations. At 24:40 he explains about the role of false positives that occur while performing the experiments.
He explains that there a 5 % chance that people get the wrong right and that people will never be knowing. He says that Google’s experiment has got like 40 times 40 comparisons and that there are a lot of 5%errors which when added up provided a really huge probability of nearly 88 % chance for the false positive to occur.
At 28:56 he comes up with a comparison that once a person climbs a hill, they will be able to predict the size of the hill and how to reach to that hill. Similarly, people must understand the data before using the metrics to measure and conclude.
He further adds on that many experiments keep happening at the Sky scanner with a varied list of performance metrics against which the interpretations are collected. He states examples like using different filters like the broken flights and find the growth activation retention.
At 36:43 a person questions to Dave Pier on he communicates to his management board his works in the form of road mapping. To this, he replies that a completely in-depth strategy is prepared in explaining the process of experiments, its interpretation, and the results.
He adds on that it is very important to understand the purpose of the product entirely, understand the functionality and bring in lots of qualitative research. Understanding the values of the various performance metrics also play a huge role in determining the trends and the results of the experiments. This would also help understand why a value went down in terms of quality.
At 40:10, he once again emphasizes that people should never be blind to number as it ruins the progress. Trusting numbers blindly would lead to a lot of confusions and false interpretations and will eradicate the scope of the real improvement and the progress of the product.
He says that at Sky Scanner, the experiments are conducted regularly, measured spontaneously without just trusting the data lone blindly. He states that at Sky scanner they used all the squads in performing the tasks perfectly as they knew their job duties clearly without any chaos.
At 45:20 he tells that experiments are conducted at the weekends as well as the weekdays. The ones conducted during the weekends generally provide a slightly different range of results owing to the traffic and that the experiments can be conducted successfully in nearly 10 minutes.
He also adds on that the company aims at the global change to the product. The company greatly aims at expanding to many more companies globally and further work more on the translation mechanism as right now it isn’t a great one.
At 50:24 he concludes that he always trusts in his abilities, knowledge of data and that the thing he did is completely right and never trust the data blindly. He adds on that people should remember that perfect data is never possible and that main decision shouldn’t be made solely based on the data.