This is a recap of Snowflake Build Summit 2021 session: Bridging the physical to digital using computer vision models on Snowflake. Brought to you by Snowflake & meldCX.

‍

Developers in all regions of the world are building applications on Snowflake to implement new and unique use cases. Watch this session to hear how meldCX built Viana™, a vision analytics product to measure how people work, shop, learn, and play in physical and digital spaces using Snowflake. Discover how Viana™ uses Snowflake to gather real-time data analytics on audience sentiment, physical store engagement, digital signage effectiveness, vehicle and number plate recognition, traffic management, and more. 

Featuring:

• Thor Turrecha - EVP of Global SaaS at meldCX

Transcript:

0:13  
Welcome to Snowflake Build Summit 2021. The theme for my talk today is about bridging the physical to digital. Using computer vision models and snowflake, you probably have seen a lot of webinars in the day. So I'll try and make it visual and entertaining for you.

0:28  
My name is Thor Turrecha, and I'm a Product builder, I have spent the last 15 years building technology software hardware and combine it together to solve epic problems for customers that we can have deemed to be unique.

0:42  
And along the way, I probably have built about 10 different products. And most of them are successful, most of them failed. But today, I want to talk to you about my recent experience when building a product in the midst of a pandemic, and managing to deliver it within 12 months, and being able to win some of the biggest retailers and banks here in Australia, and have allowed us to, at this point, operate in four continents. But before we dive deep into the story, a little bit about us. So I work for a company named meldCX and meldCX stands for building custom experience. So what we do is we specialise in building solutions that connect seemingly unconnected software hardware, different third party solutions and solve big problems. And we take something that is very complex, we simplify it. And we simplify it for customers to use, we simplified for partners to deploy and maintain. And more importantly, we make it secure. Now, there's a lot that we do, but we can sum it up that we create a bridge between the cloud and the edge premise. And you've probably heard about IoT, you've probably heard about AI on the edge AI in the cloud data science. So we stitch that all together and smelled awesome experience, and deliver something holistic, deliver a solution that basically solving a big problem for most of our customers.

2:08  
We are a bit conservative, but we're bold in in selecting the company that we partner, you'll see we've been doing innovation with the likes of Google, Cisco Meraki, Microsoft, Intel. In matters of AI, we were lucky that we managed to attract some of the best engineers there is. And we managed to take a very unique route in r&d that were small. But we have that certain uniqueness in our DNA that allowed us to be able to win the attention of some of the biggest brands there is. So most of the story that I'm sharing today is that within 12 months, we managed to build an enterprise solution that then managed to win one of the biggest banks in Australia, and one of the biggest retailer in Australia. And today, I'm happy to share that we are now operating in four continents, and managed to scale despite the pandemic.

3:03  
So there's two sides of our business. The first one, there's a reseller part where we take a partner tool like snowflake, and we bundle it with one of our solution we called meldCX Core, where it's the main ingredient for us to be able to stitch different software. And that's very, kind of useful for a lot of customers that tend to have a lot of partners. So we stitch all of them together. But today, we're going to talk about the developer side of things where the side of business where we develop products based on our platform, and tools like snowflake. So today, we want to put a spotlight on the product that I mentioned earlier, we call it Viana. Now Viana is short for Visual Analytics. It's quite complex. But to sum it up, it's literally just taking a standard camera and making it as a sensor. Now, its purpose is quite unique. Because as you know, most brands out there, they're already using CCTV cameras for security purposes. And all we're trying to do is to teach those scammers new tricks and be able to track different human behaviours, track maybe you already objects that might resemble a product, a content, a brochure, a furniture, and how humans interact with those products and objects that would be meaningful to whoever is our customer to a bank. They're interested in how the customers interact with their smart ATMs, or their tellers if they're having too much of a queue and they're worried about their service to a retailer, they want to look into how customers engage with their planogram their shelf, the product that are displayed, people engaging and people touching it and interacting with it. And this even unique use cases in manufacturing where you look into, you know, deep product inspection, before you ship your product, you want to make sure that it's the sort of quality that you expected and you don't want the customer to open the box and see a product that's not what they expecting. So there's a lot use cases that Viana can deliver. But for the purpose of this session, we want to talk something that most people can relate to.

5:07  
So before we go into use cases, I just want to, you know, shout out to my team, everything I'm talking here is a result of me leading a very, very capable team and people that are moving kind of the boundaries of what's possible in AI. So, we love to think that what we do every day is we teach machines to see like humans.

5:29  
Now, you know, we're not, we're not the big brands, who's got thousands of engineers. So we take a very specific problem statement, and we try to champion it and make good out of it. So what we do is, we understand human behaviour in scientific terms, it's called group dynamics. It's about how humans interact with their physical spaces. And this is not just simple things like walking or, or or crossing a trip line. This is more about other humans fighting? Is that particular worker making a mistake in packaging a product? Is there a particular customer having a good day inside a branch? Or is someone on aisle number eight in a shopping centre, stealing a product, these are very complex, kind of use cases and even education, we're starting to innovate around how how kids basically interact with the professors, and how they basically consume the lessons that they are basically being presented to.

6:24  
And it's quite complex if you just using cameras, but what we've discovered is that humans and and how we learned and how we basically see the world is mostly through our eyes. And we're studying that and trying to mimic that understanding. So as you know, as you can see, we require a lot of data, we're talking not just millions of training data, but billions. So right now, everything that you see that I'm presenting is a result of us managed to perfect the ability to create synthetic data. And that way, we can create synthetic data, meaning, we use a lot of 3D simulations, to build environments, which allows us to teach our AI the different behaviours and nuances of what humans will do in different environments. Alright, so first, let's start something simple and relatable from a use case perspective.

7:11  
You probably have been in the bank. Now, in a branch. As you know, there are signage screens, and typical bank would have about 1000 branches in a country. And rent banks are unique because they have a product, the moment you're born until your die. So they've got a wide range of products. And they want to market those products in what they call life moment, at which point your life, that particular product is meaningful. So no point of marketing home loan to a student, because they're still worried about schools. So understanding the life moment is quite important. And the thing with banks is that they are doing right in web and mobile, they track everything. They know what you're browsing, the websites are optimised to geo locate the tag, what kind of interests are people doing, so that's very rich. However, the moment they go in a branch, the bank doesn't know anything about you, you didn't become anonymous, and experience is bad. And if you're a marketer in a bank, and you run a what they call the name brands marketing, you really don't know which of those content works, because you just blast it on the screen in the hopes that people would consume it.

8:19  
So what we start doing is sold the problem where whenever someone asked the question, that we're spending this amount of money in our in branch marketing, which of those content works, which of those content works in which segment that those content kind of appeals to, so that they could further optimise and invest more and therefore create value of their physical space. So what we start doing is literally go back to that main principle of, we take a camera, and we use it as a sensor to track how humans how their visitors, their customers, consume content from their signage. So the way we do that is simple to explain. But if you understand computer vision, it's very difficult, not because it's complex algorithm, but it's requiring truckloads of data to be able to get to this level.

9:06  
So how it works is this the moment a customer basically walks in a branch, if there is a screen there are cameras are basically optimised and calibrated that we would know if someone's head and eyes are looking towards the screen. And the moment the person is basically close to the screen, we call that likelihood to see so you within the proximity of the screen. And from that point, the moment the customer or the visitor would look into the screen, we call that impression. That's the first second that your eyeballs are basically looking at the screen. So again, we're using this using standard CCTV camera mounted on the roof. This is not a specialised camera that basically looks interfaces. And at this point, we don't use facial recognition, we still treat all of the people within the field of view as anonymous. So we keep the data safe, and we comply on the strictest kind of data privacy globally.

9:56  
So, once we have basically taken the impression we to track whether that person will watch the content up until three seconds. And what's the purpose of that, because often, you know, you don't want to just grab data that's really not meaningful. If I had this basically larger screen, there's not much meaning out of that until they would decide to keep watching. If they watch up until three seconds, that's what we call it. Once again, we're constructing our, our, our naming conventions here. But glances are basically people that have watched the content up until three seconds.

10:26  
Now from there, the next step is views. Now views is basically a person that have watched the content up until the middle of that content that if you've got a content running for 15 seconds, seven seconds is your kind of mid range. So you will start to clock it as Okay, someone have viewed the content, because they've watched it until the middle. And then once the customer decides to watch until the end, that's your view through now, you might see a familiarity in terms of how you're doing it, because we're trying to mimic what you would do normally on websites or mobile phones. So making it familiar, gets the marketers to understand it better, and therefore they could consume the data easily. Because for us, it's no point of complicating it, because if complicated, then customers wouldn't consume it the right time. And they could make better actions and improve their conditions. So that's just one example of what we're doing with data.

11:15  
Now, there's a lot there that we're doing more than just tracking content on the screen, sometimes we trigger the right content when the right person comes. But again, at this stage, we are not looking at the specific faces, we're not doing facial recognition, we're not comparing faces, or at least age, we do not know who the person is, we just know that there is a person in front. And we don't save any videos of of this footage is we're not security company, we don't save it, we extract that information on the fly. See, it's very, very difficult thing to do, because you extract that information whilst the data is on stream. And then you make sure that you get right, because you don't basically get to do it again, because it's just streaming. So that's one important aspect of how you deal with volumes of data with a velocity that is very, very fast. But more than that, I want to talk about this. And to those guys that have been playing with data sets for a while and been into dashboards and correlating data sets a lot, you'd appreciate what what this looks like, more than just looking at content, we start to correlate information that are not just coming from cameras, but coming from websites, mobiles, other sensors that are in the branch, with purpose of correlating those seemingly unconnected data sets and understand the baton.

12:31  
So see, we stop looking at the who, but we start looking at why are they there? We stopped looking at the gender and their age, and we start looking at which location they're going in. So we start to differentiate, are they going into smart ATMs? Are they going into transaction areas? Are they going into discovery areas where they want to discover new products that might be able to help them? Or are they engaging the bankers and maybe potentially talk about home loans and personal loans and wealth management? What we have discovered is this.

13:01  
In banks, and especially the products that they carry their specific product range, that are quite personal, that requires human to human interaction. And it's important to capture that moment to understand how do you deal with those kinds of customers. A good example, these days, 90% of people that buys houses, will only buy a house once in the lifetime. And therefore the journey that they have to take often takes three months to six months, just doing research on their mobile on the web, just even before they engage a banker, because they feel that they don't have enough information yet to be able to have meaningful conversation with a banker. So you're dealing with something personal, something that involves a lot of emotion for an individual. And when they go into a branch, it's important that you recognise that, that if this person's been researching for six months, before you would engage them so that you can provide the right context once the conversation happens.

13:53  
Again, I mentioned earlier, we do approach our approach is conservative, but we do take bold actions and what kind of customers we partner. And these are the epic problems we're trying to solve. We're not saying we've solved this already. But we have started the journey and we started to see kind of meaningful results. And this dashboard shows you the kind of results that we see on the right side you see website kind of activities, mobile activities, brands, activities, and now we started to look into can we actually can combine that with all tenders. So here, we're talking about data that are coming from different sources, different forms and shape. And it's amazing what you can do with those data. But then it's daunting as well and sometimes difficult to extract the right information at the right time and basically present that to the right stakeholder to take action.

14:37  
But to get here, there's a lot of things that we have to get through and a lot of people we have to convince a lot of people we have to tell our story and and and we've been around a lot and our struggle is always how do we explain what we do in layman's term? If I stop talking, and introducing myself my name is Thor and I play with AI and data every day. People will not understand what the means and even my parents doesn't understand what I do and my uncle. So the things I do at work is something is not acceptable to be shared in a dinner table, for instance, because none of our family members understand. So we take pride of simplifying what we do so that not only we can explain it to our loved ones, but explain it to our customers. And we bought into this story. Now, everyone's probably familiar with salt.

15:22  
Apparently, to produce a kilo of salt, you need 30 litres of seawater. Now there's a lot of seawater to basically then produce a kilo of salt. And we got help but imagine and relate that it's almost like what we do with data, right? It's messy is, you know, so extracting oil from seawater, it's messy. It's often done in areas that it's a bit remote, like what we do we do some of these extraction in sources coming from cameras, that means it on prem, we're not just operating or website and requires logistics. When you look at Salt industry, they don't normally extract it near a very big city, where all the consumers are, they're normally extracted at a remote location, convert that into salt, and then put it into a truck. And then there's a logistics that needs to happen to deliver it to the consumers.

16:13  
It's similar to data to data that at least that we do, where today, one of our cameras can basically take 1.7 terabytes of data within one week of operation. This is just one camera. Right? And out of that 1.7 terabytes. When we look into the pattern, the data that will then be presented to the customer, the data that is meaningful, the data that actually carries the story. It's only 290 MB. So you can see the raw data for us is like seawater. And then the 290 MB insight is like the soul. And yes, the location, it is remote, because it's in branches, it's in physical spaces, you're dealing with hardware, you're dealing with network, you're dealing with different people, you're dealing with firewalls and infrastructures, and different types of camera. So you have different sources that basically comes to play. And you have to make sure everything is simplified, because at the end of the day, our customer is interested with inside, they're not interested in managing cameras, they're not interested in managing networks. So that's very important that whatever solution that you're trying to develop, we learn it the hard way that making it simple is important. But not to discount that is complex. But making it simple for the end user is super important. So I hope that helps in terms of explaining as well, where we're coming from into the data scientists out there, this might be a good story for you to share next time.

17:34  
So let's talk about some of our key challenges that we have to face as we as we develop our solution, our product, Viana. First is data consistency. As mentioned earlier, we are looking about terabytes of data per hour. And it's coming from different sources. And when we start to focus on selling to the enterprise customers, you cannot afford that they will start to think your data is not consistent. And when you give their data to them, they normally ask for the raw data and the process data. And they'll try to compare it themselves because most of our customers, they have their own data scientist as well.

18:11  
And decisions they make like a good example for a bank. They're often using those data to decide should we close this branch? Or should we open a new one? Should we merge two brands within one physical location, or should we keep them separate? Now you can see, it's very big decision, it would involve people losing jobs, or people getting new jobs, or brands that are merging together. So you can't just make those decisions based on data that are coming from third party without validating it. So data consistency is very important because one we're in the business where it comes from different locations, it comes from different sources comes from different formats has different requirements for logistics. But then at the end of the day, you have to make it consistent otherwise, they will not trust your data. Again, in the business where you deliver data to customers, you cannot afford that your customer lose trust in your data, otherwise, your product will not be successful. So data consistency is important. And that's one of the things that snowflake have allowed us to achieve very, very early stage in our development cycle.

19:09  
The next one is pretty much related to any kind of initiative that requires a lot of data, right? Once you basically managed to establish a logistics capability to get your on-prem data to the cloud. Once you establish the consistency of your data. At this point, your data becomes liquid gold to the customer. Anything valuable, customer wants to access it, and they want to access it really fast.

19:38  
So speed and efficiency in accessing various data sets is super important. Give you an example. Once we proven that the data we produce can be correlated on other seemingly unconnected data sets. All of a sudden people coming from web groups mobile groups call centres are asking us for data because they want to correlate it on their end so that they could augment the value of their own data sets to

20:00  
Now, if our answers would be, Oh, give us three months, because we'll try and build your pipeline for you to basically get it. When we lose that customer, we lose the opportunity to continue to expand in an enterprise environment. But that standard value of making the right decision and infrastructure, we're lucky that most of our customers use snowflake too. So by the time we'd say, we could just establish this no pipe to you, it's almost like a day job. And up it goes, the data is secure, they could access the data. And off they do augmenting the data sets. So as a result, we have become an important kind of broker for data that are coming from physical world, and making available to those people that are not very well versed in dealing with a physical space. And that's super important, because in most brands, everything's super optimised in the web, and mobile, but anything that runs on physical at best, they've got a kiosk there, that is operated by a small, small group of people that really don't have a roadmap for growth.

20:51  
But the moment you see a value of improving the use case, and what you can actually get from the physical space, and how that data becomes a documentation piece for other data that's already sitting in your database, and your warehouse becomes valuable. And that's the sort of one of the key pieces of our story that if you do this, right. And if you do this at a speed where everyone expects you to deliver, then you'll get success early on, on your product cycle product lifecycle.

21:20  
The last one is mostly the bottom line. As you know, I said earlier, I'm mostly a product builder. And with product builds, you have to find capital to basically build something that you want, and what you want, and what customers want and what the stakeholder wants a completely different thing. And the best way to do it, I believe is that, how can you basically go to market the quickest possible and prove to everybody, your customers, your stakeholders that there is value with this product, because here's the customer showing willingness to pay.

21:52  
And that's really the key story that I want to share today. If you want to build something, if you've got a great idea, you're thinking and say, right, we're gonna have to build this one up, it is important that you have to prove its value. By getting to show in your bottom line, there's no point of building something for three years, and only to find out that it doesn't make sense. And trust me, I've done that in my, in my career, where some of the products that I thought would be really successful, was done in a very long period of time. The budget is really, really good, you don't have constraints. And yet when you go to market, no one wants it.

22:29  
Versus what we've recently done, where we're launching it before the sort of 12 months mark or have done that, for the first time, we're launching it in the middle of a pandemic, we have less people. And within 12 months, we managed to basically convince the right customers to invest, not because we're new, or we're different, but because we're solving a specific problem at the right speed without compromising the quality. Right. So those are very important things that you have to mix as you basically build your own product.

22:58  
A little bit about our infrastructure, right, I mentioned that we are operating on premise, and we're operating in the cloud. On the left side, you can see devices, cameras, sensors, we work with any kinds of cameras, scanners that are basic and and something that you'd see around on CCTV or USB cameras, you see a lot on your electronic storage. And then we use what we call an edge device. Now the edge device could be as small as your standard computer, or, you know, those specialised computers that you'd basically see around kiosks, or, depending on use cases we use we, we often have servers that are often used for data centres that requires a lot of compute if you know if it's a complex use case, like theft detection, something like that. And then the right side, the right side of things, there's the cloud, where we take all the data from the edge, and then we funnel it to the cloud. And basically, that's where we constantly analyse it, squeeze insights out of it, and then give it back to our customers so that they could make better decisions. Um, there's a lot there, but probably not going to go deep dive and most of you guys are probably in development groups, and you're in an engineer, engineering team, this might be relevant for you. And you could start to reverse engineer what's basically comes behind it.

24:07  
But the most important part that I want to emphasise is that on the premise side, that's where we promise that we process tonnes and tonnes of data sets, we're talking terabytes per hour. And we're processing it really, really fast. And at the velocity, where it's almost near real time, because it's all in memory. Purpose being is that edge storage is quite expensive as well. And often, you know, customers don't want to spend that money. So you have to basically make sure that you need to develop a solution that works financially, commercially. And then at the same time sustainable. Right. So that's sort of our infrastructure and snowflake that helps us analyse data in real time.

24:50  
Give our customers and third party access to those data in a secure way so that it can continue to augment other data sets. So I think that's one key ingredient that we've got, we take something quite complex and we simplify it. And not only we simplify it within our ecosystem, we're simplifying it for others to consume.

25:07  
Now, the value proposition and I think the the punch line for the talk today is about how valuable is it to make the right decision at the right time when building a product. Now, I'm a geek myself. And given the chance, I want to build everything from ground up.

25:26  
But it's important to understand and to leverage tools that are already proven and tested, you know, tools that have been there used by many companies, and consider that to be a baseline, you know, to be your foundation in building something. Now, it's difficult to justify quickly within this period of time, but let me share you some of the documents this is a basically copy and paste on some of the documents that we have produced when we starting to build Viana when we're starting to conceived idea when we finish the designs and this is us with some of our tech leads planning. How many Sprint's do we need to to, to invest to build this product and make it to the point that it's shippable. And Sprint's by the way for us is two weeks. So initially, when we start looking into the backlog, all the user stories, all the designs or the test cases, the technical details, we reviewed all of it, we've spent a few weeks reviewing it, and the optimise estimates, 38 weeks now, that's almost 18 months. And that's 18 months, considering about 11 engineers to basically develop a product, and basically then release it and make it you know, go to market.

26:38  
And obviously, when we presented that to the board, they said, well, we can't wait for two years to basically go to market find us an ability for us to go to market within 12 months. So in short, we've tested a lot of things we've trialled we've got a lot of friends in the industry, we've asked around. And we started to consider, alright, what are the core things that we have to build that are important for us to build our own? And what are the things that we can leverage and we could almost extend the shoulder of giants, so that we could focus on our core value proposition and not worry about plumbing and infrastructure moving forward. It needs to be fast, it needs to be reliable, it needs to be secure. And it needs to be a proven and tested environment. And more importantly, it needs to be in an ecosystem so that others as well are using it, especially in the ecosystem of our customers. So got the story short, we've used different platforms and tools, and one of those tools is snowflake. And the result is this from 38 sprints, we managed to complete our build in 17 sprints. Now, you should know that when we were planning and starting the development, it was late 2019. October, November, and we're trying to start around December.

27:52  
And when we got approved, by using other tools, we could basically go to market really quick, we started the journey. And as you know, by March COVID happens. And we're lucky that the business decided let's continue on. But we have to do a lot of different so not only we finished it in 17 sprints from 38 sprints, we finished it even though we have to do a bit of pivot when the COVID It's because we need to develop develop features that that allowed us to basically play in the space debris example is that as you know, logistics becomes a big problem, because you cannot ship hardware around because the borders are closed.

28:25  
So therefore we need to be able to operate on cameras that are already installed and compete that are already installed. To do that, we need to simplify things. And we need to improve our way to collaborate and other solutions that are already in the market. So we could be like a Remora fish in a shark. And that basically was our idea to basically get into the market as quick as we can, despite the pandemic.

28:44  
So the summary is from 38 sprints going to 17, Sprint's that's like 18 months to eight months. And from 11 engineers, to five Engineers without compromising quality. Now, don't just take my word for it, like just saying, hey, you know, you might just be saying because it's your product, you didn't compromise quality. But I want to share that at this stage, right, we now have proven many times that we have taken data sets that takes three weeks to produce insights. Now we could produce it within three seconds. Now that's not my own words, those are from our customers. We managed to speed up the build process by 55%. And again, it goes beyond that 17 The first 17 Sprint's that we've done, right, we've been doing that day in day out, even after we've deployed and we managed to basically go live from, you know, initially having 11 engineers to five engineers. So by the time we actually launched that 17 Sprint kind of work. There's only five engineers that we've got. Occasionally I'll jump in and do the coding as well. But as as you do normally when you're you're doing the build, but that have enabled us to cement a baseline and today I'm proud and happy to share that.

29:58  
We are now about 70 engineers within the team. And from the first two big customers, we've kind of secured within Australia, we now operate in four continents globally. And again, those are the the benefit of figuring out an infrastructure that enables you to have the speed and agility without compromising the quality. By by that saying, You need to always make sure that you have an infrastructure that could provide data that is consistent, an infrastructure that could provide data that is easily accessible, right, and more importantly, needs to be secure.

30:34  
I want to conclude this session. By sharing a piece of insight. It's something that really resonates to me, as I develop my career and continue to build new products that do not waste your time.

30:49  
And do not waste data. Because you're too busy making your infrastructure to enable your data consistent. Enable your data available and accessible to third parties, and for securing it. Leverage tools, tools like snowflake, and enables you to have extra time to really focus on doing the activities that your customers will pay you, meaning extract that meaningful insight out from your data, find the story behind that data and give it to your customer at the right time. Because data today are perishable, there's no point of getting data that are not meaningful or not useful to customers, because it's not relevant anymore.

31:31  
I hope this session is meaningful and informative for you. And again, thank you for having me today.