AI on the edge
Artificial intelligence (AI) has already been attracting the attention of deep tech investors for some years. The reasons why are clear. In its ‘Sizing The Prize’ analysis of artificial intelligence (AI), PwC forecast that AI will contribute $15.7 trillion to the global economy by 2030, with the ‘AI boost’ available to most national economies being approximately 26%. But what investors often overlook is that AI is not singular. Many individual components must work together to create AI.
‘Traditional’ AI
At its core artificial intelligence consists essentially of detecting statistical patterns in signals with many dimensions, such as analysis of audio frequencies (voice recognition) or high-resolution images (face recognition). The repetition of this search in order to detect these patterns is the basis of artificial intelligence.
There are usually three components to AI:
- First, given a data set, learning what the patterns are.
- Second, building a model that can detect these patterns.
- Third, model deployment to the target environment.
Traditionally, data mining or learning was done by experts in the matter who would develop some sort of classifier or detector based on certain features, and then try to see their correlations. This process was tedious and time consuming.
A surge in power
Thankfully around 2010 the landscape changed drastically, powered by a surge in computational power. Suddenly, the core element in AI, known as neural networks, became viable, as computers were able to do millions of mathematical operations at great speed.
Instead of experts detecting features beforehand, the modern approach changed to feeding as much data as possible into the neural networks and letting them pick up the statistical patterns on their own. This method, or ‘learning algorithm’, allows a higher probability to obtain better results.
In a nutshell, it is the input of lots of data into a learning algorithm and then hoping it finds the right patterns. This extremely complex process is what is called ‘training’, and it generally requires lots of computer resources.
Once that a new AI model is ‘trained’, next comes the deployment of the AI to the target environment. Data will be collected from the signal and analysed. This is called ‘AI inference’. There are two ways to execute this model:
- Running this algorithm either in an ad hoc remote server or in the cloud. (AI on the cloud)
- Directly deployed on the computer where the data is produced. (AI on the edge).
AI on the edge vs AI on the cloud
The main advantage of AI on the cloud is the availability of large computational resources necessary to carry out the inference. The disadvantage is that data has to go to a server far away, perform the calculations, and then come back to the location of the source of the signal. Inevitably, just because of the laws of physics, there is going to be latency (delays) into shipping data and then returning it.
AI on the edge does not suffer from this latency issue, because the processor is located a couple of inches away from the sensor. In this case, however, computer power is the challenge to deal with: how to get this model to run in a small computer that is constrained by energy, memory, CPU power, and physical space?
There are several techniques to make a model fit in a small computer:
- Quantization is the most common one. Numbers are represented on the computer with reduced precision, thereby increasing the model speed.
- Compression. The less efficient parts of the model are reduced in size and, in this way, coming up with another lightweight model.
- Distillation. Similar to compression, a very large model is distilled into a smaller one.
These techniques are extremely complicated and require a lot of expertise and manual work. Furthermore, these techniques involve some trade-offs. The main setback is the loss of precision in the AI model. Many companies offer AI on the edge based on these techniques, but the reality is that in many cases the end result is not operative (optimal) due to the lack of precision.
A more general solution seeks to maximise the computer resources available to these systems. Equipping the system with better hardware might not be a possibility due to energy and physical space limitations, or just be too costly to be practical. It is also possible to improve the system’s software, but this is an extremely complex problem, and if executed poorly, it will result in either costly to maintain or even unusable applications.
State-of-the-art
At the beginning of April 2021, Klepsydra AI version 3 was released. Klepsydra greatly simplifies the implementation of software optimizations in all edge computers. And the level of performance gained is higher than any other software solution in the market.
Inspired by algorithmic trading techniques, Klepsydra’s software for edge data processing results in up to 8 times faster data processing and 50% less power consumption compared to standard edge computing solutions. Klepsydra AI is model agnostic, covering applications such as vision navigation and object detection.
A future on the edge
The demand for AI on the edge is exponentially growing: Satellites for earth observations and telecoms, car navigation and collision avoidance in traffic, agricultural robotics, face recognition in smart phones, etc.
All these applications need real-time response and latency makes the “AI on the cloud” approach unworkable. Waiting for hundreds of milliseconds before being able to react to the stimuli could otherwise lead to catastrophic consequences.
This is why “AI on the edge” is an extremely hot topic, and solutions like Klepsydra are critical: In contrast to the other techniques discussed, better software is the most cost-efficient optimization, and will make the difference between successful and failed AI projects.