I spent some time today listening to Dale Smith (REI Lead Engineer – Search Platform) and Chris Phillips (REI QA Engineer – Search Platform) discuss REI’s efforts and also Luis Bitencourt-Emilio of Reddit discuss their efforts around search at a event in Seattle. They had previously provided talks at Lucene/Solr Revolution 2017 Conference (which I include below). These were two very illuminating discussions about the critical nature of search and the approaches that two very different companies took to address their rapidly growing search needs.
One of the critical aspects of this effort required they move to the new Search engine without changing the existing API and while retaining their SEO impact. There were many challenges on the way and this is presentation highlights their challenges and their success and how they went about developing a parallel search engine and testing it – effectively changing the wheels of the car while they were driving it.
Luis Bitencourt-Emilio of Reddit also provided their efforts around search and their increased use and appreciation of the value of the machine learning aspect of Lucidworks Fusion AI.
This presentation is already dated (it has been six months) and their new stack has changed and they have added Kafka to their solution-set.
Related link : ML Model Training and Prediction Using Lucidworks Fusion
All of our devices are sending mammoth amounts of telemetry (= data) and a lot of that data is useful. With the emergence the “Internet of Things” (as both a concept and a reality) over the past few years Microsoft has jumped at this new opportunity. Numerous efforts they are engaged in are around Azure IoT Core, certified device catalog, providing cloud/intelligence to the edge, remote monitoring, Azure Iot Hub Device Provisioning, time series insights and quite a bit more. This is the tip of the iceberg – there are a number of other related features around, for example, machine learning.
One of the more interesting introductory talks to what Microsoft has introduced is from Sam George, Director of Microsoft Azure IoT. It is a long talk but covers a lot of territory :
One very interesting aspect of the work being done is Time Series Insights which allows you to stream put your time series data into this tool and understand what is going on with your monitored device. This is a particularly interesting application. There is no schema or telling the tool what the format of your data is, it will figure it out. This talk provides an overview of Azure IoT Time Series Insights.
The first talk hits the mark on providing a rapid-fire quick history and update on machine learning and also provides an introduction to Amazon’s 16 GPU Amazon EC2 instance to train or act as the hardware foundation for the new world of machine language frameworks.
From the Deep Water website – “Using complex multi-layer artificial neural networks, Deep Learning helps us derive insights from large unstructured data such as images, videos, sound and text or structured data from transactional databases such as financial data or time series. It enables innovations in domains as varied as medicine, social media, customer service, targeted marketing, automotive safety, security or fraud detection.”
In the next video, Amazon offers up a 8 and 16 GPU EC2 instance. A P3 instance can offer 1 PetaFLOP in a single instance.
A side note: We can differ on the optimism expressed but we are looking at the building of a new machine learning technology which will have a deeply profound ability to change society in radical ways. This is no longer speculation. An equivalent of a “Cambrian explosion” is on its way and will hit human societies in relatively short order. For example, what happens when you don’t need drivers, cashiers, analysts, and a wide range of other jobs because machine-learning agents do the work? When technology will not generate new jobs for humans but rather for AI? What effect will this have on human societies in general to see such wide-spread job losses. Will governments act on behalf of their citizens or on behalf of other interests. It is an interesting question worth pondering now (as opposed to when we are faced with the problem). Two authors Robert McChesney (professor at the University of Illinois at Urbana–Champaign) and John Nichols (journalist) offered a view of this future in 2016 in the book “People Get Ready – The fight against a jobless economy and a citizenless democracy “.
Here is an example of what computer vision systems can do. In this example, Radiant Vision Systems® offers a glimpse into a vision system and how it works. In the video, Radiant Vision Systems® imaging photometers and colorimeters are applied at each stage of a production of display devices and detects light, color, and surface defects .
If you are interested in computer vision-guided robotics there is a nice talk on the topic. David Dechow, Staff Engineer for Intelligent Robotics/Machine Vision, FANUC America Corporation provides guidance on where vision-guided robotics makes sense in different settings. It discusses the need for smart design of robot work cells and how to incorporate vision.
This article is on machine learning and open source frameworks. It covers open source frameworks. The frameworks cover the gamut of machine learning : supervised learning, unsupervised learning and reinforcement learning. If you are interested in machine learning this is a good overview of available open source frameworks.
If you are wondering what is happening in the area of machine learning and medicine – the short answer is : quite a bit. Across seven areas we are seeing advances in disease identification/diagnoses, personalized treatment, drug discover/manufacturing, clinical trial research, radiology/radiotherapy, smart electronic health records and epidemic prediction.
One of the hottest topics today is around machine learning – you could be excused if you didn’t know that it actually has been around for a long time in a different context and survived the “AI winter”. One area of machine learning focuses on ‘smart’ image recognition and its use in medical imaging. The following article provides a perspective on how medical imaging is being changed by machine learning.
It’s really seldom that you get a chance to see a different thought pattern emerge in database design that literally covers the key advantages of both relational and NoSQL databases and offers a new set of features. Datomic is an aggressive attempt to create a next-generation database that moves us beyond the usual and is built around running in current and future cloud architectures. Let’s start with the some of the features in Datomic. It provides support for ACID transactions, Joins, a sound data model and a logical query language called Datalog. Most of all it breaks with the rigidity of document-based models and relational database models. It also breaks with a model of mutability and a lack of audit data trails. To understand it is worth taking a tour of both it and it’s query language. One surprising aspect that it provides this by using engines like Couchbase, Cassandra,Riak, etc as nothing more than storage engines or services. Let’s start with the tour of Datomic :
By deconstructing the current generation of databases and implementing a modern database focused on breaking with designs from decades ago or some of the issues with the newer NoSQL databases (lack of Joins, lack of ACID, etc). To understand the power of the database you have to explore the power of the datalog query language:
Large, medium and small companies using Datomic. So you may be interested in the “why”. In this presentation – developers of the Brazilian NuBank expose four hidden super-powers of Datomic. In this case, the Clojure programming language is used.
and here is another use-case from Room Key ( joint project of Hilton, Choice, Hyatt, IHT, Marriot, Windom) :
More reading :
Here is a nice example of creating software that simulates a parking garage. Two different examples. If you are learning Clojure this might be of interest to you. In the first example, Clojure Refs are used. The problem is to simulate operations on a garage used for parking vehicles – vehicles come into the parking garage, park and then later leave. They are identified by their license plate number. This is a nice example for those looking for Clojure examples.
In the second example, the same parking garage problem from the previous post is solved using
clojure.spec . See more on the clojure.spec rationale and the guide. Here is the second example which uses Clojure.spec.