A new kind of IoT cloud for distributed AI

Like much of the world, Ithaca has pivoted sharply towards self-isolation while the coronavirus situation plays out.  But work continues, and I thought I might share some thoughts on a topic I've been exploring with my good friend Ashutosh Saxena at Caspar.ai (full disclosure: when Ashutosh launched the company, I backed him, so I'm definitely not unbiased; even so, this isn't intended as advertising for him).

The question we've been talking about centers on the proper way to create a privacy-preserving IoT infrastructure.  Ashutosh has begun to argue that the problem forces you to think in terms of hierarchical scopes: data that lives within scopes, and is only exported in aggregated forms that preserve anonymity.  He also favors distributed AI computations, again scoped in a privacy-preserving manner.  The Caspar platform, which employs this approach, is really an IoT edge operating system, and the entire structure is hierarchical, which is something I haven't seen previously (perhaps I shouldn't be surprised, because David Cheriton, a leading operating systems researcher, has been very actively involved in the design).

Ashutosh reached this perspective after years of work on robotics.    The connection to hierarchy arises because a robot often has distinct subsystems:  when designing a robotic algorithm one trains machine-learned models to solve subtasks such as motion planning, gripping objects, or reorienting a camera to get a better perspective on a scene.  This makes it natural for Ashutosh to view smart homes, multi-building developments and cities as scaled up instances of that same robotic model.

Interestingly, this hierarchical perspective is a significant departure from today's norm in smart home technologies.  Perhaps because the cloud itself hasn't favored edge computing, especially for ML, there has been a tendency to think of smart homes and similar structures as a single big infrastructure with lots of sensors, lots of data flowing in, and then some form of scalable big-data analytic platform like Spark/Databricks on which you train your models and run inference tasks, perhaps in huge batches.  Without question, this how most AI solutions work today: Google maps, Facebook's TAO social networking infrastructure, etc.

The relevance is that Google is doing this computation using a scalable system that runs on very large data repositories in an offline warehouse environment.  This warehouse creates both temptation and reward: you created the huge data warehouse to solve your primary problem, but now it becomes almost irresistible to train ad placement models on the data.  If you make your money on ads, you might even convince yourself that you haven't violated privacy, because (after all), a model is just a parameterized equation that lumps everyone together.   This rewards you, because training advertising models on larger data sets is known to improve advertising revenues.  On the other hand, data mining potentially can directly violate privacy or user intent, and even a machine-learned model could potentially reveal unauthorized information.

Ashutosh believes that any data-warehousing solution is problematic if privacy is a concern.  But he also believes that data-warehousing and centralized cloud computations miss a larger opportunity: that the quality of the local action can be washed out by "noise" coming from the massive size of the data set, and that overcoming this noise will require an amount of computation that rises to an unacceptable level.  Hence, he argues, you eventually end up with privacy violations, an unsurmountable computational barrier, and a poorly trained local model.  But on the other hand, you've gained higher ad revenues and perhaps for this reason, might be inclined to shrug off the loss of accuracy and "contextualization quality", by which I mean the ability to give the correct individualized response to a query "in the local context" of the resident who issued the query.

We shouldn't blindly accept such a claim.  What would be the best count-argument?  I think the most obvious pushback is this: when we mine a massive data warehouse in the cloud, we don't often treat the whole data set as part of some single model (sometimes we do, but that isn't somehow a baked-in obligation).  More often we view our big warehouse as co-hosting millions of separate problem instances, sharded over one big data store but still "independent".  Then we run a batched computation: millions of somewhat independent sub-computations.  We gain huge efficiencies by running these in a single batched run, but the actual subtasks are separate things that execute in parallel.

What I've outlined isn't the only option: one actually could create increasingly aggregated models, and this occurs all the time: we can extract phonemes from a million different voice snippets, then repeatedly group them and process them, ultimately arriving at a single voice-understanding model that covers all the different regional accents and unique pronunciations.  That style of computation yields one speech model at the end, rather than a million distinct ones each trained for a million distinct accents (what I called "local" models, above).

Ashutosh is well aware of this, and offers two responses.  First, he points to the issue of needing to take actions or even dynamically learn in real-time.  The problem is that to create a giant batch of a million sub-computation, with tasks that trickle in, you would often need to delay some tasks for quite a long time.   But if the task is to understand a voice command, that delay would be intolerable.  And if you try to classify the request using a model you built yesterday, when conditions differed, you might not properly contextualize the command.

In a perspective focused primarily on computational efficiencies, one needs to also note that doing things one by one is costly: a big batch computation will amortize over the huge number of parallel sub-tasks.  But in the smart home, we have computing capability close to the end user in any case, if we are willing to use it.  So this argues that we should put the computation closer to the source of the data for real-time reasons, and in the process, will gain localization in a natural way.  Contextualized queries fall right out.  Then, because we never put all our most sensitive and private data in one big warehouse, we simultaneously have saved ourselves a huge and irresistible temptation that no ad-revenue driven company is likely to resist for very long.

The distributed AI (D-AI) community, with which Ashutosh identifies himself, adopts the view that a smart home is best understood as a community of expert systems.  You might have an AI trained to operate a smart lightswitch... it learns the gestures you use for various lighting tasks.  Some other AI is an expert on water consumption in your home and will warn if you seem to have forgotten that the shower is running.  Yet another is an expert specific to your stove and will know if dinner starts burning...

For Ashutosh, with his background in robotics, this perspective leads to the view that we need a way to compose experts into cooperative assemblies: groups of varying sizes that come together to solve tasks.  Caspar does so by forming a graph of AI components, which share information but also can hold and "firewall" information.  Within this graph, components can exchange aggregated information but only in accordance with a sharing policy.  We end up with a hierarchy in which very sensitive data is held as close to the IoT device where it was captured as possible, with only increasing aggregated and less sensitive summaries rising through the hierarchy.  Thus at the layer where one might do smart power management for a small community, controlling solar panels and wall batteries and even coordinating HVAC and hot water heaters to ramp power consumption up, or ease it off, the AI element responsible for those tasks has no direct way to tap into the details of your home power use, which can reveal all sorts of sensitive and private information.

I don't want to leave the impression that privacy comes for free in a D-AI approach.  First, any given level has less information, and this could mean that it has less inference power in some situations.  For example, if some single home is a huge water user during a drought, the aggregated picture of water consumption in that home's community could easily mask the abusive behavior.  A D-AI system that aggregates must miss the issue; one that builds a data warehouse would easily flag that home as a "top ten abuser" and could dispatch the authorities.

Moreover, D-AI is more of a conceptual tool than a fully fleshed out implementation option.  Even in Caspar's hierarchical operating system, it is best to view the system as a partner, working with a D-AI component that desires protection for certain data even as it explicitly shares other data: we don't yet know how to specify data flow policies and how to tag aggregates in such a way that we could automatically enforce the desired rules.  On the other hand, we definitely can "assist" a D-AI system that has an honest need for sharing and simply wants help to protect against accidental leakage, and this is how the Caspar platform actually works.

Ashutosh argues that D-AI makes sense for a great many reasons.  One is rather mathematical: he shows that if you look at the time and power complexity of training a D-AI system (which comes down to separately training its AI elements), the costs scale.  For a single big AI, those same training costs soar as the use case gets larger and larger.  So if you want a fine-grained form of AI knowledge, a D-AI model is appealing.

The Caspar IoT cloud, as a result, isn't a centralized cloud like a standard data warehouse might use.  In fact it has a hierarchical and distributed form too: it can launch a D-AI compute element, or even an "app" created by Caspar.ai's team or by a third party, in the proper context for the task it performs, blocking it from accessing data that isn't authorized to it.  Processing nodes can then be placed close to the devices (improving real-time responsiveness), and we can associate different data flow policies at each level of the hierarchy, so that higher-level systems have increasingly less detailed knowledge from the remote and more sensitive IoT manager systems that might know far more, but only for a specific reason, such as to better understand voice commands in a particular part of the home: a "contextualized" but more sensitive task.

Then one can carry all of this even further.  We can have systems that are permitted to break the rules, but only in emergencies: if a fire is detected in the complex, or perhaps a wildfire is active in the area, we can switch to a mode in which a secondary, side-by-side hierarchy activates and is authorized to report to the first responders: " there are two people in the A section of the complex, one in unit A-3 and one in unit A-9.  In unit A-3 the resident's name is Sally Adams and she is in the northeast bedroom..."  All of this is information a standard smart home system would have sent to the cloud, so this isn't a capability unique to Caspar.  But the idea of having an architecture that localizes this kind of data unless it is actually needed for an emergency is appealing: it removes the huge incentive that cloud providers currently confront, in which by mining your most private data they can gain monetizable insights.

In the full D-AI perspective, Caspar has many of these side-by-side hierarchies.  As one instantiates such a system over a great many homes, then communities, then cities, and specializes different hierarchies for different roles, we arrive at a completely new form of IoT cloud.  For me, as an OS researcher, I find this whole idea fascinating, and I've been urging Ashutosh and Dave to write papers about the underlying technical problems and solutions (after all, before both became full time entrepreneurs, both were full time researchers!)

We tend to think of the cloud in a centralized way, even though we know that any big cloud operator has many datacenters and treats the global deployment as a kind of hierarchy: availability zones with three data centers each, interconnected into a global graph, with some datacenters having special roles: IoT edge systems, Facebook point-of-presence systems (used heavily for photo resizing and caching), bigger ones that do the heavy lifting.  So here, with D-AI, we suddenly see new forms of hierarchy.  The appeal is that whereas the traditional model simply streams all the data upwards towards the core, this D-AI approach aggregates at what we could call the leaves, and sends only summaries (perhaps even noised to achieve differential privacy, if needed) towards the larger data warehouse platforms.

So how does this play out in practice?  It seems a bit early to say that Caspar has cracked the privacy puzzle, which is the event that could make smart homes far more palatable for most of us.   On the other hand, as the distributed IoT cloud's protection barriers grow more sophisticated over time, one could believe that it might ultimately become quite robust even if some apps are maliciously trying to subvert the rules (if we look at Apple's iPhone and iPad apps, or the ones on Google's Android, this is definitely the trend).  Meanwhile, even if privacy is just one of our goals, the D-AI concept definitely offers contextualization and localization that enables real-time responsiveness of an exciting kind.  The Caspar platform is actually up and running, used in various kinds of real-estate developments worldwide.  Their strongest uptake has been in residential communities for the elderly: having a system that can help with small tasks (even watching the pets!) seems to be popular in any case, but especially popular in groups of people who need a little help now and then, yet want to preserve their autonomy.

A few 10-Year Challenges for Distributed Systems and IoT

A newspaper column on "next decade" predictions got me thinking about crystal ball prognoses.   Tempting as it is to toss in my views on climate change, surveillance in China and self-driving cars, I'll focus this particular blog on computer systems topics in my area.

1. AI Sys and RT ML.  These terms relate to computer systems created to support AI/ML applications, and that often involve addressing real-time constraints.  There is a second meaning that centers on using AI tools in networks and operating systems and database platforms. I'm open-minded but, so far, haven't seen convincing demonstrations that this will yield big advances.  I'll focus on the first meaning here.

Although AI Sys terminology is trendy, the fact is that we are still at the very earliest stages of incorporating sensing devices into applications that leverage cloud-scale data and machine learning.  As this style of system deployment accelerates in coming years, we'll start to see genuinely smart power grids (existing grids often proclaim themselves to be "smart" but honestly, not much use is being made of ML as of yet), smart homes and offices, smart cities and highways, smart farms....  The long-term potential is enormous, but to really embrace it we need to rethink the cloud, and especially, the cloud edge where much of the reactive logic needs to run.  This is why the first of my predictions centers on the IoT edge: we'll see a new and trustworthy edge IoT architecture emerge and mature, in support of systems that combine sensors, cloud intelligence and big data.

Getting there will require more than just redesigning today's cloud edge components, but the good news is that an area begging for disruptive change can be an ideal setting for a researcher to tackle.  To give just one example: in today's IoT hub services, we use a database model to track the code revision level and parameter settings for sensors, connected and accessible or not.  The IoT hub manages secure connectivity to the sensors, pushes updates to them, and filters incoming event notifications, handing off to the function service for lightweight processing.  I really like the hub concept, and I think it represents a huge advance relative to the free-for-all that currently is seen when sensors are connected to the cloud.  Moreover, companies like Microsoft are offering strong quality of service guarantees (delay, bandwidth, and even VPN security) for connectivity to the edge.  They implement the software, then contract with ISPs and teleco's to obtain the needed properties.  From the customer's perspective what matters is that the sensors are managed in a trustworthy, robust and secure manner.

The puzzle relates to the reactive path, which is very far from satisfactory right now.  When a sensor sends some form of event to the cloud, the IoT hub operates like a windowing environment handling mouse movements or clicks: it functions as the main loop for a set of handlers that can be customized and even standardized (thus, a Canon camera could eventually have standard cloud connectivity with standard events such as "focused", or "low power mode", or "image acquired.").  Like with a GUI, the incoming events that need user-defined processing are passed to these user-defined functions, which can customize what will happen next.

The core problem is with the implementation.  First, we split the path: sensor-to-cloud uploads of large objects like photos are videos follow one path, and end up with the data automatically stoed into a binary large objects (BLOB) store, replicated for fault-tolerance.

Meanwhile, other kinds of events, like the ones just mentioned, are handled by small fragments of logic: cloud functions.  But these aren't just lambdas written in C++ or Scala -- they are typically full programs coded in Linux and then handed to the cloud as containers, perhaps with configuration files and even their own virtualized network mapping.  As a result, the IoT Hub can't just perform the requested action -- it needs to launch the container and pass the event into it.

The IoT hub accomplishes these goals using the "function service", which manages a pool of machines, picks one on which to launch this container for this particular Canon photo acquisition event, and then the program will load that event's meta-data and can decide what to do.  In effect, we launch a Linux command.

Normally, launching a Linux command has an overhead of a few milliseconds.  Doing so through the IoT hub is much slower: Today, this process takes as much as two seconds.   The issues are several: first, because the IoT Hub is built on a database like SQL server or Oracle, we have overheads associated with the way databases talk to services like the function service.  Next the function service itself turns out to do a mediocre job of warm-starting functions -- here the delay would center on caching, binding the function to any microservices it will need to talk to ahead of time (off the critical path), dealing with any synchronization the function may require.

I can't conceive of a sensible realtime use case where we can tolerate two seconds of delay -- even web page interactions are down in the 10-50ms range today, well below the 100ms level at which alpha-beta tests show that click-through drops.  So I would anticipate a complete redesign of the IoT hub and function layer to warm-start commonly needed functions, allow them to pre-bind to any helper microservices they will interact with (binding is a potentially slow step but can occur out of the critical path), and otherwise maintain a shorter critical path from sensor to user-mediated action.  I think we could reasonably target sub-1ms delays... and need to do so!

There are many other unnecessarily long delays in today's IoT infrastructures, impacting everything from photo and video upload to ML computation on incoming objects.  But none of this is inevitable, and from a commercial perspective, the value of reengineering it (in a mostly or fully compatible way) would be huge.

2. Cost-efficient sharable hardware accelerators for IoT Edge.  In prior blog postings, I've written about the puzzle of hardware for the IoT Edge (many people take that to mean "outside" the cloud, but I also mean "in the outermost tier of a data center supporting the cloud, like Azure IoT).  Here, the central question involves costs: modern ML and especially model training is cost-effective only because we can leverage hardware accelerators like GPU, TPU and custom FPGA to offload the computationally parallel steps into ultra-efficient hardware.  To this, add RDMA and NVM.

The current generation of hardware components evolved in backed back-end systems, and it is no surprise to realize that they are heavily optimized for batched, offline computing.  And this leads to the key puzzle:  today's ML accelerators are expensive devices that are cost effective only when they can be kept busy.  The big batches of work seen in the back-end enable today's accelerators to run in support of very long tasks, which keeps them busy and makes them cost-effective.  If the same devices were mostly idle, this style of accelerated ML would become extremely expensive.

In some sense, today's ML accelerators could have been at home in the old-styled batch computing systems of the 1970's.  As we migrate toward a more event-driven IoT edge, we also will need to migrate machine learning (model training) and inference into real-time contexts, and this means that we'll be using hardware accelerators in settings that lack the batched pipelining that dominates in the big-data HPC-style settings were those currently reside.  To be cost-effective we will either need completely new hardware (sharable between events or between users), or novel ways to repurpose our existing hardware for use in edge settings.

It isn't obvious how to get to that point, making it a fascinating research puzzle.  As noted, edge systems are event-dominated, although we do see streams of image and video data (image-processing tasks on photo or video streams can be handled fairly well with existing GPU hardware, so that particular case can be solved cost-effectively now).  The much harder case involves singleton events: "classify this speech utterance," or "decide whether or not to retain a copy of that photo."  So the problem is to do snap analysis of an event.  And while my examples involve photos and videos, any event could require an intelligent response.  We may only have milliseconds to react, and part of that reaction may entail retraining or incrementally adjusting the ML models -- dynamic learning.

The hardware available today isn't easily sharable across scaled out event-driven systems where the events may originate in very different privacy domains, or from different users.  We lack ways to protect data inside accelerators (Intel's new SIMD instruction set offers standard protections, but a GPU or TPU or FPGA is typically operated as a single security context: it is wide-open if a task runs on behalf of me immediately after one that ran on behalf of you: the kernel I've invoked could just reach over and extract any data left behind after your task was finished).

So why not use Intel's SIMD solutions?  For classification tasks, this may be the best option, but for training, which is substantially more expensive from a computational point of view, the Intel SIMD options are currently far slower than GPU or TPU (FPGA is the cheapest of all the options, but would typically be somewhere in between the SIMD instructions and a GPU on the performance scale).

It will be interesting to watch this one play out, because we can see the end goal easily, and the market pressure is already there.  How will the hardware vendors respond?  And how will those responses force us to reshape the IoT edge software environment?

3. Solutions for the problem blockchain was supposed to solve.  I'm pretty negative about cryptocurrencies but for me, blockchain is a puzzle.  Inside the data center we've had append-only logs for ages, and the idea of securing them against tampering using entangled cryptographic signatures wasn't particularly novel back when the blockchain for Bitcoin was first proposed.  So why is a tamper-proof append-only log like Microsoft's Corfu system not a blockchain?

There are several aspects in which blockchain departs from that familiar, well-supported option.  I'll touch on them in an unusual order: from practical uses first to more esoteric (almost, "religious") considerations, which I'll tackle last.  Then I want to argue that the use cases do point to a game changing opportunity, but that the whole story is confused by the religious zealotry around some of these secondary and actually, much less important aspects.

First among the novel new stories is the concept of a smart contract, which treats the blockchain as a database and permits the developer to place executable objects into Blockchain records, with the potential of representing complex transactions like the mortgage-backed securities that triggered the 2008 meltdown.  The story goes that if we can capture the full description of the security (or whatever the contract describes), including the underlying data that should be used to value it, we end up with a tamper roof and self-validating way to price such things, and our transactions will be far more transparent.

I see the value in the concept of a smart contract, but worry that the technology has gotten ahead of the semantics: as of the end of 2019 you can find a dozen tools for implementing smart contracts (Ethereum is the leader, but Hyperledger is popular too).  Less clear is the question of precisely how these are supposed to operate.  Today's options are a bit like the early C or Java programming languages: both omitted specifications for all sorts of things that actually turned out to matter, leaving it to the compiler-writer to make a choice.  We ended up with ambiguities that gave us today's security problems with C programs.

With blockchain and smart contracts you have even nastier risks because some blockchain implementations are prone to rollback (abort), and yet smart contracts create dependency graphs in which record A can depend on a future record B.  A smart contract won't seem so smart if this kind of ambiguity is allowed to persist... I predict that 2020 will start a decade when smart contracts with strong semantics will emerge.  But I'll go out on a limb and also predict that by the time we have such an option, there will be utter chaos in the whole domain because of these early but inadequate stories.  Smart contracts, the real kind that will be robust with strong semantics?  I bet we won't have them for another fifteen years -- and when we do get them, it will be because a company like Oracle or Microsoft steps in with a grown-up product that was thought through from bottom to top.  We saw that dynamic with Java and CORBA giving way to C# and LINQ and .NET, which in turn fed back into languages like C++.  And we will see it again, but it will take just as long!

But if you talk to people enamored with blockchain, it turns out that in fact, smart contracts are often seen as a cool curiosity.  I might have a narrow understanding of the field, but among people I'm in touch with, there is little interest in cryptocurrency and even less interest in smart contracts.  More common, as far as I can tell, is a focus on the auditability of a tamperproof ledger.

I'll offer one example that I run into frequently here at Cornell, in the context of smart farming.  You see variants of it in medical centers (especially ones with partner institutions that run their own electronic health systems), human resource management, supply chains, airports that need to track airplane maintenance, and the list goes on.  At any rate, consider farm to table cold-chain shipment for produce or agricultural products like cheese or processed meats.  A cup of yoghurt will start with the cow being milked, and even at that stage we might wish to track which cow we milked, how much milk she produced, the fat content, document that she was properly washed before the milking machine kicked in, that we tested for milk safety and checked her health, that the milk was promptly chilled and then stored at the proper temperature.  Later the milk is aggregated into a big batch, transported, tested again, pasteurized, homogenized, graded by fat content, cultured (and that whole list kicks in again: in properly sterile conditions, at the right temperature...).|

So here's the challenge: Could we use a blockchain to capture records of these kinds in a secure and tamperproof manner, and then be in a position to audit that blockchain for various tasks such as to confirm that the required safety steps were preserved, or to look for optimization opportunities?  Could we run today's ML tools on it, treating the records as an ordered collection and mapping that collection into an event Tensor Flow or Spark/Databricks could ingest and analyze?  I see this a fantastic challenge problem for the coming decade.

The task is fascinating and hard, for a lot of reasons.  One is that the domain is partly disconnected (my colleagues have created a system, Vegvisir, focused on this aspect).  A second question you can ask concerns integrity of our data capture infrastructure: can I trust that this temperature record is from the proper thermometer, correctly calibrated, etc?  Do I have fault-tolerant redundancy?  How can we abstract from the chain of records to a trustworthy database, and what do trust-preserving queries look like?  How does one do machine learning on a trusted blockchain, and what trust properties would the model then carry?  Can a model be self-certifying, too?  What would the trust certificate look like (at a minimum, it would need to say that "if you trust X and Y and Z, you can trust me for purpose A under assumption B...").  I'm reminded of the question of self-certifying code... perhaps those ideas could be applied in this domain.

I commented that this is the problem blockchain really should be addressing.  I say this because as far as I can tell, the whole area is bogged down on really debates that have more to do with religion than with rigorous technical arguments.  To me this is at least in part because of the flawed belief that anonymity and permissionless mining are key properties that every blockchain should offer.  The former is of obvious value if you plan to do money laundering, but I'm pretty sure we wouldn't even want this property in an auditing setting.  As for the permissionless mining model, the intent was to spread the blockchain mining revenue fairly, but this has never really been true in any of the main blockchain systems: they are all quite unfair, and all the revenue goes to shadowy organizations that operate huge block-mining systems.  As such, the insistence on permissionless mining with anonymity really incarnates a kind of political opinion, much like the "copyleft" clause built into GNU licenses, which incarnated a view that software shouldn't be monetized.  Permissionless blockchain incarnates the view that blockchains are for cybercurrency, that cybercurrency transactions shouldn't be taxed or regulated, and that management of this infrastructure is a communal opportunity, but also a communal revenue source.

Turning to permissionless blockchain as it exists today, we have aspects of this dreamed-of technology, but the solutions aren't fair, and in fact demand a profoundly harmful mining model that squanders energy in the form of hugely expensive proof-of-work certifications.  My colleague, Robbert van Renesse, has become active in the area and has been doing a survey recently to also look at some of the other ideas people have floated: proof of stake (a model in which the rich get richer, but the compute load is much reduced, so they spend less to earn their profits...), proof of elapsed time (a lovely acronym, PoET, but in fact a problematic model because the model can be subverted using today's Intel SGX hardware), and all sorts of one-way functions that are slow to compute and easy to verify (the parallelizable ones can be used for proof-of-work but the sequential ones  simply reward whoever has the fastest computer, which causes them to fail on a different aspect of the permissionless blockchain mantra: they are "undemocratic", meaning that they fail to distribute the income for mining blocks in a fair manner).  The bottom line, according to Robbert, is that for now, permissionless blockchain demands computational cycles and those cycles make this pretty much the least-green technology on earth. There is some irony here, because those who promote this model generally seem to have rather green politics in other ways.  I suppose this says something about the corrupting influence of potentially vast wealth.

Meanwhile, more or less betting on the buzz, we have a whole ecosystem of companies convinced that what people really want are blockchain curation products for existing blockchain models.  These might include tools that build the blockchain for you using the more-or-less standard protocols, that back it up, clean up any garbage, index it for quick access, integrate it with databases and AI/ML.  We also have companies promoting some exceptionally complex protocols, many of which seem to have the force of standards simply because people are becoming familiar with their names.  It will take many years to even understand whether or not some of these are correct -- I have serious doubts about a few of the most famous ones!

But here's my bet for the coming decade: in 2029, we'll be seeing this market morph into a new generation of WAN database consumers, purchasing products from today's database companies.  Those customers won't really be particularly focused on whether they use blockchain or some other technology (and certainly won't insist on permissive models with pervasive anonymity and proof of work).   They will be more interested in tamperproof audits and ML on the temporally-ordered event set.

Proof of work per-se will have long since died from resource exhaustion: the world simply doesn't have enough electrical power and cooling to support that dreadful model much longer (don't blame the inventors: the blame here falls squarely on the zealots in the cybercoin community, who took a perfectly good idea and twisted it into something harmful as part of their quest to become billionaires off the back of a pie-in-the-sky economic model).

The future WAN databases that emerge from the rubble will have sophisticated protection against tampering and the concept of trust in a record will have been elevated to a notion of a trustworthy query result, that can be checked efficiently by the skeptical end-user.  And this, I predict, will be a huge market opportunity for the first players to pull it off.  It would surprise me if those players don't turn out to include today's big database companies.

4. Leave-nothing-sensitive behind privacy.  The role of the cloud in smart settings -- the ones listed above, or others you may be thinking about -- is deeply enshrined by now:  very few smart application systems can avoid a cloud-centric computing model in which the big data and the machine intelligence is at least partly cloud-hosted.  However, for IoT uses, we also encounter privacy and security considerations that the cloud isn't terribly good at right now, with some better examples (Azure, on the whole, is excellent) and some particularly poor ones (I won't point a finger but I will comment that companies incented to place a lot of advertising often find it hard to avoid viewing every single user interaction as an invaluable asset that must be captured in perpetuity and then mined endlessly for every possible nugget of insight).

The upshot of this is that the cloud is split today between smart systems that are trying their best to spy on us, and smart systems that are just doing smart stuff to benefit us.  But I suspect that the spying will eventually need to end, at least if we hope to preserve our Western democracies.  How then can we build privacy-preserving IoT clouds?

I've written about this in the past, but in a nutshell, I favor a partnership: a style of IoT application that tries to "leave no trace behind" coupled to a cloud vendor infrastructure that promises not to deliberately spy on the end-user.  Thus for example when a voice command is given to my smart apartment, it may well need to be resolved up on the cloud, but shouldn't somehow be used to update databases about me, my private life, my friends...

I like the mental imagery of camping in a wilderness where there are some bears roaming around.  The cloud needs a model under which it can transiently step in to assist in making sense of my accent and choice of expressions, perhaps even contextualized by knowledge of me and my apartment, and yet when the task finishes, there shouldn't be anything left behind that can leak to third party apps that will rush into my empty campsite, hungry to gobble up any private data for advertising purposes (or worse, in countries like China, where the use of the Internet to spy on the population is a serious threat to personal liberties).  We need to learn to enjoy the benefits of a smart IoT edge without risk.

Can this be done?  I think so, if the cloud partner itself is cooperative.  Conversely, the problem is almost certainly not solvable if the cloud partner will see its revenue model break without all that intrusive information, and hence is hugely incented to cheat.  We should tackle the technical aspects now, and once we've enabled such a model, I might even favor asking legislative bodies to mandate privacy-preservation as a legally binding obligation on cloud vendor models.  I think this could be done in Europe, but the key is to first create the technology so that we don't end up with an unfunded and infeasible mandate.  Let's strike a blow against all those companies that want to spy on us!  Here's a chance to do that by publishing papers in top-rated venues... a win-win for researchers!

5. Applications that prioritize real-time.  Many IoT systems confront deadlines, and really have no choice except to take actions at the scheduled time.  Yet if we want to also offer guarantees, this poses a puzzle: how do we implemented solutions that are always sure to provide the desired timing properties, yet are also "as consistent" as possible, or perhaps "as accurate as possible", given those constraints?

To me this is quite an appealing question because it is easy to rattle off a number of ways one might tackle such questions.  For example, consider an ML algorithm that iterates until it converges, which typically involves minimizing some sort of error estimate.  Could we replace the fixed estimate by adopting a model that permits somewhat more error if the deadline is approaching?

Or here's an idea: What about simply skipping some actions because it is clear we can't meet the deadline for them?  I'm reminded of work Bart Selman, a colleague of mine, did fifteen years ago.  Bart was looking at situations in which an AI system confronted an NP complete question, but in a streaming context where variations on that question would be encountered every few seconds (he was thinking about robot motion planning but similar issues arise in many AI tasks).  What he noticed was that heuristics for solving these constrained optimization problems sometimes converge rapidly but in other situations diverge and compute endlessly.  So his idea, very clever, was to take the quick answers but to just pull the plug on computations that take too long.  In effect, Bart argued that if the robot is faced with a motion-planning task it won't be able to solve before its next step occurs, take the previously-planned step and then try again.  Sooner or later the computation will converge quickly, and the overall path will be both of high quality, and fast.

We could do similar things in many IoT edge settings, like the smart-things cases enumerated earlier.  You might do better to have a smart grid that finds an optimized configuration setting once every few seconds, but then coasts along using old settings, than to pause to solve a very hard configuration problem 20 times per second if in doing so, you'll miss the deadline for actually using the solution.  The same is true for management of traffic flow on a highway or in a dense city.

For safety purposes, we will sometimes still want to maintain some form of risk envelope.  If I'm controlling a smart car in a decision-loop that runs 20 times per second, I might not run a big risk if I toss up my hands even 4 or 5 times in a role. But we would not want to abandon active control entirely for 30 seconds, so there has to be a safety mechanism too, one that kicks in long before the car could cause an accident (or miss the next turn), forcing it into a safe mode.  I don't see any reason we couldn't do this: a self-driving car (or a self-managed smart highway) would need some form of safety monitor in any case, to deal with all sorts of possible mishaps, so having it play the role of making sure the vehicle has fresh guidance data seems like a fairly basic capability.  Then in the event of a problem, we would somehow put that car into a safe shutdown mode (it might use secondary logic to pull itself into a safety lane and halt, for example).

I could probably go on indefinitely, but every crystal ball eventually fogs over, so perhaps we'll call it quits here.  Have a great holiday and see you in the next decade!

Managed sensor deployments

In my cloud computing course at Cornell, students have asked why we are focused on Azure IoT during the current spring offering.  One answer is that I like to orient our MEng students towards technology sectors that are experiencing really dramatic growth, because I want them to have strong job prospects and the skills to secure jobs that will pay above average -- and historically, this has been a good strategy for all of us teaching at this level.

But implicit in my thinking is the assumption that Azure IoT (or the Amazon AWS counterpart) really is headed towards mainstream adoption.  So the question then has to be posed: why should we believe this?  After all, I've been around long enough to remember those crazy videos of people in the Xerox SRC lab or the MIT media lab who set out to capture a digital record of their entire day, and ended up dressed like deep-sea divers, with cameras and microphones all over their bodies and aimed in every possible direction.

That can't possibly be the future of IoT.

What then will be the drivers for this particular IoT surge, and why should we bet that this time, IoT has "crossed the chasm" (a reference to a wonderful 1991 book by Geoffrey Moore)?

I would argue that the first reason centers on a form of risk that creates a powerful pent-up demand.  The risk is simply that in homes, offices and public spaces we are increasingly surrounded by wifi-enabled devices that are capable of tracking us through audio, imaging, motion detectors, swipe card and RFID sensing... the list is long.  So where are these things?  Many are simply in our hands or pockets: any smart phone fits the picture.  But then there are room environmental controls that use small devices to track room or space occupancy -- and because cameras and microphones are so cheap now, naturally are based on those technologies.

I find it kind of ironic that ubiquity would have driven the cost of spying on us down to pennies per device, but there you have it: the vast quantities of inexpensive audio and video chips that have flooded the market due to their use in phones are ending up in just about everything else.  The same goes for simple Linux-based computing platforms: with ARM, there are a tremendous number of devices that have the compute capabilities of an old-style Linux box, and indeed, run Linux or one of the stripped-down real-time capable alternatives.

So we have this unusual picture in which volume has driven costs to the floor, creating a situation in which if you want your room air conditioning "sector controller" to be smart, the easiest thing may be to just include a fully capable small PC that can watch and listen to the space to see if anyone is in it.
But this then becomes a handy option for intrusion, or compromise by the folks who like to create giant bot deployments -- why bother to compromise my PC if they can just target my router, my smart TV set and home entertainment system, my microwave and fridge, my thermostat?  And this doesn't even get at the intentional cases: Alexa, Siri, Cortana and their friends, always waiting to hear their name mentioned, always listening.

It doesn't matter much which kind of space you focus on: whether at home, in the office, walking in the park, all of us are continuously within range of something or other.  And that device is at least in theory capable of hosting intelligent spyware.

To me this single insight is already enough to justify a major bet on Azure IoT and its cousins.  But focusing just on Microsoft, we have already seen the huge success of Office 365: Microsoft dithered but finally figured out that everyone likes their tools, and found a way to integrate them into a complete IT solution for modern enterprises with all sorts of intelligent (social-networking) features to let corporate customers leverage the knowledge inherent to their organization.  There are some obvious glitches (the one that drives me crazy is that the Office 365 version of Skype (Skype for Business) and Slack (Teams) refuse to talk to my desktop telephone, even though my phone uses a standard VOIP technology -- what an annoying oversight).

Anyhow, suddenly we see all of the world's medium to large corporations adopting Office 365 as a complete IT story for internal workplace collaboration, and beyond those annoyances, there really are big wins.  The technology is making Microsoft a winner again.

But every one of those Office 365 customers needs to worry about competitors spying on them, and some also worry about randomware invasions or other forms of disruptive intrusions.  Where would you focus that worry, right now?  I think your attention would be on the smart thermostat, the routers, and the myriad other intelligent devices that pervade the enterprise, and yet are basically insecure.

This is where I see a real opportunity for Azure IoT: the chance to be "Office 365 IoT" by using the security functions of the Azure IoT Hub to wire down all of those devices.  And this is at least the stated plan -- check out Microsoft's official stories.  First, they are trying hard to convince vendors to use a small hardware component called Azure Sphere to secure the device itself -- sensors with a trusted hardware security component (and if you don't love Sphere, Berkeley's David Culler has a bunch of research papers and ideas on sensor security that you could explore.  He used to even have a sensor technology company, although with the pace of turnover in the Bay Area, I'm not sure what became of it.)

A brilliant PhD student of mine, Z Teo, has a company in this space too (free advertising for him!): IronStack.  Z's focus is on securing the corporate network by gaining better control over the routing elements, especially the programmable SDN components.

Then the plan would be to connect every single sensor in the corporate campus to Azure IoT Hub,  every controllable router, every smart microwave oven and self-flushing toilet... and by doing so, to gain  minimal level of control over all these things.  The Azure IoT Hub is basically a massive active database: it has a secure link to the devices, and it wires them down: these sensors cease to be accessible over normal networks, so that once they are connected to the hub, they aren't connected to intruders and spies.  Next, the hub makes sure that the firmware is always properly patched.  Is your fancy printer running the proper software revisions?  With Azure IoT Hub, the answer should be "yes, if it is available on the network"  because if the answer were "no", the other part of the answer would be "but we've taken it offline and dispatched someone to fix it."

I honestly think that taking control over all of these devices is, by itself, the first killer app.  We would be in a much better place if all the smart devices in our environment (and now I mean all: not just the ones in the office, but the CATV things in the street and the smart phones -- the whole game) were actively controlled by security-management infrastructures that can just keep the software patched and avoid random drive-by takeovers.

But once you have this active control over billions (maybe trillions, someday) of devices, it becomes very appealing to make the home and office smart -- and this is the second reason I would bet on Azure IoT right now.  That huge opportunity to innovate is going to be too sexy to pass up, and in the current white-hot technology market, some things are too obvious to fail.  So this won't fail.  If anything, I think we'll head towards some form of auto-configuration where just bringing a smart expresso machine into the building triggers the protocol to securely register it (and control it) from the building management system.

For truth in advertising; I actually have investments in this space -- I own plenty of shares of Microsoft stock, and I'm also an investor and advisor to Caspar.ai, a smart homes venture.  That's the one that got me thinking about the "leave no sensitive data behind" model for cloud computing, discussed earlier in January on this blog.  And while I don't happen to have any money in IronStack, I'm very loyal to my past students.  So I've got some biases here!

And yet honestly, I don't think this is a biased blog.  My bets are on this stuff because I truly believe that it has huge positive potential, even if it also represents a short-term risk for organizations that haven't bothered to think it through and are operating with insecure environments.  Worried that someone might be spying on your corporate meetings?  Well, have you thought about what devices might be in the room?  Maybe it is about time to secure them, with something like the Azure IoT Hub.

What of the huge potential?  Well, I have elderly relatives and I like the idea of a friendly little home ghost that can keep an eye on things, making sure they haven't fallen, that the stove was turned off when they left to go shopping, and that the windows are closed and latched at night.  I work in a smart building, and I like the feeling that we're being energy-smart and that the water won't somehow be left running in some unattended sink without the custodian eventually being notified.  These are good ideas.

A smart world with smart homes, smart offices, smart highways (if you've followed my blog, you would know that I have a bit of thing about smart cars... a worry that smart highways can address), smart cities, smart grid.  These are the technologies of the future.  And the Azure IoT Hub strikes me as the ideal way to start.  Which is why, in the spring 2019 offering of cloud computing, we've spent quite so much time on this model.  My slides are online, by the way, if you want to see what all this translates to in practice.