Fog computing and cloud-hosted data concentrators

When a cloud touches the ground, we get fog, an image that inspires a new buzzword: "fog computing."  If you are interested in cloud integration with edge devices you work in the fog!

Internet of Things is going to a big deal, and this suggests that as a research topic, fog computing deserves close scrutiny.  Today's most interest question is more of a meta-question: figuring out which system elements should play which roles (once this question is resolved, a series of more technical follow-on questions would arise). 

The question isn't really so new: At least since the 1980's, researchers have speculated about the challenges of computing systems that might reach very broadly into the physical world through a variety of sensing modalities (still and video imaging, radar and lidar and IR motion detectors, temperature and humidity sensors, microphones, etc), use machine-learned techniques to make sense of those inputs, and then perhaps take action.  But the game changer relative to those early days is the contemporary appreciation of how cloud computing can drive costs down, and enable a new form of nimble startup -- the kind of company that builds a cool new technology and then can scale it to support millions of users almost overnight.  Past versions of edge systems mired in issues of cost and suffered market failures: the users who might have benefitted didn't want to pay, and because the costs were genuinely steep, the technology simply didn't take off.

Today, we can at least identify a strong market pull.  In any technology media publication you read about smart homes and office complexes and cities, smart highways and power grids, smart healthcare technologies.  Clearly there is a wave of interest for this form of deeply integrated machine intelligence.  Moreover, and this points to a social shift relative to the 1980's, the dominant tone isn't really a worry about privacy, although you do see some articles that fret about the risks (and they are quite real; we need to acknowledge and engage on that front).  But the more dominant tone looks at upside, drawing analogies to our cell phones.

For example, I have an advisory role in a company called Caspar.ai.  It was founded by Ashutosh Saxena, a friend who was a Cornell faculty member until he left to do the startup, and David Cheriton, who you may know of as the first outside investor in Google (he was a Stanford professor but also a very successful entrepreneur, and when Larry and Sergie knocked on his door with their idea and an early proof of concept, Dave jumped to help them get established).  They view the apartment or condominium as the next platform, and Ashutosh actually gives talks in which he shows you the architecture diagram of an iPhone and then a nearly identical one for a condo in Santa Rosa.  The Caspar.ai system is like the iPhone O/S and could host apps, and because Caspar works with the developer who built the entire development, the technology can orient itself: it is in such-and-such a room listening to the apartment owner giving instructions about music for the party tonight, etc.

The example highlights one of the puzzles we'll want to think about: Caspar has orientation because it is built right into the condominium or home.  But most fog computing devices today are just small gadgets that individuals buy and install by hand.  How would they know where they are, which way they are pointing, etc?  And even if a very geeky technology person could configure such a thing, could her grandfather do it, or would he need to call his granddaughter to come and set the device up?  Part of the fog computing puzzle is visible right in this example, and in the contrast with Caspar.ai: how will these systems orient themselves, and how will the devices be told what roles to play?

It isn't just about smart homes.  One of the more exciting ideas I heard about most recently centers on smart agriculture: I attended a workshop on digital agriculture at Cornell a few weeks ago, and completely coincidentally, was invited to attend a second one on the concept of a smart soil "macroscope" at Chicago almost immediate afterward.

So how would the fog impact agriculture or dive into the soil?  Researchers spoke about tracking produce from farm to table, literally step by step, and using that knowledge to reduce loss due to fresh produce sitting on the shelf for too long, to improve the efficiency of the produce supply chain, prevent accidental contamination by E-Coli or other bacteria, redirect shipments to match demand more closely, etc.  A researcher at Microsoft showed that with a new kind of white-fi communications, drones could fly over fields and map out insect infestations in real-time, enabling the farmer to stamp out the pests with spot applications of insecticides, reducing unnecessary pesticides by a factor of 1000x.  You could do the same with fertilizer, or when watering.   One person talked about underground WiFi: it turns out to work surprisingly well, if you have enough power!  Who would have imagined an underground WiFi network?  But if the need ever becomes real enough, it can be done!  The one caveat is that they need fairly well-drained soil; pooled water can block the signals.

Who needs this?  I found one answer out in Napa.  I happen to love great wines, and I'm friendly with some growers who own or manage insanely expensive vineyards.   They would love to be able to visualize and track the subsurface biome, the movement of nutrients and water, and the conversion of surface materials into soil.  This might help a grower identify particularly promising spots to place the next great winery.  Of course, they are also quick to point out that no technology is a complete answer to any question, and that going from data to "useful insights" is quite a complicated matter.  But in a world of constant climatic change, they are keenly interested in knowing what is going on down there.  In fact I'm thinking I should start a little company to work on this topic, if for no other reason than as an excuse to visit and taste some of those wines!  A lot of them are way to expensive for me to actually buy and drink on a routine basis.

Getting technical again: what questions can we identify from this set of examples, and how do they shape the likely form a fog computing system might take?

Part of the puzzle centers on the limitations of sensing devices.  Devices are gaining in potential compute power, but you need to ask whether computing on the device itself is a smart move, given that more and more sensing devices are designed to operate on batteries, or with minimal power draw.  Computing at the edge isn't a very power-efficient model, and relaying data back to a data center has overwhelmingly dominated when you look at actual deployed IoT products. 

In fact there is much to be said for viewing sensors as dumb devices that might store a few hours or days of data, but don't compute very much.  First, if you want to understand an image or a spoken command, the size of database you would use to do that is huge -- deep neural networks and Bayesian networks generate models that can be terabytes or even petabytes in size when used for tasks of this kind.  Keeping the neural network models and precomputed data back in your local cloud where they can be shared among a number of users is far more cost-effective than shipping those petabytes to the devices, and then needing to keep those updated as the runtime conditions and goals evolve.

The proper way to process data might also depend on things the sensor is unlikely to have access to, such as the best estimate of its location and orientation, knowledge of which people are in the home or office, context associated with their prior commands to the system that were given in a different location, and captured by some other device.  As we saw, while Caspar.ai might actually have this kind of orientation at hand, most devices lack that sort of context information (think of an Alexa camera/microphone/speaker that people shift around much like a flower vase: it isn't in the same place from day to day, and that camera could easily end up pointing at a wall, or a stack of books!)   All of this might argue for a sensor model in which sensors capture everything in the vicinity, store a copy locally, but then just blindly relay the acquired data to the cloud.  The sensor could still do some very basic stuff: for example, perhaps it can figure out that absolutely nothing is happening at all, and skip the upload in that case, or upload a tiny marker saying "no change."  This really limited form of local computing is something that even very simple, disoriented devices can perform.

However, arguing in the other direction, there are sensing options that only make sense if deployed at the edge.  For example, you can't easily correct for focus on vibration of a video after capturing it, so that form of dynamic adjustment should be performed right on the camera.   A subsurface sensor used to track humidity in the soil may need to dynamically vary the way it operates its sensing components, because the best options for measuring moisture vary enormously depending on the depth of the water table, how far from fully saturated the soil is, etc.  So for cases like these, a dumb sensor might end up generating endless streams of low-quality data that can't be patched up later.

Broadly, I think we'll need to do both, but that the sensors will be used mostly in pretty dumb ways (like to hold video, and to discard empty content), but then will relay most of the potentially interesting stuff back to the cloud.

So this starts to answer the meta question.  Given this model, we can see what the technical need might be: the model argues that we should create a new technology base focused on cloud-hosted data concentrators that are integrated deeply into cloud storage systems: I'm fond of the term "smart memory" for this functionality.  A single instance of a concentrator, on some single mid-range compute server within the cloud, might handle some large but not unbounded number of sensors: perhaps, 10,000 smart homes in some community, or 100 video cameras.  If you need more capacity, you would just spread your incoming data streams over more data concentrators.

Notice that I'm not working within a standard 3-tier cloud model in this example.  A standard cloud has a first tier that generates web content, often using cached data and other second-tier services.  The third tier covers everything at the back end.  A concentrator is an example of a new kind of first-tier: one that is stateful and smart, perhaps with real-time guarantees and consistency promises.  This is not today's most common cloud model -- although it is close enough to it that today's cloud could definitely evolve to fit this picture, and in fact it might not even be a major reach to pull it off!

Within the data concentrator we would have a machine-learning model, dynamically updated and accurate, that could be used to immediately "understand" the data.  Thus if a person in an apartment utters a remark that only makes sense in context, we could create a dynamic machine-learned utterance model that is accurate to the second and expresses the system knowledge of recent past; even if the speaker is moving from room to room, that model would evolve to reflect the system understanding of his or her request.  For example, "Caspar, can you adjust the shade to get rid of that glare?" can be understood only by a system that knows the position of the sun, the locations of the windows and shades and the TV, and the options for adjusting that kind of window shade, but with that data, it can be done -- and that data would most likely live in a cloud-style concentrator or a smart-home "brain unit" if we put the concentrator right in the home (appealing as a response to privacy worries).  Tell Alexa or Siri to do this, and because those technologies are kind of a free-standing autonomous edge, the best you can hope for is a response like "I'm sorry, Ken, I don't know how to do that."

The other argument for cloud models leads to a data storage and update insight.  In particular, it isn't just the massive databases used for vision and speech understanding that would probably need to live on the cloud.  There is also a question of the smaller knowledge models used to make sense of edge events, and to adapt as they occur.

A smart highway might have an evolving understanding of the situation on such-and-such a segment of the freeway, and a smart farming system might build up a model of the insect infestation in a field, adapting the drone's flight plan to focus on areas that are at highest risk, while spending less time scanning areas that seem to be unaffected by the bugs.

The argument for smart storage would simply be that as we capture and make sense of these data streams, we're in a unique position to decide what to keep and what to discard, which data to route to back-end systems for further evaluation using offline techniques, etc.  The back-end systems would view the entire smart memory as a big storage cluster containing read-only files, reflecting the knowledge acquired by the smart analytic layer, and indexed by time.  Of course they could write files too, for example to update parameters of the knowledge model in use on the front end.

As an example, if a smart highway were to observe that some car is a few cm to the side relative to predictions, the system would probably just tweak the model parameters at the edge.  But if a car changes lanes unexpectedly, that would be a big event, and might be better handled by forwarding the information to a back-end system running Hadoop (Spark/Databricks), where we could recompute the entire set of expected vehicle trajectories for that segment of highway.

In other blog entries, I've shifted to a pitch for Derecho around this point, but this blog is getting long and perhaps I'll just wrap up.  In fact, it isn't just Derecho that I've omitted: I haven't even touched on the need for specialized hardware (FPGA, GPU clusters and TPU clusters seem like the most promising technologies for really understanding speech and video at scale), privacy, security, or data consistency: all could be topics for future blogs.  But those argue for a cloud model too.  Overall, it does strike me as a very promising area for study.  My one qualm, really, centers on the buzzword: fog computing isn't my favorite term; it sounds way too much like marketing drivel, and we've all heard a lot of that sort of thing.  What was wrong with plain old IoT?  Or "smart edge"?

Moving AI to the edge

In today's data centers, a dominant paradigm shapes machine learning and AI systems: massive amounts of data are collected, cleaned, and stored into some form of database or collection of files.  Then machine learning tools are used to train a model on the observed data, and finally the resulting model is used for a while in the edge application.  All of this takes time, so the edge applications operate using stale data, at least to some degree.

In online AI/Ml systems, like smart highways controlling smart cars, smart homes, or the smart power grid, a pervasive need for instant responsiveness based on the most current data is a distinguishing characteristic: today's standard cloud systems can definitely react to new events extremely rapidly (100ms or less is the usual goal), but because the edge runs on cached data -- in this situation, cached models -- and these platforms can't update their models promptly, they will continue to use a stale model long after something fundamental has changed, invalidating it. 

So why would we care about stale models?  The term model, as used by the ML community, refers to any concise representation of knowledge.  For example, on a highway, knowledge of a particular truck's behavior (its apparent route, history of speeds and lane changes, perhaps any observations of risks such as a tire that could be shredding, or a piece of cargo that might not be properly tied down) are all part of the model.  In a continuous learning setting, the model shapes behavior for all the smart cars in the vicinity.  In a smart power grid, the model is our estimate of the state of the grid; if the grid starts to show an oscillatory imbalance or signs of a shortage of power, the model can evolve in milliseconds, and the grid control algorithms need to adjust accordingly.  Yesterday's model, or even the one from ten seconds ago, might not be acceptable.

What will it take to move AI to the edge?

• The AI/ML community will need to figure out what aspects of their problems really need to run on the edge, and are incorrectly situated on the standard backend today.  Coding at the edge won't be the same as coding for the backend, so work will be required.  This suggests that many applications will need to be split into online and offline aspects, with the online parts kept as slim as possible.  The offline components will be easier to implement because they run in a more standard way.
• We need programming tools and platforms for edge-oriented AI/ML tools.  I'm betting that Derecho can be the basis of such a solution, but I also think it will take a while to reach that point.  We'll need to understand what edge-hosted AI/ML tools will actually look like: how will they represent, store, and access machine-learned models?  How big are these models, and what data rates arise?  Where are the low-latency paths, and how low does  latency need to be?
• We may need to integrate hardware accelerators into the infrastructure: if a system does vision, it probably wants to use GPU accelerators for image segmentation, tagging and for operations such as rotation, alignment, debluring, 3-D scene reconstruction, etc.  FPGA components offer a second rapid model, more focused on searching HTML text or other "byte stream" objects.  FPGA accelerators are also useful for doing quick cryptographic operations.  There could easily be other kinds of ASICs too: DFFT units, quantum thinkers, you name it. 
• All of these pieces need to synchronize properly and be easy to program in a "correct" way...

I'm fascinated by this whole area.  If it interests you too,  send me a note:  perhaps we can find ways to team up!

More thoughts on the learnability of implicit protocols

While driving from Sebastopol (in Sonoma valley, north of San Francisco) to Silicon Valley (which is south of San Francisco), one has a lot of time to think about how human-operated cars coordinate and implicitly communicate.  So I thought I might follow up on the posting I did a few weeks ago concerning the learnability of implicit protocols and see if I can't make the question a bit more concrete.

The best way to simplify a problem is to think of a base case, so I started by asking myself about the simplest model I could: vehicles modeled as points and a highway modeled as a straight line.

Interactions between vehicles would basically define a graph: any given car gets a chance to interact with the car in front of it, and the one behind it, and the protocol (whatever it might be) is like a state machine on these graphs.

So what behaviors would constitute a protocol in such a situation?  Well, we presumably would have car A in state Sa, and car B in state Sb, and then some "communication" takes place.  Not too many options: they either draw closer to one-another at some speed, move apart from one-another at some speed (I mean speed of closing or speed of separating, but probably the joint speed and direction of movement needs to be specified too), or perhaps one or the other brakes sharply or accelerates sharply.  So here we have a "vocabulary" for the cars, with which they communicate.

A protocol, then, would be a rule by which A reacts to inputs of this form from B, and vice versa, since any action by B on A is also perceived by B itself (that is, even when B probes A, B itself may be forced to take some reactive action too: the human version is that you are trying to shift from some horrendously bad radio station to a different, slightly less horrible, alternative when you look up and notice that you've gotten much closer to the car in front you than you expected.  So in this case it was your own darn fault, but just the same, you react!)

So to learn a protocol, we might imagine a series of "tests" by which vehicle B, approaching A from behind, experiments to investigate A's reaction to various inputs, recording these into a model that over time would converge towards B's approximation of A's algorithm.  A similarly is learning B's behavior (and A might elicit behaviors from B.  For example, sometimes a car comes up on my tail very close, and especially if we are already basically moving at the fastest possible speed, it annoys me enough that I gradually slow down, rather than speeding up, as he or she probably intended for me to do -- there is an optimal distance at which to follow me, if you are trying to do so.  Similarly for most drivers.  Drive 3x further back and I might ignore you completely.  Drive at half that distance and my behavior departs from the norm: I slow down, which is suboptimal for both of us.  My way of punishing your pushy behavior!).

If you could figure out how two cars can learn a protocol, then you could ask if this generalizes to situations that form trains of cars.  So here, B catches up with a train: not just A, but perhaps X-Y-Z-A, with A in the rear and some form of agreed-upon protocol in use by X,Y,Z and A.  B gets to join the train if it can learn the protocol and participate properly.  Failing to do so leaves B abandoned, or results in an accident, etc.

Optimal behavior, of course, maximizes speed and minimizes the risk of accidents.  I'm starting to see how one could add a kind of utility function here: B wants to learn A's protocol quickly and "efficiently", with as few tests as possible.  Then B wants to use the knowledge gained to achieve this highest possible speed "jointly" with A.  That defines a paired protocol, and the generalization becomes a train protocol.

Another observation: Suppose that B has a small set of models in a kind of grab-bag of models that cars commonly use.  Now B's job of learning the model is potentially easier: if A is using one of the standard models, it may only take a very small number of tests to figure that out.  Moreover, having a good guess might make some models "learnable" that would actually not be learnable with a small number of tests otherwise.  This would fit well with the experience I've had in New Jersey, Belgium, Tel Aviv and California: each country has its own norms for how closely cars tend to space themselves on high speed throughways and in other situations, so while you can feel disoriented at first, in fact this only adds up to four models.  If someone handed you those four models, I bet you could figure out which one applies with very few experiments.

Unless, of course, you accidentally end up behind a New Jersey driver who is learning the rules of the road in Tel Aviv for the first time, of course.  I think I've encountered a few such cases too.

So that would be a further interesting case to consider: roads with mixtures of models: subsets of cars, with each subset using a distinct model.  In fact my story of New York City cab drivers, from last time, would fall into that category.  The cabs figure out how to form school-of-fish packs that flow around obstacles and other cars, no matter what those other cars might be doing.  Meanwhile, there could easily be other protocols: New York public transit buses, for example, have their own rules (they basically are bigger than you, and can do whatever they like, and that is precisely how they behave).

Conversely, the New Jersey driver also illustrates a complication: the implicit protocols of interest to me are cooperative behaviors that maximize utility for those participating in the protocol, and yield higher utility than could be gained from a selfish behavior.  But any vehicle also will have an autonomous set of behaviors: a protocol too, but one that it engages in purely for its own mysterious goals.  And these selfish behaviors might not be at all optimal: I recently watched a teenage driver in the car behind me putting on makeup while talking on a cell phone and apparently texting as well.  So this particular driver wasn't maximizing any normal concept of utility!  And all of us have experienced drivers in the grips of road-rage, driving hyper aggressively, or teenage guys hot-dogging on motorcycles or weaving through traffic in their cars.  Their sense of utility is clearly very warped towards speed and has a very diminished negative utility for things like accidents, traffic violations and tickets, or sudden death.  So even as we learn to cooperate with vehicles prepared to cooperate, and need to figure out the protocols they are using, the environment is subject to heavy noise from these kinds of loners, inattentive drivers, reckless drivers, and the list goes on.

One thing that strikes me is that the cab driver protocol in New York, once you know it, is a good example of a very easily discovered policy.  Despite a huge level of non-compliant vehicles, anyone who knows the cab protocol and drives using it will quickly fall into synchrony with a pack of others doing so.  So the level of positive feedback must be massive, if we had the proper metric, relative to the level of noise. 

Interestingly, there is clearly a pack-like behavior even for a line of cars driving on a single lane.  This would be a very good case to start by understanding.

Of course the real-world-school-of-fish behaviors involve (1) shifting lanes, which is the easy case and (2) ignoring lane boundaries, which is a harder case.  Given that even the cars on a string case taxes my imaginative capabilities, I think I'll leave the full 2-dimensional generalization to the reader!

Will smart memory need an ACID model?

This is really part II of my posting on smart memory from ten days ago.

Let's imagine that we've created the ultimate data warehouse, using Derecho.  This warehouse hosts terabytes of persistent memory, It can absorb updates at a staggering rate: hundreds of gigabits per second, has tens of thousands of processors that crunch the data down and then store it as collections of higher level "knowledge models", and is always hungry for more.  

We used to think of data warehouses as respositories for nuggets of data, so perhaps we could call these "muggets:" models, that can now be queried.  Hopefully this term isn't painfully cute.

Next, we imagine some collection of machine learning applications that consume these muggets and learn from them, or compute some sort of response to them.  For example if the muggets represent knowledge collected on the local highway, the queries might be posed by smart cars trying to optimize their driving plans.  "Is it ok with everyone if I shift to the passing lane for a moment?"  "Does anyone know if there are obstacles on the other side of this big truck?"  "What's the driving history for this motorcycle approaching me: is this guy some sort of a daredevil who might swoop in front of me with inches to spare?"  "Is that refrigerator coming loose, the one strapped to the pickup truck way up ahead?"

If you drive enough, you realize that answers to such questions would be very important to a smart car!  Honestly, I could use help with such things now and then too, and my car is pretty dumb.

We clearly want to answer the queries with strong consistency (for a smart car, a quick but incorrect answer might not be helpful!), but also very rapidly, even if this entails using slightly stale data.  In Dercho, we have a new way to do this that adapts the FFFS snapshot approach described in our SOCC paper to run in what we call version vectors, which is how Derecho stores volatile and persistent data. Details will be forthcoming shortly, I promise.

Here's my question: Derecho's kind of data warehouse currently can't support the full ACID database style of computation, because at present, Derecho only has read-only consistent queries against its temporally precise, causally consistent snapshots.  So we have strong consistency for updates, which are totally ordered atomic actions against sets of version vectors, and strong consistency for read-only queries, but not read/write queries, where an application might read the current state of the data warehouse, compute on it, and then update it.  Is this a bad thing?

I've been turning the question over as I bike around on the bumpy, potholed roads here in Sebastopol, where we are finishing up my sabbatical (visiting my daughter, but I've also dropped in down in the valley and at Berkeley now and then).  Honestly, cyclists could use a little smart warehouse help too, around here!  I'm getting really paranoid about fast downhills with oak trees: the shadows invariably conceal massive threats!  But I digress...

The argument that Derecho's time-lagged model suffices is roughly as follows: ACID databases are  hard to scale, as Jim Gray observed in his lovely paper on "Dangers of Database Scalability".  Basically, the standard model slows down as n^5 (n is the number of nodes running the system).  This observation gave us CAP and BASE and ultimately, today's wonderfully scalable key-value stores and noSQL databases.  But those have very weak consistency guarantees.

Our Derecho warehouse, sketched above (fleshed out in the TOCS paper we are just about to submit) gets a little further. Derecho can work quite well for that smart highway or similar purposes, especially if we keep the latency low enough.  Sure, queries will only be able to access the state as of perhaps 100ms in the past, because the incoming database pipeline is busy computing on the more current state.  But this isn't so terrible.

So the question we are left with is this: for machine learning in IoT settings, or similar online systems, are there compelling use cases that actually need the full ACID model?  Or can Maine learning systems always manage with a big "nearly synchronous" warehouse, running with strong consistency but 100ms or so lagged relative to the current state of the world?  Is there some important class of control systems or applications that can probably be ruled out by that limitation?  I really want a proof, if so: show me that omitting the fully ACID behavior will be a huge mistake, and convince me with mathematics, not handwaving... (facts, not alt-facts).

I'm leaning towards the "Derecho would suffice" answer.  But very curious to hear other thoughts...

Will smart memory be the next big thing?

We're fast approaching a new era of online machine learning and online machine-learned behavior: platforms that will capture real-time data streams at high data rates, process the information instantly, and then act on the output of the processing stage.  A good example would be a smart highway that tells cars what to expect up around the next curve, or a real-time television feed integrated with social networking tools (like a world-cup soccer broadcast that lets the viewers control camera angles while their friends join from a remote location).

If you ask how such systems will need to be structured, part of the story is familiar: as cloud-hosted services capturing data from sensors of various kinds (I'm including video cameras here), crunching on it, then initiating actions. 

A great example of such a service is Microsoft's Cosmos data farm.  This isn't a research platform but there have been talks on it at various forums.  The company organized a large number of storage and compute nodes (hundreds of thousands) into a data warehouse that absorbs incoming objects,  then replicates them onto a few SSD storage units (generally three replicas per object, with the replication pattern fairly randomized to smooth loads, but in such a way that the replicas are on fault-independent machines: ones that live in different parts of the warehouse and are therefore unlikely to crash simultaneously).

Once the data is securely stored, Cosmos computes on it: it might compress or resize an image or video, or it could deduplicate, or run an image segmentation program.  This yields intermediary results, which it also replicates, stores, and then might further process: perhaps, given the segmented image, it could run a face recognition program to automatically tag people in a photo.  Eventually, the useful data is served back to the front-end for actions (like the smart highway that tells cars what to do).  Cold but valuable data is stored to a massive backend storage system, like Microsoft's Pelican shingled storage server.  Unneeded intermediary data is deleted to make room for new inputs.

Thus Cosmos has a lot in common with Spark, the famous data processing platform, except on a much larger scale, and with an emphasis on a pipeline of transformations rather than on MapReduce.

If we step way back, we can start to perceive Cosmos as an example of a smart memory system: it remembers things, and also can think about them (process them), and could potentially query them too, although as far as I know Cosmos and Spark have limited interactive database query functionality.  But you could easily imagine a massive storage system of this kind with an SQL front-end, and with some form of internal schema, dynamically managed secondary index structures, etc.

With such a capability, a program could interrogate the memory even as new data is received and stored into it.  With Cornell's Derecho system, the data capture and storage steps can be an asynchronous pipeline that would still guarantee consistency.  Then, because Derecho stores data into version vectors, queries can run asynchronously too, by accessing specific versions or data at specific times.  It seems to me that the temporal style of indexing is particularly powerful.

The interesting mix here is massive parallelism, and massive amounts of storage, with strong consistency... and it is especially interesting that in Derecho, the data is all moved asynchronously using RDMA transfers.  Nobody has to wait for anything, and queries can be done in parallel.

Tomorrow's machine learning systems will surely need this kind of smart memory, so for those of us working in systems, I would say that smart memory architectures jump out as a very exciting next topic to explore.  How should such a system be organized, and what compute model should it support? As you've gathered, I think Cosmos is already pretty much on the right track (and that Derecho can support this, even better than Cosmos does).  How can we query it? (Again, my comment about SQL wasn't haphazard: I would bet that we want this thing to look like a database).  How efficiently can it use the next general of memory hardware: 3-D XPoint, phase-change storage, high-density RAID-style SSD, RDMA and GenZ communication fabrics?

Check my web page in a few years: I'll let you know what we come up with at Cornell...

Implicit protocols and dynamic protocol discovery

During the past ten days I've driven in the area around Tel Aviv, then Brussels, then New Jersey/New York City.  The rules of the road are widely different in each of these areas.

Beyond the actual driving laws, there is an interesting kind of social networking at play, and it seems to suggest a possible research topic... one outside of my immediate area (anyhow, Derecho has me busy!), so I'll share it here in the hope that someone else might find the thought useful.

To set the stage, let me describe a few examples of what I'll call "implicit protocols" that drivers employ in these different areas.  Even before giving examples, I should explain what I mean with this term.  I have in mind situations where some actual communication occurs between drivers: we might look at each other and I might nod, meaning that I'm letting you pull in front of me on a crowded street.  You could drift slightly towards my lane, and I might acknowledge that by slowing my car, just a hair, and this is enough for you to realize that you have my permission to shift lanes.  The list goes on, and becomes pretty long.  So, an implicit driving protocol is (1) a behavioral rule, familiar to the people who drive in the region, and (2) one that isn't automatically used; it has to somehow be requested, and acknowledged.

So here are some examples:

Example 1: In Israel, as far as I can tell, drivers almost never look behind themselves when shifting lanes (they do check in other situations, I'm talking her specifically about lane-to-lane movement). They seem to check behind them normally, but once they decide that a lane-change is needed, the decision of when to do that apparently is based on the 180 degree region sideways and in front,  Once you get used to this you can easily understand intentions, but if you don't realize that they never check behind themselves, Israeli drivers often seem to be cutting you off because they will shift into your lane with inches to spare.  The key point is that you, as the driver coming up from the rear, were supposed to realize "long ago" that the driver up ahead was planning to shift into your lane and leave room for that.   If a driver seems to intend to change lanes and they don't (even with just inches to spare, as I said), you as the driver behind them are allowed to get impatient and honk the horn: "get on with it!".  And then conversely, if they shift lanes and nearly crash into you, they will be angry at you if they think the rules made it quite clear that this is what they were doing.  "You idiot, are you driving with your eyes closed?!!  Anyone could see what I was doing!"  They don't actually say stuff like that, by the way.  I just mean that they are thinking it.

In fact the first time you drive in Tel Aviv you feel as if people break the law whenever they find it convenient to do so.  But later you realize they actually have social rules for when to defy the laws, and even the city police respect those.  Break the rules in a way that they don't normally consider appropriate, do something nobody should ever do, and I promise you: every driver in blocks around will lean on his or her horn instantly, and you just might find that the local policeman wants to chat, too!  (There is also a whole protocol around weird rude behavior, like stopping your car in a way the blocks the entire street and then hanging out and smoking a cigarette because you are waiting for your cousin to come down from the fifth floor, but she's elderly and slow and it might take five minutes, during which every single car will be honking non-stop...  because after all, if you don't wait for her right at that spot, well, you might have to drive around the block, or she might have to stand there...  but let's not even go there!)

Example 2: In Belgium, the whole country uses a rule called "Priority a Droit".  You didn't know it, but probably you are familiar with priority-to-the-left rule, and a good way to understand it is to start by thinking about a round-about (a traffic circle).  Suppose someone wants to get into the circle, and you are driving around the circle.  Priority to the left is the usual rule: the driver in the circle has the right to keep driving, and the one who wants to enter has to wait for a free slot.  As you can imagine, the traffic circle deadlocks with priority to the right ("a droit" in French) because new cars can enter in priority over cars wishing to exit.  (Fun fact: for historical reasons rooted deep in the past, France has two major traffic circles that actually use priority a droit, namely the ones at the Etoile and at Place Victor Hugo.  I've seen them get deadlocked and stay that way for hours, literally hours.  The identical cars just sit there... forever.  Very interesting for me as a computer scientist!)

I'll toss out an example 2-a but this is a minor one: They have a ton of traffic circles, in fact, all over Europe.  And there often are two lanes.  How will they be used?  Well, the rule is to use the inner lane if you will exit two or more exits from here, but be in the outer lane if this next exit is where you will leave the circle.  All the drivers let you do this, if you know the rule and follow the rule.  But tourists often get it wrong, and cause fend-benders.

But enough about traffic circles.  Let's get to the real example 2.  So, here's the actual issue for Belgium.  It turns out that in Belgium, a great many small streets lack any kind of stop sign or traffic lights on the corners.  So you are driving on a street, maybe even a nice large road, 2 lanes in each direction, and in comes this dinky little street from the right with no markings on the intersection, and in fact the little thing could almost be a driveway, it looks so dubious as far as streetness goes. 

In the US, without question, you would assume that you on the main road have priority over the little road coming in from the right.  And actually, legally, you do have priority: we use a priority to the left rule in this situation, like on normal traffic circles.  So when an intersection is unmarked, the car to the left gets to drive directly through, and the car to the right must stop, check that the left is clear, and only then can advance into the intersection.

In Belgium, the rule is actually the opposite: dinky or not, the driver on that little alley has priority over the vehicles on the main 2-lane road if the intersection is unmarked, unless they tell you that you are on a "route prioritaire", which literally means "a road with priority over incoming traffic".  So everyone on the main road is guaranteed to tap the brakes at every intersection.  As the driver behind such a car you could crash right into them by not realizing they are forced to do this, and for that matter if you were that clueless, you could easily drive through without checking to the right, causing a major accident.  As it happens, this rule also gets used in the south of Europe quite often, so do make sure to understand it if you plan to drive there.

Where is the implicit protocol?  Partly, of course, the protocol is the legal rule.  But beyond that there is also this question of whether you are too far into the intersection to slow down and yield.  So there is still an exchange of context between you and the other driver that takes place: a form of communication that might be eye to eye, or might relate to how you drive your car.  But information is conveyed and helps you know if the guy coming in is ok with you passing even though he has priority, or if he is unaware of you, or deeply committed, in which case, slam on your brakes my friend!

Example 3:  In the area around New York and especially in the city itself, cars collaborate "school of fish" style to maximize throughput on streets that happen to have slightly coordinated traffic lights, such as the main north/south avenues that run for miles from the top of Manhattan to the bottom.  So on these, all the drivers are implicitly in agreement that the plan is to get as far as possible before stopping for the next red light, and with this in mind, they need to shift lanes and otherwise coordinate much like fish in a school of fish.  The car density might be very high, and yet the cars try to flow around obstacles, which are unfortunately common in New York City: gaping holes in the pavement where work on a steam pipe is underway, perhaps with fencing around the hole, perhaps just a big hole; steel plates that look unstable, maybe a homeless person pushing a shopping cart of stuff across the street right at that moment.  The cars just all flow, seamlessly. 

When you understand this it feels like a kind of dancing: a ballet of cars, and especially of taxi cars because there are additional complications: the taxi drivers all drive "correctly" but people from out of town on these roads are treated kind of like potholes: the taxis are very wary around them because they know that those drivers are clueless and might do stupid (or inefficient) things.  But there is a form of implicit communication here too: very quickly, I find that I'm in sync with the taxi and other drivers around me: I'm not in a yellow cab, yet the school of cabs accepts me.   It isn't an eye-to-eye thing: it just comes to the rules you are following. 

I could definitely extend this list: Switzerland has its own style (absurdly, excessively, polite and respectful of the rules, plus: "tous qui n'est pas interdit est obligatoire", meaning that "If it isn't illegal, it is obligatory").  California has its own style.  The style in Mexico can be rather creative. 

The computer science puzzle:  Ok, hopefully we're on the same page now.  As you can see, by driving in all of these different settings in a short amount of time, I've had to repeatedly adapt my meta-driving-protocol: I've had to shift to different implicit protocols, so that I would behave as expected from the point of view of all those others on the road.  Make the shift and you drive more safely and efficiently; fail to make it, and horns blare at you from all directions!

But how am I doing this?  And how did I figure out the rules in Tel Aviv, which is a new environment for me?  Or other such places?

I'll postulate that first of all, humans have explicit and implicit ways to communicate.  We talk, but we also communicate eye-to-eye through glances and little nods, and through behavior like when your car drifts just a little to the right or left while in motion in a way that "makes sense" under the local implicit driving rules.  So we send information through a dozen little "tells", as a poker player might express the idea, and this is often a two-way thing or even an n-way thing, as in the case of schools of drivers on heading south on 2nd Avenue in New York.

Next, the rules that matter on roadways are discoverable: a person who drives well and doesn't get stressed easily can learn these rules very quickly.  How?  I have no idea: I'm not adept at languages (I know like five words of Hebrew by now, a bit less than two words per month for the past three months: not impressive!)  But somehow these implicit protocols are so evident that they are just totally obvious once you settle into the situation, and you develop hypotheses and quickly validate them.

The means of dynamic discovery is also incredibly robust: we can learn these rules even though perhaps half or more of the drivers aren't using them at all, or not using them properly.  We can learn which cars "get it" and which ones don't, and then we as a school of cars can all cooperate to shun the ones that are clueless.  In fact you can figure out that the guy driving that SUV is clueless, and even though you are way up ahead of me, I'll realize this too, and will route around the SUV too, because I've learned something from you.  And yet I can't see you, don't know you.  Somehow you communicate this knowledge through your behavior!  And I somehow deduce it from that behavior.

Why does all this matter?  Here's why.  First, I think this represents a kind of exciting topic for research in social networks.  I bet that if you plan to come to do a PhD at Cornell and knock on Jon Kleinberg's door, you could even talk him into working on this with you.  (The topic is more his kind of thing than mine).  So do that!  We always need more brilliant PhD students.

And the second reason is that self-driving cars will never be safe until we understand this aspect of human behavior.  Obviously, if you follow these blog postings of mine, you'll know that I've become very worried about self-driving cars (check out that tag if you missed my prior comments on this).  But setting that stuff to the side, the situation we have is that Uber and others want to create self-driving taxis that will be safe in New York, and in California too, and Tel Aviv, and Brussels.  Well, good luck to them on this if they don't figure this stuff out!  I myself will try to opt out for as long as possible.

And in case you are wondering, I'm still driving standard shifts, as much as possible.  Of course I can drive an automatic.  But I just don't trust those automatic shifting systems...

On systematic errors in complex computer programs

Society as a whole, including many computing professionals, seems to assume that mistakes made by unsupervised deep learning and other forms of machine-learning will be like Heisenbugs: rare, random, and controllable through a more professional code development and testing process. This belief underlies a growing tendency to trust computing systems that embody unknown (indeed, unknowable) machine-learned behaviors.

Why is this a concern?  

In many parts of computer science, there is a belief in "ultimate correctness": a perception that if we merely formalize our goals and our algorithms and successfully carry out a proof that our code accomplishes the goals, then we will have created a correct artifact.  If the goals covered a set of safety objectives, then the artifact should be safe to use, up to the quality of our specification.  Highly professional development practices and extensive testing strengthen our confidence; model-checking or other forms of machine-checked proofs that run on the real software carry this confidence to the point at which little more can be said, if you trust the specification.

Yet we also know how unrealistic such a perspective can be.  Typical computing systems depend upon tens or hundreds of millions of lines of complex code, spread over multiple modules, perhaps including firmware programs downloaded into the hardware.  All of this might then be deployed onto the cloud, and hence dependent on the Internet, and on the cloud data-center's thousands of computers. The normal behavior of such programs resides somewhere in the cross-product of the individual states of the component elements: an impossibly large space of possible configurations. 

While we can certainly debug systems through thorough testing, software is never even close to flawless: by some estimates, a bug might lurk in every few tens or hundreds of lines of logic.  The compiler and the operating system and various helper programs are probably buggy too.  Testing helps us gain confidence that these latent bugs are rarely exercised under production conditions, not that the code is genuinely correct.

Beyond what is testable we enter the realm of Heisen-behaviors: irreproducible oddities that can be impossible to explain: perhaps caused by unexpected scheduling effects, or by cosmic rays that flip bits, or by tiny hardware glitches.  The hardware itself is often better modelled as being probabilistic than deterministic: 1 + 1 certainly should equal 2, but perhaps sometimes the answer comes out as 0, or as 3.  Obviously, frequent problems of that sort would make a machine sufficiently unreliable that we would probably repair or replace it.  But an event happening perhaps once a week?  Such problems often pass unnoticed.

Thus, ultimate correctness is elusive; we know this, and we've accommodated to the limitations of computing systems by doing our best to fully specify systems, holding semi-adversarial code reviews aimed at finding design bugs, employing clean-room development practices, then red-team testing, then acceptance testing, then integration testing.  The process works surprisingly well, although patches and other upgrades commonly introduce new problems, and adapting a stable existing system to operate on new hardware or in a new setting can reveal surprising unnoticed issues that were concealed or even compensated for by the prior pattern of use, or by mechanisms that evolved specifically for that purpose.

The puzzle with deep learning and other forms of unsupervised or semi-supervised training is that we create systems that lack a true specification.  Instead, they have a self-defined behavioral goal: reinforcement learning trains a system to respond to situations resembling ones it has seen before by repeating whatever response dominated in the training data.  In effect: "when the traffic light turns yellow, drive very fast."

Thus we have a kind of autonomously-learned specification, and because the specification is extracted automatically by training against a data set, the learned model is inherently shaped by the content of the data set.

Train such a system on a language sample in which plurals always end in "s", and it won't realize that "cattle" and "calamari" are plural.  Train it on images in which all the terrorists have dark hair and complexions, and the system will learn that anyone with dark hair or skin is a potential threat.  Teach it to drive in California, where every intersection either has stop signs on one or both streets, or has a traffic signal, and it won't understand how to drive in Europe, where many regions use a "priority to the right" model, whereby incoming traffic (even from a small street) has priority over any traffic from the left (even if from a major road). 

Machine learning systems trained in this way conflate correlation with causation.  In contrast, human learning teases out causal explanations from examples.  The resulting knowledge is different from a knowledge model learned by training today's machine learning technologies, no matter how impressive the machine learning system's ability to do pattern matching.

Human knowledge also understands time, and understands that behavior must evolve over time.  Stephen Gould often wrote about being diagnosed as a young adult with a fatal circulatory cancer.  Medical statistics of the period gave him a life expectancy of no more than a few months, perhaps a year at best.  But as it happened, a new medication proved to be a true magic bullet: he was cured.   The large-population statistics were based on prior treatments and hence not predictive of the outcomes for those who received this new treatment.  The story resonated in Gould's case because in his academic life, he studied "punctuated equilibria", which are situations in which a population that has been relatively static suddenly evolves in dramatic ways: often, because of some significant change in the environment.  Which is precisely he point.

Those who fail to learn from the past are doomed to repeat it.  But those who fail to appreciate that the past may not predict the future are also doomed.  Genuine wisdom comes not from raw knowledge, but also from the ability to reason about novel situations in robust ways.

Machine learning  systems tend to learn a single set of models at a time.  They squeeze everything into a limited collection of models, which blurs information if the system lacks a needed category: "drives on the left", or "uses social networking apps".  Humans create models, revise models, and are constantly on the lookout for exceptions.  "Is  that really a  pile of  leaves, or has the cheetah realized it can hide in a pile of leaves?  It never did that before.  Clever cheetah!"   Such insights once were of life-or-death importance.

Today, a new element enters the mix: systematic error in which a system is programmed to learn a pattern, but overgeneralizes and consequently behaves incorrectly every time a situation arises that exercises the erroneous generalization.  Systematic error is counterintuitive, and perhaps this explains our seeming inability to recognize the risk: viewing artificially intelligent systems as mirrors of ourselves, we are blind to the idea that actually, they can exhibit bizarre and very non-random misbehavior.  Indeed, it is in the nature of this form of machine learning to misbehave in unintuitive  ways!

My concern is this: while we've learned to create robust solutions from somewhat unreliable components, little of what we know about reliability extends to this new world of machine-learning components that can embody systematic error, model inadequacies, or an inability to adapt and learn as conditions evolve.  This exposes us to a wide range of new failure modalities never before seen, and that could challenge the industry and the computer science community to overcome.  We lack systematic ways to recognize and respond to these new kinds of systematic flaws.

Systematic error also creates new and worrying attack surfaces that hackers and others might exploit.  Knowing how a machine learning system is trained, a terrorist might circulate some photoshopped images of him or herself with very pale makeup and light brown or blond hair, to bind other biometrics that are harder to change (like fingerprints, corneal patterns) with interpretations suggesting "not a threat".  Knowing how a self-driving car makes decisions, a hacker might trick it into driving into a pylon.

Welcome to the new world of complex systems with inadequate artificial intelligences.  The public places far too much confidence in these systems, and our research community has been far too complacent.  We need to open our eyes to the risks, and to teach the public about them, too.  

Big Infrastructure

It surprises me that for all the talk of big data, and the huge investments rushing to capture niches in the big data space, so little attention is given to the infrastructure that will need to move that data.

Today's internet is a mostly-wired infrastructure paid for by the owners of the end-point systems, be those consumers who have a wireless router in their living room, or consumers with mobile devices that connect wirelessly to the local cell phone operator.

It isn't at all clear to me that either model can work for tomorrow's world in which the Internet of Things will be capturing much of the data and taking many kinds of actions: a world of small devices that will often be installed in the relative "wilderness" of our cities, highways, buildings and other spaces. 

Consider a small sensor playing some sort of role in a smart building: perhaps it monitors temperature and humidity in a particular room.  This little device will need:

• A stable source of electrical power.  You may be thinking "batteries" and sure, that can work, but this assumes a consumer willing to replace those batteries once a year or so.  If our device wants to generate its own power, that could be possible.  I've seen work on harvesting ambient EM power from background signals, and of course our unit could also have some form of optical cell to convert light to electricity.  But until that question is solved, the IoT world will be a world of things you plug in, which already tells you something about "form factors" and roles.
• Some idea of where it is located.  If the sensor in the room is telling the house whether or not to run the A/C for that area, or the dehumidifier, clearly that sensor has to know which room it is located in.  Perhaps even which way it is pointing: if you want the lighting for guests on the couch to be just right for reading, or for whatever else they may be up to, you need to be pretty smart about the room.  A smart traffic light wants to know which intersection it is at, on which corner, and which way it is oriented relative to the streets.  Traffic sensors will want to know these things too, even when located in the middle of a block or on a pole high above a freeway.  A smart printer wants to know who is allowed to use it, and your phone wants to know which of the printers it can sense belong to your home, or to your friend's home if you happen to need to print something while hanging out elsewhere.
• Some idea of role.  Many sensors will be created from the same basic elements (sensors that can detect light, or take photos or video, or track moving things in the vicinity, or its own motion, or humidity, or sound, etc).  These general purpose units will be cheap to manufacture in huge volume, but then in general will need to be programmed to do some specific thing that may use just parts of the functionality available.
• Who's in charge.  Small devices will generally operate under control of more centralized ones, which in turn might talk to still large-scale controllers.  When you turn the device on, how can it figure this out?
• How to network back to the boss.  Even a tiny device may have many options available: Bluetooth, various WiFi options, cellular telephony options, you name it.  Which should it use?

Then we run into questions of the network side of the Internet of Things.  Will this really be the Internet, or will the IoT demand some other style of networking? 

For some uses, today's network probably can get us there.  For example, I think one of the biggest things that will be coming soon is delivery of TV-style live content, including various forms of group video conferencing, or live sports events.  Those applications center on moving bytes, and our Internet service providers are pretty motivated to enhance their networks and then sell us more and better bandwidth.  The technology to build data centers for replicating data rapidly inside the data center is in hand (Derecho!), and the existing last-hop options are probably fast enough, so we can get that video stream to the end user without a massive upgrade: it really could be done.  But TV turns out to be a fairly easy case.

Networking for little devices isn't at all clear. Here we run into an issue like the gigabit network last leg: something enormous needs to be built, yet it currently isn't monetized, even to a limited degree.

A student I've worked with is convinced that standard networking isn't really suitable in many of the environments that the Internet of Things will run in: he points out that the people who service traffic signals aren't networking experts, and yet when traffic lights evolve into smart signaling units, those same people will probably be the ones in charge of replacing broken devices.  We'll need a very different model for how the broken system reports its faults, prioritizes repairs, and how it communicates all this to the infrastructure owner and the repair crew.  Since failures and environmental challenges (like ice, snow, humidity, physical damage...) are going to be common in many settings, and some devices may be in fairly remote and isolated settings, he believes that repair won't be instantaneous either: we'll want networking redundancy, using ideas like duplicate links, RAID-style data coding across the various available network pathways, and fault-tolerant routing.

Security will be an issue too.  It is nice to imagine the benefits of moving towards a world of smart factories or smart transportation, but we'll want this world to have much stronger security than today's Internet (which probably wants much stronger security than it actually possesses!)   But quite likely the sensors in the Internet of Things will need built-in security keys, and the architecture will be such that a given device will only accept instructions from properly authenticated, properly authorized control units. We'll want to encrypt the data sent and received, both to protect against unauthorized spying, and to ensure that the received commands really originated at a trusted source.

And then all of this will need to connect to the cloud, since the whole point of the Internet of Things will be that by using "big data" tools to learn about environments, we can often optimize them in useful ways: to waste less power, for example, or personalize the environment so that a room is a bit cool if I'm it, but a bit warmer when my sister comes for a visit (she's used to a much warmer climate and finds the places I live most of the year frigid).  We might even imagine personalized advice: today is February 7 and I was swimming in the Mediterranean sea off Tel Aviv.  For me, the advice would be that the water was chilly but quite swimmable.  For my sister, same beach, and the message would be "deadly cold, don't even think about it!"  The cloud will be the entity in this picture that builds up the personalization story: Ken likes it kind of chilly; his sister prefers it a bit on the warm side.  So the Internet of Things will feed data to a cloud that in turn takes actions through both small actuators (like the traffic signal) but also through our mobile devices.

We're going to need a new technology for building and operating these systems: a new kind of "Big Infrastructure", more or less the physical counterpart of today's "Big Data" analytics and tools.  But surprisingly little work seems to be underway on creating that technology, or deploying it.

Here's the very biggest puzzle of all: who is going to pay?  In fact the real answer is obvious: in the end the consumer will pay, either directly in the sense of finding the technology so valuable that he or she voluntarily buys it, or indirectly, through municipal taxes and fees that pay for the smart traffic lights, or even by accepting personalized advertising.  Money is invariably the key: spend enough, and most problems can be solved, and the money tends to be where the majority of the people are to be found.

Viewed this way, it becomes clear why the Internet of Things revolution is stalled, at least for the time being.  The value of doing an Internet of Things is pretty clear.  Once it exists, it will be an immense value generator, so if you could solve that chicken-and-egg puzzle, you'll enable a very, very lucrative new world.  But until it exists, we have a version of the so-called last-mile problem: it is very easy to get gigabit networking to within a mile of everyone, maybe even to within 100 meters.  Then the costs simply explore for that last leg, and while we would see a tremendous burst of innovation if we had gigabit networking into our homes, the cost of bridging that last leg is a tremendous barrier to starting to reap the benefits of the faster technology.  Thus nobody actually has gigabit networking now, so the applications that would be enabled by that sort of development aren't yet being created: if you did so, nobody would buy them, because few people have a fast connection.

Earlier I said that live TV is coming, but that's because the bandwidth seems to be within the range that the existing infrastructure can manage.  With a gigabit technology we could do far more, except that as far as I know, nobody has yet figured how to upgrade that last leg to support it.  And the small devices version of the problem has this even greater puzzle of lacking an existing paying customer.

Where will the big infrastructure come from?  If you figure this out, let me know: I'll want to be an early investor. 

Infrastructures matter too

This posting is kind of a gripe, so I'll try to keep is short and more blog-like.

In my research area, which is broadly considered to be "systems" and more narrowly has names like fault-tolerance, distributed computing or cloud computing, funding used to be extremely generous.  I think that this was perhaps because the US military (DARPA, AFRL/AFOSR, etc) were finding it difficult to deal with their distributed infrastructures and wanted to catalyze change.  We had a great run, in many senses: systems research led to today's massive cloud computing platforms, gave us big leaps forward in terms of the performance of all kinds of systems, has given us genuinely exciting new security options, and enabled far higher productivity.  Our students (at every level: PhD, MS and MEng, even undergraduates) did projects that prepared them for jobs at the top companies.  Some actually went on to found those companies, or to lead their engineering sides.

So the investment paid off.

Even so, under the Bush/Cheney administration, there was a sudden and very striking shift.  First, security research flourished and expanded hugely.  Somewhat ironically, I actually led the initial studies that created DARPA's original security research programs, but that was back in 1995; I myself never became a real security researcher.  But be that as a it may, there was a huge increase in funding for work on security (by now we see the value of this: many great ideas emerged from all that investment). Today there are new kinds of trusted platform modules with really remarkable capabilities, which definitely trace to this big push to create new secure computing options. 

Meanwhile, systems research continued to receive attention, and that mattered because with the emergence of scaled-out systems for the cloud, we needed non-industry study of all the questions that aren't of immediate, "we need it yesterday" importance to companies like Microsoft and Google.  But there was an odd flavor to it: to do systems work in that period, you often needed a big brother: a company that somehow was viewed favorably by the Cheney crowd and the DARPA leadership.  Jokes were made about needing a "FOT project leader" ("Friend of Tony", for Tony Tether).  And many projects were cancelled abruptly or suspended for months pending reviews to make sure they were really innovative.  The FOT projects usually avoided those reviews; the non-FOT projects suffered.  Over at NSF, the Bush/Cheney administration imposed sharp cuts, which Congress then tended to reverse.  A very complex picture.

This was often justified by funding leaders who would explain that the US government shouldn't invest in IT research if industry will do the work anyhow, or if there isn't some unique need specific to the government, or the military.

In fact, you can count on industry to solve today's problems, and yesterday's problems.  But if your needs are a bit unusual (as in the critical computing sector, or the military, or hospitals), don't hold your breath.  Anything that isn't aligned with the largest market sectors will languish, and our job as academic researchers is to solve the special needs of those (important) areas, while also educating the students who go on to fill all the key roles in industry.

Fast forward eight years.  Bush and Cheney are now out, replaced by the Obama administration.  One might have expected research funding to flourish.

In fact, at least in my area (but I think more broadly), a major problem emerged instead.

As you perhaps know, President Obama and the Congress he had to work with simply didn't see eye to eye, on anything.  One side-effect of the disagreement is that big discretionary funding cuts were imposed: the so-called "sequester".  For the national research programs at DARPA and NSF and other large players, this all took place just as machine learning and AI began to have really dramatic payoff.  Yet at the government budgeting level, agency research budgets stopped growing and at best, slipped into a mode in which they barely tracked inflation.  We had what is called a "zero-sum" situation: one in which someone has to lose if someone else wins.  So as funding shifted towards security on the systems side, and towards AI and ML on the other side, money for systems research dried up.  This happened first in the database area, and then crept into all areas of systems with the possible exception of computer networks.  For whatever reason, networking research was much more favorably treated.  A new area called Cyberphysical Systems emerged and was funded fairly well, but perhaps this is because it has unusually effective leadership.  Still, in the large picture, we saw growth on the AI/ML side coupled with shrinkage in the areas corresponding to systems.

When I talk to people at DARPA or NSF, I get a very mixed and often, inaccurate story of how and why this happened, and how they perceive it in retrospect.  NSF program managers insist that their programs were and remain as healthy as ever, and that the issue is simply that they can't find program managers interested in running programs in systems areas. They also point to the robust health of the network research programs.  But such comments usually brush over the dearth of programs in other areas, like distributed and cloud computing, or databases, or other kinds of scaled-out systems for important roles, like controlling self-driving cars. 

NSF also has absurdly low success rates now for programs in these areas.  They run a big competition, then fund one proposal out of 25, and halve the budget.  This may seem like a success to the NSF program officers but honestly, when you get allocated one half a student per year for your research, and have to submit 5 proposals to see 1 funded, it doesn't look like such a wonderful model!

So that summarizes what I tend to hear from folks at NSF.  At DARPA one hears two kinds of messages.  One rather frequent "message" takes the form of blame.  Several years ago, DARPA hired a researcher named Peter Lee from CMU, and gave him a special opportunity: to create a new DARPA advanced research topics office.  This would have owned the whole area I'm talking about.

But when Rick Rachid announced retirement plans at Microsoft,  Peter took Rick's old role, and DARPA couldn't find anyone as strong as Peter to replace him.  After a bit of dithering around, DARPA folded those new programs into I2O, but I2O's main agenda is AI and ML, together with a general search for breakthrough opportunities in associated areas like security.  Maybe asking I2O to also tackle ambitious cloud computing needs is too much?  At any rate, after an initially strong set of programs in cloud computing, I2O backed away in the face of the sequester and funding was cut very deeply.

If you ask about this cut, specifically, you hear a second kind of message -- one that can sound cold and brutal.  People will tell you that the perception at DARPA is that today, the best systems research happens in industry now and that if DARPA can't create and shape the industry, then DARPA has little interest in investing in the research.  And they will tell you that whether NSF acknowledges this or not, the same view holds in the front offices of NSF, AFOSR, ONR, you name it.

Then you hear that same rather tired nonsensical claim: that there "are no program officers interested in these topics."  Obviously, there would be candidates if NSF and DARPA and DOE and so forth tried in a serious way to recruit them.  But the front offices of those agencies don't identify these areas as priorities, and don't make a show of looking for stars to lead major programs.  Given the downplayed opportunity, who would want the job?  A program manager wants to help shape the future of a field, not to babysit a shrinking area on the road to funding extinction.  So blaming the lack of program officers just ignores the obvious fact that the program officers are a mirror image of the priorities of the front office leadership.  Recruit and thou shall find.

Three reasons (one especially doubtful).  But you know what?  I think all three are just dumb and wrong.

One problem with this situation is that the industry needs top-quality employees, who need to be trained as students: trained by people like me in labs like mine.  Without research funding, this pipeline stalls, and over time, industry ends up with fewer and fewer young people.  The older ones become established, less active, and move towards managerial roles.  So innovation stalls.  As a result, the current priorities guarantee that systems platforms will stop evolving in really radical ways. Of course if systems platforms are boring, useless, unimportant -- well, then nobody should care.  But I would argue that systems platforms are core parts of everything that matters.  So of course we should care!  And if we strangle the pipeline of people to build those platforms, they are not going to evolve in exciting ways.

Second, innovation requires out of the box thinking, and under pressure to compete and be first to market with new cloud computing concepts or other new functionality, there is a furious focus on just a subset of topics.   So by starving academic research, we end up with great systems work on the topics industry prioritizes most urgently, but little attention to topics that aren't of near-term impact.  And you can see this concretely: when MSR shut down their Silicon Valley lab, the word most of us heard was that the work there was super, but that the degree of impact on the company was deemed to be fairly low.  MSR no longer had the resources, internally, to invest in speculative projects and was sending a message: have impact, or do your work elsewhere.  Google's employees hear that same message daily.  Honestly, almost all the big companies have this mindset.

So there is a sense in which the government pivot away from systems research, however it may have been intended, starves innovation in the systems area, broadly defined (so including distributed systems, databases, operating systems, etc).  All of us are spending more and more time chasing less and less funding.

The bottom line as I see it is this.  On the one hand, we all see merit in self-driving cars (I would add "... provided that they get lots of help from smart highways"), smart power grids, other kinds of smart city, building and infrastructures.  So all of this is for the good.

But when you look at these kinds of Internet of Things use cases, there is always a mix of needs: you need a platform to robustly (by which I mean securely, perhaps privately, fault-tolerantly) capture data.  You need the AI/ML system to think about this data and make decisions.  And then you need the platform to get those decisions back to the places where the actions occur.

This dynamic is being ignored: we are building the AI/ML systems as if only the self-driving car runs continuously.  All the cloud-hosted aspects are being designed using data in static files, ignoring the real-time dynamics and timing challenges.  The cloud platforms today are, if anything, designed to be slow and to pipeline their actions, trading staleness at the edge for higher transaction rates deep inside: a good tradeoff for Facebook, but less evident for a self-driving car control system!  And the needed research to create these needed platforms is just not occurring: not in industry, and not in academic settings either.

Even if you only care about AI/ML you actually should care about this last point.  Suppose that the "enemy" (maybe some kind of future belligerent country) has fighter planes with less effective AI/ML (they only use the open source stuff, so they run a few years behind us).  But their AI/ML isn't terrible (the open source stuff is pretty good).  And now suppose that they invested in those ignored platform areas and managed to get everything to run 10x or 100x faster, or to scale amazingly well, just by leveraging RDMA hardware, or something along those lines. 

So suppose now that we have an incoming Foo Fighter closing at Mach 4.5 and armed with last year's AI/ML, but running that code 100x faster.  And our coalition fighter has the cutting edge new AI/ML, maybe some kind of deep neural network trained from data collected from the best pilots in the USAF.  But the software platform is sluggish and doesn't even make use of the cutting edge NetFPGA cards and GPU accelerators because the 2010 version of Linux they run didn't support those kinds of things.  Who would you bet on?  I myself might bet on the Foo Fighter.

Slightly dumber AI on a blazingly fast hardware/software infrastructure beats AI++ on old crappy platforms.  This is just a reality that our funding leadership needs to come to grips with.  You can't run shiny new AI on rusty old platforms and expect magic.  But you definitely can take open source AI and run it on an amazing breakthrough platform and you might actually get real magic that way.

Platforms matter.  They really do.  Systems research is every bit as important as AI/ML research!

We have a new administration taking office now, and there will clearly be a shakeup in the DC research establishment.  Here's my hope: we need a bit of clear thinking about balance at NSF and DARPA (and at DOE/ARPAe, AFOSR, ONR, AFRL, you name it).  All of this investment in security and machine learning, is great, and should continue.  But that work shouldn't be funded at the expense of investment in platforms for the real-time Internet of Things, or new kinds of databases for mobile end-users (including mobile human users, mobile cars, mobile drones, mobile warfighters and police officers and firemen...).  We need these things, and the work simply takes money: money to do the research, and as a side-effect, to train the students who will later create the products.

I'm not casting a net for my own next job: I love what I do at Cornell and plan to continue to do it, even if research funding is in a drought right now.   I'm managing to scrape together enough funding to do what I do.  Anyhow, I don't plan to move to DC and fix this.  The real issue is priorities and funding allocations: we've been through a long period during which those priorities have gotten way out of balance, and the effect is to squeeze young talent out of the field entirely, to dry up the pipeline, and to kill innovation that departs from what industry happens to view as its top priorities. 

The issues will vanish if priorities align with the real needs and money is allocated by Congress.  Let's hope that with the changing of the guard comes a rebalancing of priorities!