LearningSys

Workshop on Machine Learning Systems

at Neural Information Processing Systems (NIPS)

December 12, 2015 - Room 511d

Speakers

Jeff Dean (Google)

TensorFlow Overview and Future Directions [Slides] [Video]
Over the past few years, we have built two large-scale computer systems for training neural networks, and then applied these systems to a wide variety of problems that have traditionally been very difficult for computers. We have made significant improvements in the state-of-the-art in many of these areas, and our software systems and algorithms have been used by dozens of different groups at Google to train state-of-the-art models for speech recognition, image recognition, various visual detection tasks, language modeling, language translation, and many other tasks. Our second-generation system, TensorFlow, has been designed and implemented based on what we have learned from building and using DistBelief, our first generation system. The TensorFlow API and an initial implementation was released as an open-source project in November, 2015 (see tensorflow.org). In this talk, I'll discuss the design and implementation of TensorFlow, and discuss some future directions for improving the system.
Bio:
Jeff joined Google in 1999 and is currently a Google Senior Fellow. He currently works in Google's Research division, where he co-founded and leads Google's deep learning research team in Mountain View. He has co-desiged/implemented multiple generations of Google's crawling, indexing, and query serving systems, and major pieces of Google's initial advertising and AdSense for Content systems. He is also a co-designer and co-implementor of Google's distributed computing infrastructure, including the MapReduce, BigTable, Spanner, DistBelief and TensorFlow systems, protocol buffers, LevelDB, systems infrastructure for statistical machine translation, and a variety of internal and external libraries and developer tools. He received a Ph.D. in Computer Science from the University of Washington in 1996, working with Craig Chambers on compiler techniques for object-oriented languages. He is a Fellow of the ACM, a Fellow of the AAAS, a member of the U.S. National Academy of Engineering, and a recipient of the Mark Weiser Award and the ACM-Infosys Foundation Award in the Computing Sciences.

Joseph Gonzalez (UC Berkeley)

Intelligent Services: Serving Machine Learning [Slides] [Video]
Machine learning has become the key component in building intelligent services and applications. However, much of the focus in machine learning research and software systems is aimed at the design of increasingly sophisticated models, training algorithms, and large-scale systems for model training. In this talk I will discuss the other and perhaps more important challenge machine learning systems: serving and maintaining trained models. I will describe the machine learning life-cycle that spans beyond the fiction of training an isolated model on static data, to learning on changing data, competing models developed by multiple teams, and versioning. I will characterize some of the unique challenges and opportunities of model serving in a low-latency and high-availability setting. I will present mechanisms to enable models to compete and learn from feedback in real-time and discuss the opportunity to trade-off latency and accuracy. I will present this work in the context of two systems, one being developed at UC Berkeley and another already in production at Dato Inc.
Bio:
Joseph Gonzalez an assistant professor at UC Berkeley and co-founder of Dato Inc. Joseph holds a PhD in Machine Learning from CMU where he created the GraphLab open-source graph processing system and a collection of tools for Graphical Model inference. Joseph is part of the UC Berkeley AMPLab where he created, GraphX, the graph analytics framework in Apache Spark.

Sarah Bird (Microsoft Research)

Multiworld Testing: Contextual Bandits as a Service [Video]
There is a growing demand in today’s products and services to provide experiences that are highly personalized and relevant to the user’s situation at that moment. However, optimizing models for contextual decisions can often be challenging and error-prone. In this talk, I present the Multiworld Testing Decision Service, an Azure cloud service that we have designed and deployed to optimize contextual decisions as a service for applications. Based on cutting-edge contextual bandits research, our system can manage contextual decisions for applications in real-time using experimentation and online learning to rapidly respond to observed outcomes and optimize model performance. We have deployed multiworld testing in several Microsoft products and services, for example, to make personalized news recommendations on the MSN homepage. In our MSN deployment, the decision service is making hundreds of millions of recommendations per day and adapting the model to the most recent clicks every few minutes. As a result, we have increased clicks on the optimized portions of the homepage more than 20% over the editorially managed version. The Multiworld Testing Decision Service learning components are open-source, and the service is available for external use.
Bio:
Sarah is postdoctoral researcher at Microsoft Research NYC. Her background is in systems research and her current research focuses on problems at the intersection of systems and machine learning. In particular, she is working on systems for interactive and online machine learning and on designing systems that can be controlled and optimized with learning algorithms. Sarah did her Ph.D. in computer science at UC Berkeley’s Parallel Computing Laboratory (ParLab) advised by Krste Asanovic and David Patterson at Berkeley and Burton Smith at Microsoft Research. She has B.S. in Electrical Engineering (Computer Engineering) from the University of Texas at Austin.

Mathias Brandewinder (Clear Lines Consulting)

Functional Programming and Machine Learning: a natural fit [Video]

In the past decade, Functional Programming has evolved from a niche specialty into the mainstream. Modern programming languages such as F# or Scala have broadened adoption by demonstrating the practical benefits of functional techniques, and these ideas are progressively being incorporated into most traditional, well-established languages, such as C++ or Java.

In this talk, we will discuss specifically why Functional Programming and Machine Learning are a great fit. Using live coding in F#, on real-world examples, we will demonstrate how classic algorithms can be expressed very clearly - and correctly - in a functional style, and how functional idioms such as pipes-and-filters enable effective data exploration. Finally, we will show how functional programming lends itself naturally to parallelization, allowing for large-scale data processing.

Bio:
Mathias Brandewinder has been developing software professionally for about 10 years, with a focus on forecasting and risk analysis models. He is a Microsoft F# MVP, and speaks regularly on functional programming and related topics at conferences worldwide. He is the author of "Machine Learning Projects for .NET Developers" (Apress), and the founder of Clear Lines Consulting. Mathias is based in San Francisco, blogs at www.clear-lines.com/blog, and can be found on Twitter as @brandewinder. He holds degrees in Business from ESSEC, Economics from University Paris X, and Operations Research from Stanford University.

Roxana Geambasu (Columbia University)

Privacy in a Data-Driven World [Video]
The concept of personal privacy as a precious and fragile commodity worthy of protection has come under siege in today's data-driven world. Users are eager to share their data online, and mobile applications and web services aggressively collect and monetize that information. This talk describes our vision for a new, privacy-preserving world; in it, users are more aware of the privacy implications of their online actions, and systems and applications are designed from the ground up with privacy in mind. In support of this vision, we describe our research agenda to design, build, and evaluate new transparency tools that increase users' and privacy watchdogs' visibility into how personal data is being used by applications, and programming abstractions and tools that facilitate the construction of privacy-mindful applications. We provide two examples of such tools and abstractions. First, we describe Sunlight, a new web transparency tool that helps privacy watchdogs track how web services use individuals' personal data to target ads, personalize content, or adjust prices. Second, we describe FairTest, a new testing toolkit that helps programmers test for unfair or discriminatory effects within their data-driven applications. Overall, our tools and abstractions aim to increase privacy by promoting a more responsible, fair, and accountable approach to user data management.
Bio:
Roxana Geambasu is an Assistant Professor of Computer Science at Columbia University. She joined Columbia in Fall 2011 after finishing her Ph.D. at the University of Washington. For her work in cloud and mobile data privacy, she received an Early Career Award in Cybersecurity from the University of Washington Center for Academic Excellence, a Microsoft Research Faculty Fellowship, a 2014 "Brilliant 10" Popular Science nomination, an NSF CAREER award, an Honorable Mention for the 2013 inaugural Dennis M. Ritchie Doctoral Dissertation Award, a William Chan Dissertation Award, two best paper awards at top systems conferences, and the first Google Ph.D. Fellowship in Cloud Computing.

Alex Smola (Carnegie Mellon University)

Scaling Machine Learning: Three easy pieces [Slides] [Video]
In this talk I will give an overview over three simple techniques to accelerate algorithms for recommendation (matrix factorization, factorization machines), estimation of probabilities (logistic regression), sampling (Latent Dirichlet Allocation), and for deep architectures. The talk will focus on the guiding principles (cache locality, power laws, prefetching, lock-free computation, asynchrony) to achieve very high performance on single machines and in the cloud.

Chris Re (Stanford University)

A Selection Hardware Trends for Machine learning Systems [Video]
Modern (beefy) servers with TBs of memory and tens of cores are ubiquitous and (relatively) inexpensive; they are even readily available from cloud providers. These systems have sufficient powers that they may obviate the need of scale out architectures for all but the largest problems. Inside those beefy machines, a few different hardware trends have taken shape: multiple cores, non-uniform memory access, and single instruction multiple data parallelism (SIMD). In some cases, these trends are doubling the throughput in each generation of processor. But there is a catch. One's algorithm must be able to take advantage of incredibly fine grained parallelism and instruction reordering. Using SGD as a case study, this talk describes three of our engines that take advantage of these trends. We describe how this changes both the analysis and how one implements these approaches. Some of these ideas have found their way into use at web companies for deep learning systems and into other analytics frameworks including our own DeepDive framework.

Everything discussed in this talk is in open source, see http://deepdive.stanford.edu/ and our github.

Bio:
Christopher (Chris) Re is an assistant professor in the Department of Computer Science at Stanford University and a Robert N. Noyce Family Faculty Scholar. His work's goal is to enable users and developers to build applications that more deeply understand and exploit data. Chris received his PhD from the University of Washington in Seattle under the supervision of Dan Suciu. For his PhD work in probabilistic data management, Chris received the SIGMOD 2010 Jim Gray Dissertation Award. He then spent four wonderful years on the faculty of the University of Wisconsin, Madison, before moving to Stanford in 2013. He helped discover the first join algorithm with worst-case optimal running time, which won the best paper at PODS 2012. He also helped develop a framework for feature engineering that won the best paper at SIGMOD 2014. In addition, work from his group has been incorporated into scientific efforts including the IceCube neutrino detector and PaleoDeepDive, and into Cloudera's Impala and products from Oracle, Pivotal, and Microsoft's Adam. He received an NSF CAREER Award in 2011, an Alfred P. Sloan Fellowship in 2013, a Moore Data Driven Investigator Award in 2014, the VLDB early Career Award in 2015, and the MacArthur Foundation Fellowship in 2015.