It's easy to complain about bad tools. It's harder to write a great tool yourself. Once you step into the shoes of an internal tools developer, you realize that you're balancing feature requests and constraints from a dozen different teams and an entire universe of complexity behind what should be a simple UI. In this talk, we'll talk about how to gather information about your users' needs, how to balance between conflicting requirements, and ways you can fail to build a tool that your coworkers want to use.

Black-box testing methods such as Jepsen have toppled many guarantees such as strong consistency and snapshot isolation promised by distributed SQL and SQL-like databases. Such implementations are difficult to get right, and often wrought with errors. Accordingly, there has been an increase in the use of such black-box testing tools to ensure consistency guarantees are upheld during development. The increased interest in Jepsen is evidence that traditional software quality approaches such as formal verification and white-box testing are inadequate for these systems due to their complexity. However, though this solution is quite attractive, the question arises as to whether the tools themselves are correct.This talk discusses a generative method for assembling nontrivial, non-serializable histories to test these black-box serializability checkers rooted in theoretical explorations of consistency anomalies.

We can test our code before deploying to have some confidence that it will work as expected, but what happens after deployment? How will we know that it will continue to work over time? How can we detect problems before our users do? It’s becoming more and more important to have insight into our applications and services after they’ve been deployed, and monitoring can help us achieve that. In this session, we’ll learn about what monitoring is, why it is useful, and how to get started with monitoring using Prometheus.

“Who is Jenkins?”

“I thought dockers and containers were used in transport and logistics - aren’t you a developer?”

“Why do you have puppets at work?”

“Why is there a vagrant in your office?”

“What are you talking about??”

If you’re reading this, you probably understand the double-meanings behind words like Jenkins, Docker, Puppet, and Vagrant. However, for those working outside the technology space - and even for some within it - it can seem very strange to use these terms in the context of development and technology. So, what do other people think of developers and DevOps? Do your colleagues in product or customer support teams understand what it takes for you to get a feature or fix deployed? What was the recruiter thinking when they wrote that job spec? Do the people trying to get into the technology industry right now even know what they’re in for? It just so happens that I was in all of those roles at one point or another: product owner, tech support specialist, recruiter, aspiring technologist, and more. I can give you some insight into what I thought developers and DevOps were all about while in each of these positions. Although it might seem somewhat obvious or self-explanatory to you now that you clearly understand them, you may find that when speaking to other people about DevOps or your job as a developer, people who aren’t directly involved in these things might just seem more confused, the longer the conversation goes on. Join my session to find out how others perceive devs and DevOps, how you can better communicate technical concepts to the people around you, and why doing so benefits you too.

Born in a lab, matured in the enterprise, and now fully baked as an open-source, cross-platform, professionally-supported language: the F# journey has always been about reconciling the apparently irreconcilable. Javascript and .NET, Linux and Windows, Functional and Objects, Company and Openness, Android and iOS, Programming and Data, Server and Client - F# finds a way. Come along and take a journey with me through the modern programming landscape and the F# approach to language design, interoperability, tooling and community.

Big Data, every company loves the idea of it, but often, selling the Business Case is a challenge. So how to build a successful Business Case for your Big Data initiative for the Business Users? This presentation is based on the most common objections one gets, and how to deal with them. We'll go through one of my most surprising projects, look at the lessons learned and how can we optimize the Business Case?

Kotlin’s approach to asynchronous programming is about giving developers the freedom to choose what fits best. By leveraging the conventions of the language and its ability to be somewhat extended by using simple functions, Kotlin provides an asynchronous model by implementing coroutines, and provides a series of out of the box functionality such as yield, async/await as part of the standard library. In this session we’re going to see how all of this works and how Kotlin even provides us the ability to implement our own asynchronous functionality.

Every-time there is a new piece of big data technology we often see many different specific implementations of the concepts, which often eventually consolidate down to a few viable options, and then frequently end up getting rolled into part of another larger project. This talk will examine this trend in big data ecosystem, look at the exceptions to the "rule", and look at how better interchange formats like Apache Arrow have the potential to change this going forward. In addition to general vague happy feelings (or sad depending on your ideas about how software should be made), this talk will look at some specific examples with deep learning, so if anyone is looking for a little bit of pixie dust to sprinkle on a failing business plan to take to silicon valley to raise a series A, you'll get something out this as well.

Nowadays there are many available devices to turn our house into a smart home, to make our lives easier or to keep it safe when you are on vacation on the beach. In this talk we will show you easy and inexpensive projects to connect your home (or whatever you want) for which we will use devices that cost only a few euros and cloud services.

The just-in-time compilers in the Java Virtual Machines make sure that the Java code you wrote runs as fast as possible and that the implicit and explicit checks do not compromise the performance. What are the mechanics of decision making within JITs and what happens when those turn to be wrong? The talk will cover the profile collection path, code transformations and the various interfaces to influence the JIT including the ReadyNow technology from Azul Systems.

More and more tech companies are setting up Site Reliability Engineering teams, inspired by the O’Reilly book about Google SREs. This story is about doing SRE in a more traditional and more regulated environment: in the largest bank of the Netherlands. This talk will introduce SRE concepts and address the history, present and future of SRE within a financial institution with a BizDevOps way of working. In doing so, we will talk about how our journey started, why we wanted and need to do SRE within ING, how our SREs started, the distributed way of working of SRE within ING, what technologies we actually work with and why, and our plans for the future. Lastly, we will share our SRE do’s and dont’s after two years of experience. We hope our talk inspires engineers or tech leads to start doing SRE within their (possibly more traditional) company and that those who already implemented SRE can learn from our journey.

Over the years there have been many good ideas in computer science and many stupid ideas. What can we learn from history? Which of the old and forgotten ideas should be brought back to life and re-evaluated? What can we learn from the stupid ideas of the past?

In this talk, we will explore the nature of events, what it means to be event-driven, and how we can unleash the power of events and commands by applying an events-first domain-driven design to microservices-based architectures.

We will start by developing a solid theoretical understanding of how to design systems of event-driven microservices. Then we will discuss the practical tools and techniques you can use to reap the most benefit from that design, as well as, most importantly, what to avoid along the way.

We’ll discuss how an events-first design approach to building microservices can improve the following characteristics over competing techniques:

- increase certainty

- increase resilience

- increase scalability

- increase traceability

- increase loose coupling

- reduce risk

Skeptics should definitely attend.

Nowadays there are many available devices to turn our house into a smart home, to make our lives easier or to keep it safe when you are on vacation on the beach. In this talk we will show you easy and inexpensive projects to connect your home (or whatever you want) for which we will use devices that cost only a few euros and cloud services.

Like all imperative languages Java is, with some minor but notable exceptions, an eagerly evaluated programming language. Nevertheless the introduction of lambdas in Java 8 also allowed the adoption of some lazy patterns and data structures that are more typically employed in functional languages. Streams represent the most evident example of how also native Java API has taken advantage of laziness, but there is a number of other interesting scenarios where laziness can be an effective solution to quite common problems. In fact laziness is the only possible technique to process potentially infinite amount of data, or more in general to delay the expensive evaluation of an expression only when and if it is necessary. But laziness is even more than that: for instance the reader monad delays not only a computation but also the need of external dependencies thus lowering the abuse of dependency injection, while a trampoline uses laziness to delay and then linearize recursive calls preventing the overflow of the stack. The purpose of this talk is illustrating why and how implementing laziness in Java with practical examples delivered with both slides and live coding sessions.

Want to know what is possible in the field of high performance consensus algorithms for distributed systems? If the answer is yes then come to this talk.

Once upon a time, we used software that ran on our own computers, that worked offline, and that stored its data in files on the local disk. Then we decided to put it all in the cloud. We gained some great features: real-time collaboration, like in Google Docs, for example. But we also lost control of our own data, and became dependent on far-away servers to allow us to access the data that we created. Automerge is part of an effort to get the best of both worlds. It is a JavaScript library for building real-time collaborative applications. However, apps built with Automerge also work offline, storing data locally, and synchronise their data with collaborators whenever a network is available. And although you can use it with servers, you don’t have to: synchronisation also works peer-to-peer, or via any network you choose. In this talk we will explore how Automerge deals with different users independently modifying shared data in a collaborative application (hint: by merging the changes... automatically!), how it achieves consistency in highly distributed settings, and where it is heading in the future.

Terraform enables you to describe almost anything with an API as code. From infrastructure to high level services, Terraform can help construct everything required to run your applications. In this talk, we’ll explore using Terraform to manage both traditional and non-traditional resources and dive into how to write best practices Terraform for reusability within teams.

Scalability and agility are the buzz words of modern times. This has led to new architecture concepts like microservices and functions. These architectures don’t need a complicated server backend, but can be deployed quickly and scale out easily. Often the runtime environments for these functions are Node.js, python and other scripting languages. This means, as a Java Developer, that you would have to learn a new programming language if you want to apply this concept. However, last October an open source serverless framework leveraging Java was announced: fn. It is cloud agnostic (so it runs on any cloud platform, not just AWS) and is built by the team that created IronFunctions. In this session you will learn what a serverless framework is, how you can create a function in Java using the function developer kit and how to call this function.

What is a container? Is it really a “lightweight VM”? What happens when you type in "docker run"? In this talk you'll see exactly what a container is, as Liz builds one from scratch in a few lines of Go code. You'll learn what's happening under the covers when you start a container, and understand how namespaces, controls and chroot each contribute to the making of a container, We'll also cover what it means to run a privileged or non-privileged container.

On-line transaction processing (OLTP) database management systems (DBMSs) are a critical part of the operation of many large enterprises. These systems often serve time-varying workloads due to daily, weekly or seasonal fluctuations in demand, or because of rapid growth in demand due to a company’s business success. In addition, many OLTP workloads are heavily skewed to “hot” tuples or ranges of tuples. For example, the majority of NYSE volume involves only 40 stocks. To deal with such fluctuations, an OLTP DBMS needs to be elastic; that is, it must be able to expand and contract resources in response to load fluctuations and dynamically balance load as hot tuples vary over time. In this talk, I will present a system we built at MIT called P-Store, which uses predictive modeling to reconfigure the system in advance of predicted load changes. I will show that when running a real database workload, P-Store achieves comparable performance to a traditional static OLTP DBMS while using 50% fewer computing resources.

More and more tech companies are setting up Site Reliability Engineering teams, inspired by the O’Reilly book about Google SREs. This story is about doing SRE in a more traditional and more regulated environment: in the largest bank of the Netherlands. This talk will introduce SRE concepts and address the history, present and future of SRE within a financial institution with a BizDevOps way of working. In doing so, we will talk about how our journey started, why we wanted and need to do SRE within ING, how our SREs started, the distributed way of working of SRE within ING, what technologies we actually work with and why, and our plans for the future. Lastly, we will share our SRE do’s and dont’s after two years of experience. We hope our talk inspires engineers or tech leads to start doing SRE within their (possibly more traditional) company and that those who already implemented SRE can learn from our journey.

The Reactive Manifesto motivates the reactive tenets with large-scale systems that serve millions of users worldwide, needing exorbitant data size and bandwidth coupled with sub-second latencies. This presentation puts the spotlight on the other end of the spectrum: the systems that operate manufacturing processes and logistics are vital to a factory’s success. Data sizes and rates are easily manageable with today’s hardware, the top priority is absolute reliability. In these systems resilience is the prime motivator for loose coupling, elasticity is just a consequence. This presentation explains the architecture approach taken at Actyx and discusses the suitability of different patterns in this context, including some war stories.

As "we all know", distributed transactions pretty much died with the advent of the Internet. MTS/MSDTC/J2EE books collect dust while developers of cloud services struggle programming against eventually consistent data stores. Orleans never shied away from questioning conventional wisdom. It pioneered the Virtual Actor Model that offered a compelling alternative, especially for building scalable distributed and cloud applications, to the prevailing at the time view of actors. Support for scalable distributed transactions is one of the latest innovations in Orleans that dispels popular misconceptions about them. Sergey Bykov will talk about Orleans transactions and other advanced features of the framework.

If Java 8 was all about how we code, Java 9 is all about how we will build. Modularization will have the biggest impact of any change that happened in Java since its inception. In this presentation we will learn about the need for modularization, how it impacts development, the rules to follow when creating modules, and the effect it has on legacy code. We will explore creating module, using modules, readability, exports, automatic modules, and unnamed modules.

Operators of large distributed systems often try to optimize for scarce computational resources — memory, network bandwidth, and CPU cycles. However, research shows that if we want to maximize the health of our systems, there’s another scarce resource we should be prioritizing instead: the attention of our users.

In this talk, I’ll show how we can use machine learning to improve how we present data and information and tell a story to our users, optimizing for their attention and holistic understanding, instead of overwhelming them with a barrage of low-level metrics. More intriguingly, we’ll see how this approach has produced massive improvements in several real-world systems at large Fortune 100 institutions. By the end of the talk, we’ll hopefully have convinced you that happier and more informed operators isn’t merely a good outcome, but rather is the most ethical thing to do if we don’t want to lead people astray.

Transactions and Concurrency Control are very of paramount importance when it comes to enterprise systems data integrity. However, this topic is very tough since you have to understand the inner workings of the database system, its concurrency control design choices (e.g. 2PL, MVCC), transaction isolation levels and locking schemes.

In this presentation, I’m going to explain what data anomalies can happen depending on the transaction isolation level, with references to Oracle, SQL Server, PostgreSQL, and MySQL.

I will also demonstrate that database transactions are not enough, especially for multi-request web flows. For this reason, I’m going to present multiple application-level transaction patterns based on both optimistic and pessimistic locking mechanisms.

Last, I’m going to talk about Concurrency Control strategies used in the Hibernate second-level caching mechanism, which can boost performance without compromising strong consistency.

In 2018 we will see a big change in the IoT landscape: based on LTE-m devices can connect to the internet over long distances and will be running off batteries for years. Learn about the principles and protocols involved and how to leverage JavaScript down to the hardware to build your own solution.

The software industry has spent the last 10 years creating tools and frameworks that reduce the friction in the delivery process, like continuous integration tools, test driven development, automated deployments, etc. There should not any reason why projects that have hardware components cannot use the same tools. However, in practice, this process can be really costly and risky and it requires a mindset that hardware and embedded system developers are not used to.

In this talk, I explain how to apply continuous delivery practice in projects that contain hardware and software components and embedded systems and how to solve the most common challenges.

Main topics include:

• Strategy for creating continuous integration pipelines
• Tips on how to reduce the development feedback cycle
• How to automated deployments and test on physical boards
• What development practices work in this environment

Have you ever wondered how to actually achieve elasticity on the cloud, outside of your cushy local dev environment?

Akka is the most mature choice to implement the traits of the Reactive Manifesto, thanks to the Actor model. Resilience is enabled by asynchronous message passing and Actor supervision. Location transparency, message routing and the “share nothing” approach enable seamless scalability, but once our app is deployed we need to rely on some external infrastructure to automatically scale up or down our services. At this point, Akka alone is not sufficient anymore. What else do we need?

We found Docker & Kubernetes to be a perfect match for clustered Akka applications running in the cloud. Thanks to the Kubernetes API, one can write infrastructure logic that complements the business logic. This leads to a general approach of infrastructure-as-code tailored to each different business case, which in turn lets companies achieve maximum efficiency and resource usage.

During this talk we will cover features from both Akka and Kubernetes, plus example code in Scala: familiarity with these technologies is recommended. Our example will rely on Google Cloud infrastructure.

Software is flexible, specialized hardware is extremely fast. So why not write software, then turn it into a computer chip? This is what Hastlayer does by transforming .NET software into electronic circuits. The result is faster and uses less power while you simply keep on writing software!

The rise of microservices has allowed ML systems to grow in complexity but has also introduced new challenges when things inevitably go wrong. Most companies provide isolated development environments for engineers to work within. While a necessity once a team reaches even a small size, this same organizational choice introduces potentially frustrating dependencies when those individual environments inevitably drift. Kinnary Jangla explains how Pinterest dockerized the services powering its home feed to accelerate development and decrease operational complexity and outlines the benefits Pinterest gained from this change that may be applicable to other microservice-based ML systems. This project was initially motivated by challenges arising from the difficulty of testing individual changes in a reproducible way. Without standardized environments, predeployment testing often yielded nonrepresentative results, causing downtime and confusion for those responsible for keeping the service up.

The Docker solution that was eventually deployed prepackages all dependencies found in each microservice, allowing developers to quickly set up large portions of the home feed stack and always test on the current team-wide configs. This architecture has enabled the team to debug latency issues, expand its testing suite to include connecting to simulated databases, and more quickly do development on our thrift APIs.

Kinnary shares tips and tricks for dockerizing a large-scale legacy production service and discusses how an architectural change like this can change how an ML team works.

At RNEL, we have been working hard to lay the foundation to better manage our data and be able to integrate big data and AI technologies into our data management and analysis pipelines. These needs have arisen from the very size of the experimental data that push the limits: they are simply becoming unmanageable even on powerful workstations. We also determined that better query methodologies, validation and visualization tools are needed.

Our long term goal is finding the answer to the following question: Will we ever be able to go from experimental raw data to query curated data with a simple SQL-like language without spending humongous resources and manpower, while using a process that is organic, intuitive and flexible? Can we also leverage modern big data technologies and data science to achieve our goal?

This presentation is the story of an inter-disciplinary journey that started approximately 5 years ago. The journey enabled us to build a deeper knowledge of our data, a better system of management methodologies, as well as tools that allow us to query and aggregate across various datasets and easily improve such functionalities.

In this presentation, we will provide a general background of the work that we do in our lab. First, we will provide some examples of experiments that we conduct as a context in which to explain the data that are acquired and the challenge that comes with them. Next, we will outline some of the questions that researchers asked (and keep asking) when they attempt to work with large data structures to answer their own scientific questions without having to be bogged down by the technologies used and the original format of the data. Finally, we will raise some questions related to data management, which will help to improve validation and reduce the manpower necessary to curate the data. From the big picture, we will walk through the decisions and requirements that came out of our brainstorm sessions and show how far along we currently are in our journey and the path we took to get here. We will conclude by highlighting some of the amazing results that we were able to achieve, such as activation maps and central nervous system stimulation counts.

In this presentation we will talk about our approach to build complex and distributed applications following DDD and Reactive principles.

You will learn on how we approached this problem from a pure functional way using our opensource framework (reactive-platform), how Scala helped us through its advanced type system and DSL capabilities to combine underlying technologies such as Kafka, Cassandra and Akka to implement an actor distributed system based on CQRS/ES and NoSQL.

In this presentation we will talk about our approach to build complex and distributed applications following DDD and Reactive principles.

You will learn on how we approached this problem from a pure functional way using our opensource framework (reactive-platform), how Scala helped us through its advanced type system and DSL capabilities to combine underlying technologies such as Kafka, Cassandra and Akka to implement an actor distributed system based on CQRS/ES and NoSQL.

You only look once is a state-of-the-art, high speed real-time object detection algorithm. It looks at the whole image at test time so its predictions are informed by global context in the image. This talk teaches you to develop your own application to detect and classify objects in images & videos.

1.Intro to YOLO algorithm 
2. Image detection on video with YOLO
3. Processing images in Python, adding bounding boxes and labels
4. Processing complete videos in Python in the similar way as the previous section 
5. Processing real time video from webcam

Software is flexible, specialized hardware is extremely fast. So why not write software, then turn it into a computer chip? This is what Hastlayer does by transforming .NET software into electronic circuits. The result is faster and uses less power while you simply keep on writing software!

2gether, a financial platform, built in on Big Data, AI and Blockchain. On the following keynote, we try to give the hints on the evolution of the project, from the beginning up until ending up with a beta version. The main technological side’s challenges, the situational methodologies that have been implemented and the next technological steps to turn the beta version into a launched product to the European market.

Kubernetes has taken the container world by storm, becoming the popular choice for developers and operations teams alike to manage their container deployments at scale. In this session learn how Oracle's managed Kubernetes service combines the power of Kubernetes with the raw performance of Oracle Cloud Infrastructure. See how you can avoid the complexity of standing up and maintaining your own Kubernetes infrastructure while giving your containers direct access to native bare metal performance and empowering development teams to achieve continuous integration and continuous delivery goals with Wercker.

ArangoDB is a scalable, distributed multi-model database. However, for this talk, it is not necessary to know what this means. Rather the only crucial fact is that it is distributed and written in C++.

Before you stop reading: This talk is about a golang success story. Namely, we had to implement resilient data center to data center (DC2DC) replication for ArangoDB clusters from scratch within 6 weeks (plus some time for testing and debugging). Therefore, we built upon – ArangoDB’s HTTP-based API for asynchronous replication, – the existing golang driver, – the fault tolerant scalable message queue system Kafka, – a lot of existing golang libraries and – golang’s fantastic capabilities for parallelism, communication and data manipulation and pulled this task off. This talk is the story of this project with its many challenges and successes and ends with a surprising revelation about which of the above we did not actually need in the end.

After spending four years building a proprietary all-in-one streaming analytics engine for financial services, it became clear that open-source was starting to pull ahead.  Alvaro will talk about the challenges of creating an IT operations solution for financial services; what to build, what not to build, and how to use open source tools to get past the infrastructure and focus on the business problems that matter.

After spending four years building a proprietary all-in-one streaming analytics engine for financial services, it became clear that open-source was starting to pull ahead.  Alvaro will talk about the challenges of creating an IT operations solution for financial services; what to build, what not to build, and how to use open source tools to get past the infrastructure and focus on the business problems that matter.

Big Data, every company loves the idea of it, but often, selling the Business Case is a challenge. So how to build a successful Business Case for your Big Data initiative for the Business Users? This presentation is based on the most common objections one gets, and how to deal with them. We'll go through one of my most surprising projects, look at the lessons learned and how can we optimize the Business Case?

This meetup 'YWT On The Beach' event will be guided by some fantastic members of the Yes We Tech community that will present short talks and share their experiences on Distributed Systems, Data Science and Big Data Analytics.

This meetup is free and everyone is invited to join.

How can I obtain UV data?

by Gloria Passarello

UVI. UltraViolet Index. Everyone of us heard this word at least once, but what is it exactly? Which kind of data do we need and how could we calculate it? Data coming from satellites and Python. Nothing more. We will explore this particular kind of data and how to obtain UVI from them. And in the end we will enjoy with an animation and we will see how print it on a gif.

Leveraging Amazon cloud-computing resources to support big data analytics.

by Victoria Caparrós

In this talk I will present the system architecture used at RavenPack to provide fast, big data analytics.

In contrast to traditional approaches that make heavy use of proprietary resources, we leverage various services from the cloud-computing platform that provide a flexible, serverless platform that allows us to focus on the application and forget about managing the underlying infrastructure.

Distributing your app like a pro with asyncio

by Marta Gómez

In this talk, we'll define a simple echo server using asynchronous programming.

This meetup 'YWT On The Beach' event will be guided by some fantastic members of the Yes We Tech community that will present short talks and share their experiences on Distributed Systems, Data Science and Big Data Analytics.

This meetup is free and everyone is invited to join.

How can I obtain UV data?

by Gloria Passarello

UVI. UltraViolet Index. Everyone of us heard this word at least once, but what is it exactly? Which kind of data do we need and how could we calculate it? Data coming from satellites and Python. Nothing more. We will explore this particular kind of data and how to obtain UVI from them. And in the end we will enjoy with an animation and we will see how print it on a gif.

Leveraging Amazon cloud-computing resources to support big data analytics.

by Victoria Caparrós

In this talk I will present the system architecture used at RavenPack to provide fast, big data analytics.

In contrast to traditional approaches that make heavy use of proprietary resources, we leverage various services from the cloud-computing platform that provide a flexible, serverless platform that allows us to focus on the application and forget about managing the underlying infrastructure.

Distributing your app like a pro with asyncio

by Marta Gómez

In this talk, we'll define a simple echo server using asynchronous programming.

This meetup 'YWT On The Beach' event will be guided by some fantastic members of the Yes We Tech community that will present short talks and share their experiences on Distributed Systems, Data Science and Big Data Analytics.

This meetup is free and everyone is invited to join.

How can I obtain UV data?

by Gloria Passarello

UVI. UltraViolet Index. Everyone of us heard this word at least once, but what is it exactly? Which kind of data do we need and how could we calculate it? Data coming from satellites and Python. Nothing more. We will explore this particular kind of data and how to obtain UVI from them. And in the end we will enjoy with an animation and we will see how print it on a gif.

Leveraging Amazon cloud-computing resources to support big data analytics.

by Victoria Caparrós

In this talk I will present the system architecture used at RavenPack to provide fast, big data analytics.

In contrast to traditional approaches that make heavy use of proprietary resources, we leverage various services from the cloud-computing platform that provide a flexible, serverless platform that allows us to focus on the application and forget about managing the underlying infrastructure.

Distributing your app like a pro with asyncio

by Marta Gómez

In this talk, we'll define a simple echo server using asynchronous programming.