MapR and Mesosphere collaborate

Mesosphere and MapR Technologies have introduced Myriad, the first resource management framework that allows Apache YARN jobs to run alongside other applications and services in enterprise and cloud datacentres. Based on an open source and collaborative development effort between MapR, Mesosphere and eBay, Myriad (available on GitHub) is an open source project built on the vision of consolidating big data with other workloads in the datacentre into a single pool of resources for greater utilisation and operational efficiency. Concurrently, there are plans to submit Myriad as an Apache Incubator project with the Apache Software Foundation in the first quarter of 2015.

To date, Hadoop developers have been forced to run big data jobs on dedicated clusters, leaving those resources isolated from other applications and services in production, and typically resulting in poor server utilisation rates. Myriad leverages both Apache YARN and Apache Mesos, allowing big data workloads to run alongside other applications including long-running Web services, streaming applications (like Storm), build systems, continuous integration tools (like Jenkins), HPC jobs (like MPI), Docker containers, as well as custom scripts and applications.

“Big data developers no longer have to choose between YARN and Mesos for managing clusters,” said Florian Leibert, CEO and co-founder of Mesosphere. “Myriad allows you to run both, and to run all of your big data workloads and distributed applications and systems on a single pool of resources. Big data developers get the best of YARN’s power for Hadoop-driven workloads, and Mesos’ ability to run any other kind of workload, including non-Hadoop applications like Web applications and other long-running services.”

Apache Mesos is a distributed systems kernel that abstracts CPU, memory, storage and other compute resources allowing developers to programme against the datacentre like a single pool of resources. It forms the core of Mesosphere’s Datacentre Operating System (DCOS). With Myriad, organisations running Mesos or Mesosphere DCOS can now run YARN and other big data frameworks on the same Mesos cluster. For YARN users, Mesos unlocks resource scheduling and isolation beyond just CPU - and allows memory, networking and storage to be dynamically managed on the same cluster that runs their big data jobs.

“Myriad enables businesses to tear down the walls between isolated clusters just as Hadoop enables businesses to tear down the walls between data silos,” said Jim Scott, director, enterprise strategy and architecture, MapR Technologies. “Developers can now focus on the data and applications which the business depends on, while IT operations can manage compute resources to maximise business agility and minimise operating expenses.”