The Team Behind the Software Powering Our Nectar Research Cloud

The Australian Research Data Commons (ARDC) Nectar Research Cloud’s self-service structure allows users to access and analyse their own data at any time, create dedicated virtual servers on demand, and collaborate with others from their desktop in a fast and efficient way. Behind this infrastructure, there are many things running in the background to ensure our users can store, access, and process their research data, remotely, rapidly and autonomously.

OpenStack is the underlying open source cloud operating software that runs the Nectar Research Cloud. OpenStack provides the ability to control compute, storage and network infrastructure, allowing users to launch virtual machines, provide storage and create virtual network resources like routers to connect up their computers in the cloud.

The software is run by the ARDC Core Services team in collaboration with technical staff at each of the Nectar Research Cloud’s federation sites (TPAC, QCIF, Intersect, NCI, Monash and UoM nodes, and the Swinburne University and University of Auckland sites). Together, they not only support and build upon the Research Cloud, but they’re the driving force behind the ARDC’s eHelp support site, a self-help portal that features a dedicated cloud and data storage knowledge base along with multi-channelled support services. This portal is staffed by these federation members, meaning local support to our Research Cloud users is always available.

As the OpenStack software is built by an international community of developers in collaboration with its users, this means input by teams such as Core Services is vital to informing the development of future releases and updates. Through running OpenStack, the Core Services team often encounter issues or bugs that might have missed testing by the developers. When this happens, the team submit bug reports and code fixes that improve the stability and useability of OpenStack for the wider community. By doing so, the team ensures it continues to provide a seamless, ubiquitous platform to support researchers compute and storage needs.

OpenStack has a new software release every 6 months, with names going in alphabetical order. The 19th and latest release is called Stein. The Nectar Research Cloud has been operating since the third release, Cactus, and the Core Services team has made many contributions to the software since those very early days.

Andy Botting, Systems Engineer, ARDC Core Services talks about how the improved functionality in this latest release will benefit Nectar Research Cloud users.

“We are constantly upgrading the research cloud as new releases become available which bring new features and stability improvements. A new feature in this release includes ‘Application credentials’ which allows our users to create credentials that they use in scripts and applications like Jupyter Notebooks which are separate from their main credentials. This means better security for their research data.”

Jake Yip, a DevOps Engineer at ARDC Core Services adds, “We have also recently added the Container Orchestration Engine Service, powered by the OpenStack project known as Magnum, which allows users to spin up a Kubernetes cluster. Kubernetes itself is a complex software stack, but by the ARDC offering this service, users can jump right into having their own Kubernetes cluster without a lengthy set up process.

“From our work setting up Magnum, we encountered several issues that we have patched and contributed back upstream. We also worked closely with the Magnum developers and other organisations like CERN, a european organization for nuclear research, on improvements to Magnum. For example, public floating IPs in the ARDC is a scarce resource, and Magnum needed a floating IP for each worker node. This is not strictly necessary, so we have submitted a feature request and upstream is working on incorporating it. By working closely with the Magnum project team, it is a win-win scenario – we get a service that works well for us, and is also beneficial to the broader community. We have also helped send in a number of Python 3 compatibility patches, as well as patches to the testing framework.”

And it doesn’t stop there.

“In terms of what to expect in future releases, our users can expect functionality such as usage graphs for compute resources, improved support for containers, and reservation of compute resources for specific periods of time,” said Andy.

Read more about what the latest OpenStack software means for you. Not yet a user? Find out more about how the Nectar Research Cloud can help you and your research.