New: Research and Engineering Studio on AWS

Your brain is extravagantly equipped for visualization. More than 50% of your cerebral cortex is dedicated to processing sensory input from your eyes, making it the information super highway that helps complex ideas form in your head incredibly quickly. Our output devices – our limbs and our voices – had to develop unique articulation skills, just to match the speed at which we can learn from seeing.

So, it’s fitting that today we’re announcing Research and Engineering Studio on AWS (RES), a self-service portal to help scientists and engineers access and manage their research and design workspaces, including virtual desktops to see their data and run their interactive applications in the cloud.

Setting up – and then managing – virtual desktops manually takes a lot of work, taking time away from the core mission of research and development groups: solving hard scientific problems. We’ve also noticed that when processes to access resources become complicated, users often take shortcuts (like sharing passwords or setting file permissions to be globally read/write).

With RES, we’re putting all the launch and monitoring tools behind a single pane of glass, so administrators can deploy new desktop environments quickly, from their own pre-prepared library of software images and trusted applications. This gives their users the freedom to come and go when needed – incredibly important when chasing new lines of inquiry – without impacting their budgets or the security posture of their organization.

In today’s post, we’ll explain what RES is and how it works, and we’ll explain how to deploy it in your own AWS account, and get it ready for your users.

Visualization without the headaches

Research and Engineering Studio on AWS has been designed to make it easier for site admins to provide a suite of applications, data, and large-scale compute resources through familiar interfaces – starting today with interactive desktops.

When you login to RES through your browser, you’ll have the ability to see the virtual desktops that are available to you, based on permissions that draw from your membership of user Groups that are associated with Projects. We’ll explain this a little further on.

On the main Virtual Desktops screen, you’ll have the ability to see thumbnails of sessions that are live and running, and other sessions that are suspended. All have sets of controls to spin them up, or shut them down. You can even schedule sessions start and stop during specific time windows (like 9-5 Mon-Fri, if that’s your thing). This is also where you can create new sessions, based on software stacks your admin has provided for you – this only take a couple of minutes, typically.

By clicking Connect in the RES interface for a running session, you can open a new browser tab containing an encrypted shared desktop session with the remote system using NICE DCV – our own high performance streaming protocol. DCV uses a lot of tricks with network protocols designed to make remote desktops feel like they’re local. The techniques it uses to do this are remarkable enough that Netflix uses it to provide their editors and FX teams access to powerful Amazon EC2 systems so they can produce the great content you’re probably watching this week on TV. This is great for scientists and engineers, since our interactive applications often need precision screen controls and only work well with access to truly large memory models.

Figure 1 – the RES Virtual Desktops screen lists all the sessions a user has previously created, with controls to spin up, shutdown, or schedule uptime.

RES allows users to share their desktops, too. While the sessions they start in RES will always belong to them, they can share a session too, like the way you can do that in a conferencing system like Zoom. This can be incredibly useful for working sessions where a group of engineers in a team need draw design insights by poring over visualizations of massive datasets – without needing to move the datasets themselves. A desktop share can have an expiration date attached to it, too.

Figure 2 – RES permits users to share their virtual desktop sessions with others in a controlled fashion.

Projects, users, and groups

To manage all of this, RES works from a few important concepts, all of which are likely familiar to you already.

1. First is the Project. In most organizations – public and private – everything revolves around project teams. Resources are typically created for – and consumed by – project teams, with access, billing, and even software stacks being driven by project needs.
2. Users and Groups: Users can be members of multiple Groups, and in RES a Group can be associated with one or more Projects.

Like most institutions, you probably have an investment in a user identity and authentication system, complete with existing project groups that you’ve defined to define access for shared files and other resources. We wanted to align with that, so RES – out of the box – is built to connect with your existing directory service, starting with Active Directory (AD), helping you avoid doing this work twice.

At launch, RES can connect to your existing AD server on-premises or in the cloud. If you want to separate your RES environment from your regular user space – to create an engineering sandbox, for instance – you can spin up a managed AD using AWS Directory Service to privately manage a set of users.

Figure 3 – Identity functions, like defining users and groups, are done centrally in an Active Directory environment, which you specify when installing RES in your AWS environment. The AD can be on-premises or in the cloud and can – if you wish – be separate from your corporate directory service.

RES enables access to Project resources, by specifying which Groups are assigned to the Project. You define this at Project creation, but of course you can change it afterwards, too (see Figure 4).

Figure 4 – when you create a Project in RES, you can specify which user groups are permitted to access its resources. The Groups and Users are inherited from your Active Directory service.

RES is designed to bind to your AD and draw its knowledge of your teams from there, which allows RES to reflect changes you make to Group memberships right away.

Storage

At launch, RES supports two file systems which customers have told us are most useful for managing users’ files. Amazon Elastic File System (EFS) and Amazon FSx for NetApp ONTAP. While EFS is great for customers with large Linux deployments using NFSv4, FSx for NetApp ONTAP can offer the same file system using either NFS or SMB – which is helpful if you’re running a hybrid Windows and Linux environment and your users regularly flip back and forth, depending on the applications they’re using.

Figure 5 – You can manage file systems in RES, and associate them with Projects. At launch, RES supports Amazon EFS, and Amazon FSx for NetApp ONTAP.

RES allows an administrator to spin up new filesystems or to onboard existing filesystems which you might have previously created for your project. You need to associate a file system resource with a Project to make it visible to the users who are member of the Groups associated with that project. Individual file and folder access is still controlled in the usual way at an operating system level with permissions, which again will usually map back to your Users and Groups definitions in the AD you’re using for your directory service.

Software stacks and machine images

RES uses Amazon Machine Images (AMIs) as the basis for customizing the environments you deploy for your users. If this is new to you, an AMI is an image that provides the information required to launch an instance. You need to specify an AMI whenever you launch an instance. You can launch multiple instances from a single AMI when you need lots of machines with the same configuration. And you can use different AMIs to launch instances when you need add some local flavor to your users’ environment.

RES extends this concept a little further by narrowing the set of instances and GPUs it will offer based on an AMI. And – as you’d expect – the resulting stack needs to be associated with a Project before a user will know about it. This is helpful not just for customizing what your users can do, but it’s great when you have commercially licensed software, too. For example – all members of Project Beeblebrox can access an AMI with the Marvin license embedded in it.

Creating an AMI is almost as easy as launching an existing one, making your changes, and then asking the Amazon EC2 console to create a new AMI from it. After that, you just need to take the ID of your AMI to RES admin screen and register it.

Starting today, RES is designed to support Windows and Linux operating systems, specifically: Windows Server 2019 (Datacenter Version 1809), Amazon Linux 2, CentOS 7, Red Hat Enterprise Linux 7, Red Hat Enterprise Linux 8, and Red Hat Enterprise Linux 9. We’ll keep the list of supported operating systems up to date in the online documentation for RES.

One click deployment – or two

Research and Engineering Studio is a self-managed, open-source product which you install in your own AWS account.

There are several ways of getting up and running with RES. Which one you choose depends on your experience with AWS, and how heavily you want to customize it to suit your local needs. This includes a “batteries included” method which is a full, one-click launchable AWS CloudFormation stack, that can be used to build an entire environment, including a managed AD server.

RES is a supported open-source project, available via its public GitHub repository, which is where you can find the RES installation recipes (including several components, supplied by the HPC Recipe Library).

RES can be used in the following Regions: US East (Ohio), US East (N. Virginia), US West (N. California), US West (Oregon), Asia Pacific (Singapore), Asia Pacific (Sydney), Europe (Ireland), Europe (Frankfurt), Europe (London), Europe (Paris), Asia Pacific (Mumbai) and AWS GovCloud (US-West).

Links to documentation, and more are available on the Research and Engineering Studio homepage. Try it out and let us know what you think at ask-hpc@amazon.com.