Too often, contacting tech support can be like seeking help from the wizard. In CLA, however, we take a different approach -- we enjoy talking to you!

Have general or specific questions about server infrastructure? Interested in a consultation about supporting a new research project? Work off campus frequently and prefer to schedule a specific time to meet? Connect with us in person, via phone, or in a Google Hangout.

To ensure we're ready to assist you, and so you don't have to worry about coordinating with a specific engineer, select from our calendar of appointment times:

http://z.umn.edu/a04

Learn more about Google Calendar appointment slots

• Using appointment slots
• Introducing appointment slots in Google Calendar

CLA-OIT's Linux terminal server, lts.cla.umn.edu, allows you to access dozens of other servers from within a single session. While the access is very convenient, it would be very inconvenient if you had to type your password every time you wanted to log in to a server. There is a very elegant, but technically complicated way, of removing passwords from the login process while still keeping logins secure.

We have documented how you can configure this for your login on lts.cla.umn.edu, as well as provided links to resources describing the same process for OSX and Windows:

Configuring Public/Private Key Authentication

Feedback about the format or content of this article? Email us at 4help@umn.edu.

When you connect to the new farm, you'll be routed automatically to one of the Remote Desktop servers. If you already have an existing session on one of the servers, you'll be reconnected to it, wherever it may be.

The new farm will have a different name: wts.umn.edu. Starting in August, connect to wts.umn.edu instead of freya.cla.umn.edu. Note that freya will be retired on August 4, at which point incoming connections will automatically be routed to the new farm.

Questions or comments? Email us at 4help@umn.edu.

SAS

We've upgraded SAS on freya, the Windows terminal server, to version 9.3. We are still running SAS 9.2 on loki. Since loki is one of our older servers, and is already overloaded with the SPSS server software it is running, we have decided to move our SAS install to one of our new machines, apollo. This will be happening over the next few days.

Matlab

We are now running Matlab 2012a on all our Linux servers and freya. We have not recently upgraded Matlab since there has been no demand for the newer versions, but we have received requests for version 2012a and decided that it was time to move on. We have also removed some of the older versions that we've been keeping around for backwards compatibility. We have done this because we needed to make room for new software installs, and also because some older versions stopped running under current Linux versions.

Stata

Stata has been updated to version 12 on all Linux servers.

Mathematica

Mathematica has been updated to version 8 on Windows and Linux servers.

Tell Us What You Think

As you use the servers, you might notice that the "default" versions you get are different from one machine to the other. Since we do not have a fully automated method to deploy these applications yet, we might have overlooked some servers for some of the applications. If you notice something outdated or missing, on any given server, please let us know.

There are many other applications that we are running on our servers. Most of these are specialized ones that we will only update when explicitly asked. If you happen to use one of these and would like it updated, send us a request.

In recent years, computing technology has grown by leaps and bounds. While our current generation of servers can have sixteen processor cores and almost a terabyte of memory, there is almost no task big enough to require this kind of power in our environment. Even the least expensive server we can buy, for around $4,000, has twelve cores and 16GB memory.

Now virtualization has become really useful--running multiple independent servers on a single piece of hardware, under the virtualization hypervisor, offers a way to make use of all the excess power in a modern server.

Our Server Infrastructure

CLA-OIT is running more than 120 virtual servers on 10 physical servers using VMWare's technology. This saves a tremendous amount of money and makes it very easy to create new servers. While it might take weeks to choose, order, receive, and configure a new physical server, we can create a virtual server in a couple days, (or hours, if needed).

Upsides

Virtualization is a great convenience for system administrators, and wonderful news for accountants, but why might it be important to you? Imagine a research project or a class needs a single-purpose "simple" server that does not require a lot of horsepower. Spending thousands of dollars on a physical machine that will be 90% idle is not the best idea, and running server software on an office computer is even worse. Our no cost virtual server hosting service makes it easier to build prototypes, initiate pilot projects, or support classroom work that lack funding for hardware investments.

In addition to the ease and speed of creating new servers, virtualization offers great benefits in redundancy and high availability. For example, if you are using a physical server, any hardware failure means that you are dead in the water until the server is repaired. Since our virtualization environment in composed of servers working in tandem (in a cluster), a hardware failure is just an inconvenience--it is only a matter of minutes to move virtual servers that were running on a failed piece of hardware to the next available server. Over the next couple of years, we will be working to enhance this capability, so that our virtual infrastructure will be able to keep running even when the University's entire datacenter is unavailable.

Downsides

One downside of virtualization is server "sprawl," which is what happens when it is so easy to get a new server. Five years ago, when we first started our virtualization project, we had fewer than 100 servers. Today, we have more than 300--most of them virtual. Even though we tripled the number of servers we're running, we haven't tripled our staff so we must find ways to make running servers easier. Standardization and manageability have become key points for our server design. When you request a virtual server we will ask you to follow our standards, and you might have to reconsider certain decisions to fit those standards.

Requesting a Server

If you have a server-related need, we want to hear from you--even if it is about a service we aren't currently offering. While a dedicated virtual server may be the right answer, we offer many options. In any case, our staff will go through an in-depth planning process with you to ensure we find the solution that best meets your needs.

NX Client Downloads

• Windows and Mac OS versions prior to 10.7: Nomachine NX 3.5 (NOT NX 4)
• Mac OS X 10.7 "Lion": OpenNX

What Sets us Apart

Your home and shared directory storage is not restricted to one server (like it would be on your desktop or laptop) but is shared with all the other servers. What makes this useful are two services that we run on our servers:

Network File System (NFS). This protocol provides a method to share directories to trusted remote servers that run Unix and make them look like regular directories. We use this system to transparently "mount" your home directory when you log in to any trusted server. That way, you always get the same contents in your home directory regardless of which server you are using, and the contents are synchronized. This synchronization also applies to any shared directories you have access to.

Common Internet File System (CIFS). Also known as Samba, this protocol extends the same functionality to Windows and Mac computers. You can create a network drive on your computer with synchronized contents of your home directory or shared directories.

How it's Useful

Making use of NFS and CIFS, you can edit a Matlab program in your home directory directly on your own computer, save it, and then run it immediately on an application server. When your program is finished and saves its output, you can view and print the output file from your office printer. Working this way, you never need to worry about where you stored the latest version of your dissertation, or remember to send the right copy to the server.

Home Directories

Each account holder is allowed up to 25GB of storage in the home directory. CLA-OIT system engineers don't police your usage, or monitor what you store there (except in the case of a security breach or legal issue), so you are free to use that space for whatever you want. We prefer not to host your music collection or copies of YouTube videos, but we won't stop you unless you use a lot more than 25GB. If you need additional space, you can double your allocation to 50GB simply by emailing us at 4help@umn.edu. We will ask you why you need the extra space, and expect you to have a good reason, but we will almost always grant such requests.

Shared Directories

We provide shared directories for research projects with well-defined needs, class work, or other legitimate University business, so don't be surprised if we reject a request for space to share your personal movie collection with your classmates.

Private Data

We strongly discourage you from storing legally private data in either your home directory or any shared directories. While these systems are secure and well-protected, they are designed to facilitate sharing between our users, not to provide protection for highly sensitive data. If you are responsible for the safety of patient or student data, you should not use these systems. We offer alternatives that provide better security, and we would be happy to talk to you about your needs.

But wait, there's more...

Keeping your data on the server, rather than on your laptop that can be stolen from your car or die unexpectedly, is a good first step to keep your data safe. What can save you from a mistyped delete command, or the server itself crashing and burning? For added protection against disasters, CLA-OIT runs an extensive backup system.

Everything you keep on our systems is backed up nightly using three high-performance servers located a mile away from the data center our file servers are located in. The backups are kept on disks, for quick retrieval, and copied to tapes for longer term retention. The tapes are then moved from the location of the backup server to a different building for extra safety. We keep backups for a maximum of 90 days, giving you ample time to notice when something is amiss, and recover your files. For more information about backups, see our File Backup & Restore service.

What is I use Dropbox/iCloud/Crashplan?

In the last few years, many cloud-based storage solutions have provided free or low-cost methods for backups and synchronization. We know many of you use these solutions (even though the University prefers you don't). While each solution has its strengths, CLA-OIT's system is better in several ways.

Privacy. While even we rely on cloud services to store a lot of our data, there are certain things that you should think twice about storing outside the U. Dropbox, iCloud, and other companies have very little legal responsibility for your privacy, including private research data, student grades, and in the case of some of these services, your original intellectual property.

Cost. Most of these services charge a fee if you need more than 5GB of space and the costs can really add up.

Access. CLA-OIT servers do not connect with your Dropbox folders. Storing your research on our system keeps all of your materials in one place.

Backups. While many cloud-based storage offerings allow you to retrieve older versions of your documents, they do not really offer a proper backup option. There are some backup-oriented services that you can use for your personal backups, but they do not offer all the flexibility that you might want for your daily work.

Load Average

In the screenshot above, the "load average" (in lime green) is a summary of processor utilization. The numbers are average load figures from the last minute, five minutes ago, and fifteen minutes ago. A load of 3.18 should be read to indicate that the computer has enough tasks to keep an average of 3.18 cores/processors occupied. When this number exceeds the number of cores available on the server, jobs will start to slow down. At this point you should avoid submitting new jobs.

Memory

Also indicated above, the "Mem:" (in red) line describes memory utilization, with the numbers in kilobytes. In this example, the server has 32GB total memory, and at this moment approximately 10GB is used with 22GB free. The figure for free memory is the one you should watch. When memory goes too low the computer will start using virtual memory (see the "Swap:" line) and everything will start to slow down. In this instance, virtual memory is barely used because there is ample real memory.

Job Details

The image above displays many details about the "top" jobs currently running. Two important columns in this display are "RES" and "%CPU" (in blue). These show the amount of physical memory and the percentage of one processor each job is using, respectively. In this display, user john1395 is running three concurrent Stata jobs that are fully utilizing three cores, which violates the policy.

License to Kill (misbehaving jobs)

CLA-OIT system administrators do not typically watch servers for violations. Instead, we hear about overuse issues directly from researchers whose jobs cannot run because someone else is monopolizing a server or when a server becomes completely unresponsive. When jobs start behaving badly, we'll respond in one of the following ways:

Lower job priority. Each job on the server has a priority, called the "nice" level, denoted by the NI column (in fuchsia) on the job table from top. The lower the number, the higher the priority. When we demote a job's priority it is forced to wait until jobs with a higher priority are finished. This is the simplest, least destructive, way to make sure a single person is not blocking others from doing work, by sending them to the back of the queue and only allowing their extra jobs to finish when nothing else is happening.

Stop job. Sometimes more drastic action is necessary if a job is trying to consume more memory than is available. This can cause an entire server to crash, and lowering priority will not solve the problem because it only puts the job on virtual memory. Stopping a job allows a researcher to resume it later, without losing data. View the run/stop status of a job in the "S" column in the top display (to the left of the "%CPU" column.) An "R" in this field denotes running jobs, "S" sleeping jobs, and "T" stopped jobs. You can restart your own stopped jobs or we can do it for you. Note: if we restart your stopped job, talk to us so we can avoid repeating the problem.

Kill job. A last resort, the kill job option is irreversible and almost always causes loss of data. We will only kill jobs that are obvious "runaways," meaning crashed or unresponsive, but still consuming resources. Learn how to kill jobs yourself by checking out the documentation for the kill command -- type "man kill" at a command prompt.

As always, send us an email if you'd like to learn more about managing your jobs or have questions about our policies.

For computational workloads (and everything else, really) there are two different scarce resources on a computer--processor and memory. Running a typical task on a computer, such as your email program, doesn't use the processor continuously--it is in short, small bursts. As a result, many small tasks can share a processor and get work done.

In the case of large computational tasks, the processor is kept more or less continuously busy, and it has no time for any additional tasks. Since modern computers have multiple logical processors ("cores",) they can easily run a computational task and many other things at the same time. But if you try to run more large tasks than there are cores available, the computer will become unresponsive and difficult to use for the duration of those tasks. Any new tasks will be queued up, waiting for the active ones to finish.

In the case of memory, the behavior is slightly different. If active tasks occupy all available memory on a computer, new ones are not queued up, but end up using "virtual memory," which is on the computer's hard drives. The problem is that virtual memory is much slower than real memory, and any tasks relegated there will run at a fraction of the speed that they would otherwise. In addition, if there is enough pressure on the physical memory of the computer, it might become entirely unresponsive, or crash outright.

Playing Nice

To avoid overwhelming our servers we ask you to limit your work to two simultaneous jobs per server. This policy applies to jobs that take several hours to complete. You may run many jobs that require a short time within reason--just don't queue up thousands of jobs at a time.

Since it is very difficult to monitor memory usage, we simply expect you to monitor your own jobs and avoid consuming more than half the memory on a given server (less if there are many others also running jobs).

Note that these rules are not strictly enforced as we can't watch the servers all the time looking for violators. If you notice anyone using too many resources and it is preventing you from doing work, notify us by email at 4help@umn.edu.

Shared software licenses. Many statistical packages are fairly expensive for individual licenses. CLA has negotiated agreements with many vendors and the University to offer packages free of charge on our application servers. This saves you, and the University, thousands of dollars in licensing costs.

Linux. We run application servers using a recent version of Ubuntu Linux. While Linux offers very powerful tools for computational researchers, it is not an ideal choice for everyday use, and most people prefer to use Windows or OSX. These servers offer you a way to make use of the capabilities of Linux without having to run it yourself.

Memory. While computer memory is affordable, it is still rare to find a personal computer with more than 8GB. At the same time, many researchers work with data sets that require larger amounts of memory. Our servers have 32 or 64GB memory, enabling you to work with large data sets. If you need more memory, and have (a lot of) funding for your research projects, we can buy servers with 768GB.

Shared file servers. Our application servers have fast access to centralized file servers where you can store your data. This eliminates the need to make multiple copies of your data and keep those copies synchronized. All data is backed up daily, offering security and reliability.

Public Application Servers

These machines are open to all account holders, without any limitations.

Private Application Servers

In addition to public servers, we operate private servers purchased by individual researchers or departments. These machines are configured identically to the public ones, but are restricted to account holders from sponsoring studies or departments.

If you are working with colleagues from external institutions they'll need to get access to the U of M network and our systems in one of two ways:

• Sponsored Internet account. This is a University service that gets you a full-fledged U of M Internet account for your collaborators, at a cost of $96 per person per year. Sponsored account holders can log in to CLA's systems as well as use the University's Virtual Private Network (VPN), and make use of certain other University services.
• "Service" account. These are pseudo-Internet accounts that can be created for free, but they are limited to accessing CLA servers and nothing else. We set these accounts with strict deadlines because they are difficult to track and manage.

While sponsored accounts are our preferred solution, it may not be possible for every research project, with many external participants, to pay hundreds of dollars a year for access. As the University continues progress on the Shibboleth implementation, we will be able to provide people from external higher education institutions the ability to log in to U of M systems using accounts from their home institutions. We will let you know once this option becomes available so we can quickly replace existing sponsored and service accounts.

• Servers use the same account and password for a given user. This means you only need to remember one set of credentials.
• Network shares defined on file servers are available on all member servers. For example, an instructor who needs to distribute data files to her students can save them to a central location and students can use any one of our servers, or their personal machines, to run programs and analyze the data.
• Access to resources, such as servers and network shares, can be restricted and controlled easily. For example, a research study collecting sensitive data can restrict access to its project folders or servers to the members of a certain group. When a member leaves the study, removing them from the group ensures their access rights are revoked, without visiting each machine or configuring each share individually.

In our implementation of the protocol, we have chosen to use the University's Internet ID system for authentication. Access to our systems does not require a separate account so once your user ID is enabled in our LDAP system, you will be able to log in to our servers with your University account.

The Unix environment is dedicated for use by researchers needing statistical analysis and data storage. We also operate a large number of database, web, and other servers that are "attached" to the same environment. Most of these servers do not allow direct logins, but provide access to the services they run. For example, you can connect to a database on a database server and use it, but you would not be allowed to log in because there is nothing else on the server that needs to be "exposed" to you.

We will focus on the individual components of the environment in upcoming posts. We invite any and all questions - let us know what you think or contact the CLA Service Desk at 612-624-4357 or 4help@umn.edu.