JaWS Software Admin News

test2

Sun, 10 Feb 2013 12:25:26 -0600

Testing again!

Test

Sun, 10 Feb 2013 10:14:28 -0600

Just testing some functionality.

Overhead Cabling moves, part 1

Thu, 05 Aug 2010 15:18:18 -0600

As part of WBOB's move from underfloor to overhead cabling for the entire WBOB data center the ethernet for JaWS Servers will need to be swapped to overhead routed cables that are already in place. Each move will result in a brief 1 minute interruption in network connectivity while the cables are swapped. The applications residing on the affected servers will be restarted if necesary, it should only be required for applications with persistent connections to external databases or NFS mounts.

Production servers will be recabled in multiple batches, one this Sunday (8/8) between 7:00a - 10:00a and another one next Sunday (8/15) between 7:00a - 10:00a. The 8/8 batch swadm responsibilities will be handled by Ben Larson, the 8/15 batch will be handled by Chris Ament.

Non production servers will be done during the week of 8/9 on Tuesday 8/10 7:00 - 9:00a with Ben Larson coordinating with OIA and on Wednesday 8/11 8:00a - 10:00a with Garrett Kuchta coordinating with OIA.

To determine when a server is slated to be moved please refer to the Google Doc viewable here. NOTE: all times after 8/15 are considered tentative pending CAB approval!

There are approved 'urgent' changes in Service Center for two of these production windows currently, the 8/8 window is change C67894 and the 8/15 window is change C67897 .

VPN hardware upgrade on March 23rd

Mon, 08 Mar 2010 09:54:13 -0600

NTS will be upgrading the VPN hardware on March 23rd. Details of the announcement are below.

On March 23, 2010, the existing VPN hardware and software will be
upgraded. A new Cisco VPN client will also be introduced for use with the
upgraded hardware and software. Depending on your needs and what OS you
currently run, you may need to upgrade your VPN client:

If you run on a 64 bit version of Windows, and/or you are using IPv6, you
will need to upgrade to the new Cisco VPN client (called AnyConnect). The
current VPN client software is referred to as EasyConnect.

NTS is running a test system now if you would like to try out the new VPN
client. The test hardware for both the AnyConnect and EasyConnect clients
is: anyconnect-cluster-test.vpn.umn.edu

L2TP is also an option for connecting to the UofMvpn workgroup. The
advantage of using L2TP is that it is native to Windows, Mac and Linux and
thus you don't need to install any additional software.
L2TP as implemented only supports UofMvpn workgroup. To use other
workgroups, you will need to use the AnyConnect or the EasyConnect client.
The L2TP client does not work behind firewalled desktop networks.
The test hardware for L2TP is: nct.vpn.umn.edu

Only use the UofMvpn VPN workgroup for testing with any of the clients for
now.

VPN workgroups are also called instances or tunnel groups.

Draft documentation and client software is available at:
https://netfiles.umn.edu/users/peiffer/

Please let me know if I can provide further information via oialinux@umn.edu.

Web Hotel Maintenance on Sunday, March 7th, 7:00-7:30AM

Fri, 19 Feb 2010 11:00:16 -0600

OIA will be performing maintenance on the Web Hotel server (warrior) on Sunday, March 7th, from 7-7:30AM. The work will include:

- Additional network interface/VLAN to support migration from tank to warrior
- Operating system patches

The outage is expected to last less than 5 minutes if work goes as planned. Please inform customers who are sensitive to downtime during this time period.

In preparation for this work, ultraman and pursuit will have operating system patches applied and be rebooted on Wednesday, March 3rd at 8:30AM.

Filenet hotel sites to be decommissioned

Thu, 28 Jan 2010 16:27:39 -0600

The following sites are set to be decommissioned 2:00p Tuesday February 2nd. They all live on the old filenet hotel (zoar/nebula) and the cnames either point off to other web servers or they merely serve redirects now. If you have questions or concerns about this please contact swadm.

www.hhh.umn.edu
www.cdes.umn.edu
www.academic.umn.edu
stpaulcareers.umn.edu
ssw.cehd.umn.edu
fsos.cehd.umn.edu
fscn-old.cfans.umn.edu
dha.cdes.umn.edu

Added 2-2-2010:
goldstein.cdes.umn.edu
rjp.umn.edu

Added 2-3-2010:
www.che.umn.edu

Changes to IPSca (free-but-valid) certs

Wed, 23 Dec 2009 12:51:58 -0600

The IPSca root CA under which all the IPSca free-but-valid .edu certificates issued to date have been created, and is widely supported in current browsers, expires 12-29-2009. The solution outlined by IPSca is to renew all IPSca issued certificates to utilize their newly issued CA's. I have submitted all current IPSca certs for renewal and begun installing the new certificates and certificate chain files to use their new CA's.

However, the new CA's are not recognized in the majority of browsers currently, meaning that these certs will flag as untrusted in most browsers. As of today the only two browsers that are supporting IPSca's new CA's are Internet Explorer and Chrome on Windows for users who have applied the optional windows updates since November 2009. Firefox is working on getting it included with no ETC.

If you have a site on the list below and need a valid cert for it then please write to certificates@software.umn.edu and request a Thawte. (Note that a new Thawte will cost $144 .)

List of sites with IPSca certs as of 12-23-09:

*.software.umn.edu
leaf.oit.umn.edu
motel-test.software.umn.edu
templates.umsurvey.umn.edu
timber.software.umn.edu
*.test-hotel.software.umn.edu
*.dev-hotel.software.umn.edu
test-status.umn.edu
wiki-test.software.umn.edu
mnscupachy-test.software.umn.edu
pachyderm.umn.edu
pachyderm-test.umn.edu
sphpachy.umn.edu
mnscupachy.umn.edu
sphpachy-test.umn.edu
test-hotel.software.umn.edu
sysstatdev.software.umn.edu
dev-status.umn.edu

Web Hotel failback complete

Mon, 26 Oct 2009 09:25:43 -0600

The transition of the Web & MySQL Hotel back to their respective production environments was completed at 7AM this morning. Please let us know if you notice any problems.

Web Hotel failover complete

Fri, 23 Oct 2009 07:25:13 -0600

The Web Hotel failover scheduled for 5AM this morning was completed at 5:30AM. Pong is was promoted to production web hotel server; badlands production MySQL hotel

IPF Change to production Solaris Hosts

Wed, 14 Oct 2009 16:07:34 -0600

OIA will be altering the IPF config on assassin, assault, kram, kengo, and stargate Thursday (10-15) morning at 9:00 am. This update to the firewall will allow the OITsec's scan appliance to ping the host but block full scans from happening. This will allow them to take our hosts off the exception list but should prevent our hosts from getting DoS'd as has happened several times in the past.

Please report any odd behavior from these hosts right away to support@software .

NAS storage maintenance this Sunday, Sept 27, 6:30AM

Wed, 23 Sep 2009 16:56:11 -0600

There will be a third (and final) planned NAS outage this Sunday beginning at 6:30am and lasting for approximately 2 hours. NFS mounted home directories will be unavailable during this time, and any login sessions may be disrupted. Please log out before you leave on Friday.

Affected mount points are:
adcs_stellent_nfs_04
DR_oia_linux_nfs_09
DR_LIB_NFS_01
jaws_backup
jaws_homedir_02
jaws_moodle_nfs_02
oia_webhost_nfs_06, 07,.08
UMWiki_NFS_02

MySQL Hotel upgrade, Bastion Hosts, and Server provisioning

Thu, 17 Sep 2009 12:36:14 -0600

There are some server infrastructure changes coming (or already here) that will affect everyone in one way or another. Details are on the wiki - I'll summarize them here and you can decide which parts are relevant to your work. Please come to the All-staff meeting on Monday with questions or concerns.

MySQL Hotel Upgrade:
We're upgrading our MySQL Hotel (venus) to new hardware with a master/slave configuration. The upgrade will occur on Sunday, October 11th, between 8-10AM. See https://wiki.umn.edu/view/Swadm/WebHotel#MySQL_Database for details including migration options.

Bastion Hosts:
SSH access to all OIT servers is restricted to either the OIT VPN or selected bastion hosts. We will be consolidating this infrastructure, effectively retiring the original JaWS bastions (frogger & naomi) in favor of the OIA bastions (ale01, ale02, and ale03). Access to the OIA bastions requires 2FA -- use of SSH keys is allowed and encouraged. There is no date set, but we are targeting having this completed before the end of the year. See https://wiki.umn.edu/view/Swadm/BastionHost for specifics.

Server Provisioning:
All new server installations (physical or virtual) are based on a new RHEL image that has some significant differences compared to the traditional JaWS image. Access and security policies are more restrictive. See https://wiki.umn.edu/view/Swadm/ServerConfig for additional detail and an inventory of differences.

Additional details on the upcoming WBOB outage from OIA

Wed, 16 Sep 2009 09:34:30 -0600

This has also been added to the wiki.
https://wiki.umn.edu/view/Swadm/WbobOutage10242009

-------- Original Message --------
Subject: [OITDCTENANTS] IT Outage on Saturday, October 24, 2009
Date: Wed, 16 Sep 2009 08:13:15 -0500
From: Joe Zanmiller
Reply-To: OITDC Tenants
To: OITDCTENANTS@LISTS.UMN.EDU

Beginning at 2:00 a.m. on Saturday, October 24, 2009, all University of
Minnesota IT systems and applications will be unavailable for use.
Beginning at 2:00 a.m., University of Minnesota IT staff from the Office
of Information Technology (OIT) and from departments with servers in the
WBOB Data Center will begin bringing down their applications with a goal
that systems and applications will be unavailable for use by 4:45 a.m.
The Data Center will powered off at 7 a.m. and will be available for
staff to restart applications on Sunday, October 25 at approximately1:00
a.m. All applications supported by OIT are expected to be restored to
service by 7:00 a.m. on October 25 and will become available on their
normal schedule.

OIT has created a Web site that will be continuously updated as we plan
for and execute this outage:
https://systemstatus.umn.edu/view/Main/OutageInfo.

PLANNED OUTAGE
The Office of Information Technology (OIT) has been planning for this
outage and communicating regularly with collegiate IT Directors. It is
absolutely necessary that we perform this work, as it will decrease the
risk of an unplanned outage to the University's IT applications during a
critical time, such as student registration or final exams.

OIT acknowledges that this is a very difficult time, and we recognize
the importance of the University's core systems. However, we ask you to
understand the need for us to perform this critical work, and we ask you
to help us by sharing this information with your students, staff,
faculty, and customers who will need to adjust their schedules to
accommodate this critical work.

TECHNICAL DETAILS
This outage is being scheduled so that OIT can increase the amount of
utility and emergency power to its primary data center, which is located
in the West Bank Office Building (WBOB). This will allow OIT to increase
the reliability, availability, and capacity of this data center.

All Enterprise and University-wide applications supported by OIT run on
servers and disk in the WBOB Data Center. In addition, servers and disk
for OIT Hosting Services are located in the WBOB Data Center. Finally,
several University departments are tenants in the WBOB Data Center and
have critical servers and storage there.

If we have an event causing the loss of all power (i.e., both utility
and generator backup power) in the WBOB Data Center, we currently do not
have the capability to provide uninterrupted availability of Enterprise
and University-wide Applications and many Departmental Applications.
Last summer, we had two catastrophic power-loss events in the WBOB Data
Center that caused applications to not be available for several hours By
upgrading the power system in this data center, OIT will be better able
to support the applications and systems that reside there with decreased
risk for unscheduled interruption to service. If we do not perform this
necessary work, the University will continue to be at considerably
greater risk for unplanned outages to Enterprise and University-wide
applications similar to the two outages in July and August 2008.

SHUTDOWN SCHEDULE FOR SATURDAY, OCTOBER 24
* No later than 4:45 a.m. All OIT Hosting customers and Data Center
tenants must have their applications and databases shut down.

* Beginning at 4:45 a.m.: Tenant staff will be given access to the Data
Center, if necessary, to power-off their equipment.

* No later than 5:45a.m.: All tenant equipment in the Primary Data
Center must be powered-off.

* Beginning at 5:45 a.m.: OIA will finish powering-off all of our
servers and the SAN.

* 7:00 a.m.: The Data Center will be turned over to the construction
engineers and contractor, and all power to the Primary Data Center will
be turned off. (Note: power should remain available in the WBOB building
during construction activities, except for 6th floor.)

RESTART SCHEDULE FOR SUNDAY, OCTOBER 25
* 1:00 a.m.: After OIT has restarted the network and the SAN, the Data
Center will be open for tenants to begin restarting their systems. At
this time, OIT will begin starting servers for our hosting customers and
for Enterprise Applications.

* Beginning at 3 a.m.: Servers will be available for Hosting Services
customers to restart their databases and applications. OIT engineers
will verify the functionality of all OIT-supported databases and
applications by 7:00 a.m. Applications will be available to customers
per the normal schedule on Sunday.

EARLY FINISH
OIT will make every effort to reduce the timeframe of this outage. If we
finish construction early, OIT staff will immediately begin the process
to restart, beginning with the network, SAN, and servers. If tenants and
Hosting Services customers have requested, OIT engineers will call
tenants when the SAN becomes available so they can restart their
servers, and OIT engineers will call Hosting Service customers when the
servers are ready for them to restart their databases and applications.
If you would like to receive this notification, reply to this e-mail and
provide a contact phone number and e-mail address that can be used the
day of the event.

COMMUNICATIONS DURING THE EVENT
During the outage, you can visit
https://systemstatus.umn.edu/view/Main/OutageInfo for up-to-date
information about the progress of the construction and system restart
activities. In addition, the System Status page will be updated
throughout the weekend to reflect the status of individual applications:
http://systemstatus.umn.edu/view/Main.

Please share this information with any staff or faculty that will need
to plan for this event.

Al Pierce
Senior Manager
Operations, Infrastructure, and Architecture Data Center Operations
Office of Information Technology

WBOB Data Center Outage on Saturday, October 24th

Thu, 10 Sep 2009 11:38:38 -0600

We've been getting an number of questions regarding the planned outage for the WBOB data center on Saturday, October 24th. The backup generator for the data center is being replaced with a larger unit that can handle the current and expected future demands on that facility. We don't have specifics on exact times at this point, but the expectation is that the entire facility will be shutdown for most of the day on Saturday, with service being restored as late as Sunday morning. We'll have more specific times in the coming weeks.

We're using this event as an opportunity to test our DR readiness for a number of services. High availability, redundancy, and recover-ability are on the minds of many of us and our customers. In some cases the necessary infrastructure is already in place. In others (read: most), we're not there yet, and we're working with application custodians and OIA to assess our options and prioritize work towards that end.

At the end of the day, many services will simply be unavailable during the outage. Our goal is to test what we have, fix what we can, and get insights into expectations and the scope of the work ahead for the others.

Additional Information will be posted on the wiki as it becomes available:
https://wiki.umn.edu/view/Swadm/WbobOutage10242009

NAS Maintenance this Sunday, Sept 13, 6-10AM

Tue, 08 Sep 2009 10:20:08 -0600

The storage group is performing maintenance on NAS infrastructure that will affect the NFS mounted home directories on many JAWS servers. Please close any open terminal sessions/logins no later than 11pm on Saturday evening to avoid data loss from open files.