Director's Blog

Well, we ran into yet another issue with the Cisco ACE appliances. In order to effectively troubleshoot the issue, we are rolling back the DNS change we made yesterday.

I apologize (again) for the delay on this. If we can't get the Cisco ACE to work properly, we'll update the Cisco CSS to use the new web servers. That way, you can still get the benefit of the hardware and software upgrade.

I'll post more information tomorrow after we conduct more testing.

Well, we updated the DNS records today for the 'www' and the 'ozone' sites, but only for internal users. Client browsers outside of the OHSU network are still using the older servers. (We'll update external DNS in a week or so.)

We've run into a few configuration issues that were missed, but we've also been resolving them as they are reported.

If you are experiencing any problems with the site, please contact the ITG Helpdesk or Web Strategies.

Thanks for your patience!

We're going to launch on Monday, May 18th at 9:30am. Hopefully, this should be a seamless transition this time. I've reviewed the old CSS rules line by line (all 5186 of them). We're very confident that the new servers are now configured to support every URL that the old servers were supporting.

Wish us luck! And of course, be ready to test your websites Monday morning!

I just discovered that the IT engineers who can update the DNS servers (needed to launch Tier II and revert back if there are major problems) will all be busy today taking examinations. Therefore, we are re-scheduling the launch for Monday, May 4th, at 11am.

We are planning to actually launch the upgraded servers on Thursday, April 30th at 10:00am. I really, really, really hope I don't have to retract this announcement!

Damon

The Network Engineering team will be applying a firmware patch to the Cisco ACE network appliance modules on Thursday, April 23rd. Web Engineering would like to wait a few days after this patch to ensure that the ACE modules are stable after the upgrade before we make the DNS change that send traffic for Tier II web sites to the ACE.

Thanks for your patience!

I'm sorry to have to report that the Tier II upgrade has once again been postponed. I probably jumped the gun a little on announcing that we would launch yesterday.

The issue at hand is that the Cisco ACE (load balancer) is having some problems. Specifically, one of our VIPs (Virtual IP addresses) is failing, and we don't know why. The failing VIP happens to be the one that serves http://dev.ozone.ohsu.edu. The 'production' VIPs seem to be working. However, until we identify the root cause of this issue, I'm worried that the same issue might affect a production VIP. Therefore, we are once again postponing the launch.

I will keep this blog updated with status reports as often as I can.

Thanks in advance for your patience!

Ok all. We are finally ready to launch the new upgraded Tier II servers. At 11am today (or shortly thereafter), the DNS records for

www.ohsu.edu and ozone.ohsu.edu

will be updated for INTERNAL network clients only. Anyone on the internal network browsing the OHSU or O-Zone sites will be using the new servers. All external web browsers will continue to view our site on the old servers.

If you experience any issues with the new servers, please follow our standard support notification process by completing our support form at: https://www.ohsu.edu/webstrategies/support.cfm.

Thanks for your patience!

UPDATE:

Yet another postponement. I apologize for the series of delays here. This time, the network engineering team has discovered a bug with the Cisco ACE appliances. They are working with the vendor to get the issue resolved.

Currently, they are expecting to have the ACE issue fixed later this week. I'll keep this blog updated with status as soon as we see a change.

For now, Tier II is still running CFMX 6 on the old 'cluster'.

Hi all. We discovered an issue with the NFS Mounts and symbolic links that would have affected lots of applications in the production environment. We should be resolving that today and will begin testing as soon as possible.

At the moment, the revised ETA for Tier II launch is Tuesday, April 7th.