UC San Diego Menu

PCI Network outage

Status: info

Update: Update: Returning PCI firewall service to M-core was not successful last night, so service remains on B-core. We will schedule another change and troubleshooting session at a later time.

14:46:36, 26 August 2015

Update: Update: Now that Datacom has diagnosed and addressed the issues with M-core, tonight (Tuesday) at midnight, we will return the PCI networks to the primary firewalls in M-core. This is expected to be transparent but if there are any unexpected issues, we will return PCI firewall service to B-core and plan for additional troubleshooting at a later time.
14:39:57, 25 August 2015

Update: Update: In addition to the PCI outage previously described, PCI activity was intermittently impacted due to the M-core connectivity issues yesterday afternoon. These were resolved at approximately 20:30 on Sunday. PCI firewalling remains on the backup firewalls in B-core until we can schedule a time to return operations to M-core.

11:23:02, 24 August 2015

Incident report: All PCI firewalled networks were unavailable from approximately 11:11 until 13:03 this afternoon as Datacom is having problems with some networks resulting from their M-core upgrade earlier today. For now, all PCI connectivity is routed through the redundant links at B-core until the M-core issues are resolved.

We apologize for this inconvenience.
13:53:19, 23 August 2015

Contact for this incident report: security@ucsd.edu

This incident has been open for about 7 days

Connectivity to on-campus sites from off campus

Status: info

Update: This should be fully resolved. Please report problems to the ACT Help Desk.
19:38:15, 23 August 2015

Update: We believe we have resolved this issue. Please contact the ACT Help Desk to report any network issues.
19:37:11, 23 August 2015

Update: The team has made some promising progress. Hopefully will have another update soon.

Thank you for your patience and we apologize for the inconvenience.


19:05:45, 23 August 2015

Update: We're still troubleshooting this.

Sorry for the delay/problem. We have our best folks on it.
16:53:12, 23 August 2015

Incident report: We have some reports of connectivity problems to on-campus websites from off campus providers (uverse, cox).

Other off-campus people are reporting no problems at all.

We are currently investigating.

James Seddon
Datacom
14:03:17, 23 August 2015

Contact for this incident report: datacom-mgmt-l@ucsd.edu

This incident has been open for about 7 days

Network Upgrade

Status: info

Update: The work is complete and all production services should be up and running.

Please notify the ACT Help Desk if you are experiencing network trouble.
11:16:45, 23 August 2015

Update: There was a problem with Resnet's routing that caused it to experience an outage during the work. That should be resolved.

Additionally, EBU1, EBU2, and SERF may also have experienced an outage.

Lastly, there was an intermittent problem with the Health Systems connection with campus that caused several briefer outages. This also should be resolved at this time.

We are looking into the root causes of these.
10:45:16, 23 August 2015

Incident report: START: Sunday August 23, 5:00 A.M.
STOP: Sunday August 23, 11:00 A.M.

SCOPE: This maintenance will involve the physical hardware replacement of the primary campus backbone Node-M-Core chassis.

IMPACT: During this maintenance the campus routing path will migrate to the secondary Node-B Core routing path. There may be a brief disruption while services and routing fail over. The campus should retain full connectivity to other parts of the campus and Internet via that backup path once all connectivity has failed over to the secondary path.

Once the upgrade has been completed service and routing will migrate back to the primary path via Node-M Core.
13:57:01, 20 August 2015

Contact for this incident report: Janet Keith

This incident has been open for about 10 days

Urgent AD/Exchange Patch Installation 7/20 (9:00PM - 1:00AM)

Status: info

Incident report: An urgent security patch will be applied to our systems between the hours of (9:00PM - 1:00AM). Services will continue to be online, but there may be intermittent minor interruptions during this time.
17:39:57, 20 July 2015

Contact for this incident report: adteam@ucsd.edu

This incident has been open for about 41 days