|
|
|
| Browse NETS topics: A B C D E F G H I J K L M N O P Q R S T U V W X Y Z | |
PREP
|
Verify
|
|
|
|
Send outage notice 2 weeks in advance with specific details, include NCAB - NOAA |
|
|
Notify hosted users 2 weeks in advance of shutdown and scheduled times and request shutdown by 3pm on Friday. |
|
|
Remind NE that servers must be down by 5 a.m. for a 6 a.m. shutdown |
|
|
Machines will be powered up after all of our other equipment is done |
|
|
Check UCAR Room reservations to be sure there are no conferences that need wireless network access on the day of the shutdown. Email Allison Smith that the service will be unavailable. |
| Validate that time.nist.gov servers are causing a CPU spike on mlra when the servers are shutdown. If so, whats the best way to handle the cpu spike? Ride it out, null route to the servers, other? |
Friday after 3pm, shutdown:
|
Verify
|
Hosted Servers in NETS racks | Notify | |
|
|
Time | jlevine@boulder.nist.gov | Judah Levine |
|
|
COBmon | williamsp@ci.boulder.co.us | Paul Williams |
|
|
Echo | rapier@psc.edu | Chris Rapier |
|
|
wbu-us-ark.caida.org | monitor-info@caida.org | Caida |
|
|
Netmon (teragrid) | neteng@ncsa.uiuc.edu | NCSA |
Friday sometime
Power down test gear
|
Verify
|
Device | Rack |
|
|
movaz gear | BK49&48 |
|
|
ipt testbed | BM49 |
NAGIOS snapshot - see whats down before work begins.
MLRA snapshot: show interface summary, show ospf neighbors,
SHUTDOWN
Start powerdown at 5:00 a.m.
|
Verify
|
NETS Servers | Shutdown Process |
|
|
Netserver | Automatic / Scheduled |
|
|
Netstat | Automatic / Scheduled |
|
|
Netflow | Automatic / Scheduled |
|
|
Nagman | Automatic / Scheduled Validate fl-nagman can be reached |
|
|
WirelessDHCP (DMTest) | Automatic / Scheduled |
|
|
Nets-DB | Login / Manual |
|
|
NPAD | Automatic / Scheduled |
| Flowserver | Automatic / Scheduled |
|
Verify
|
IPT Servers | Shutdown Process |
|
|
IPTMON | Login / Manual |
|
|
NetsCM | Login / Manual |
|
|
NetsCM2 | Login / Manual |
|
|
NetsCER | Login / Manual |
|
|
IPTServ | Login / Manual - shutdown ISI application first !!! |
|
|
DSView | Login / Manual (needs to be available to shutdown other servers) |
Shutdown SERVER rack plug strip(s)
Depending on scheduled work, these switches may or may not be
shutdown:
|
Verify
|
Other Nets Equipment |
|
|
ml-mr-c6-gs |
|
|
ml-mr-c4-gs |
|
|
ml-mr-c3-gs |
|
|
ml-mr-c2-gs |
|
|
mlra |
|
|
ml-mr-c1-gw (terminal server) |
STARTUP
|
Verify
|
Other Nets Equipment |
|
|
ml-mr-c1-gw (terminal server) |
| mlra | |
|
|
ml-mr-c2-gs |
|
|
ml-mr-c3-gs |
|
|
ml-mr-c4-gs |
|
|
ml-mr-c6-gs |
ALL Network Equipment must be up and available BEFORE starting servers
|
Verify
|
|
|
|
Check network connectivity, verify MLRA snapshot info |
|
|
Check etherchannel trunks and connections
(mlra cards 2 & 3) specifically fb-12-c1-es, fb-12-c2-es, ml-34-c1-es, ml-350-c1-es, and ml-gen-gc1-es |
|
|
Re-Establish BGP on NOAA wireless link http://netserver.ucar.edu/nets/internal/projects/wireless/ml-fl/documentation/noaa-proxim-information.html#bgp |
Turn on rack plug
strip(s)
Turn on Servers
|
Verify
|
NETS Servers |
|
|
DSView |
|
|
Netserver |
|
|
Netstat |
|
|
Netflow |
|
|
Nagman - verify nagios |
|
|
WirelessDHCP (DMTest) |
|
|
Nets-DB |
|
|
NPAD |
| Flowserver |
IPT Servers
Restart
in this order, verifying that each is completely operational before
moving to next number.
Perform IPT checklist when complete.
http://netserver.ucar.edu/nets/projects/voip/documentation/internal/change-and-outage-checklist.html
| Verify | ||
| IPTMON | 1 | |
| NetsCM | 2 | |
| NetsCM2 | 3 | |
| NetsCER | 4 | |
| IPTServ | 4 | |
| Restart NetsCM3 | 4 |
Start up Hosted servers
| Verify | Hosted Servers in NETS racks |
| Time | |
| COBmon | |
| Echo | |
| wbu-us-ark.caida.org | |
| Netmon (teragrid) |
| Verify | |
| Follow up with Nagios |