30 Emergency Repair Plan for Automation System 自动化系统应急抢修预案

AALRT
AALRT Enterprise Standard
Special Response Plan
Approved by:
()
Checked by (Law):
()
Checked by (English):
()
Checked by (Chinese):
Reviewed by: Sun Hongyu
2015.08.20
Date: August 20, 2015
Prepared by: Wu Hailong
Dept.: Maintenance Center
2015.7.30
Date: July 30, 2015
A
Version: A
Revised by:
Revised on:
Controlled State: Internal

Control
Issue No.:
2015.08.30
Issued On: August 30, 2015
2015.11.20
Implemented on: Nov.20,2015
A
Emergency Repair Plan for Automation
System (Rev. A)
E/AALRT 1030-2015
1 34
Page 1 of 34
AALRT

Contents
1.
1. PURPOSE 3
2.
2. SCOPE
3.
3. REFERENCES 3
4.
4. DEFINITIONS AND TERMS
5. 4
5. PMIS EMERGENCY ORGANIZATION AND RESPONSIBILITY 4
6.
6. KEY GUIDELINES 6
7.
7. EMERGENCY PLAN FOR PMIS

8.
34
8. APPENDIX 34
A
System (Rev. A)
E/AALRT 1030-2015
2 34
Page 2 of 34
AALRT
Emergency Repair Plan for Automation System

1.
1. Purpose
ICT
AALRT
In order to define the division of responsibility and processing requirements for the sudden fault of
production management information system (PMIS) of ICT Division of Maintenance Center and specify
the emergency management and emergency response procedure for the sudden fault of PMIS, so as to
timely and effectively conduct emergency repair, minimize the influence of the information system on
production and guarantee safe, efficient and stable operation of various AALRT systems, hereby this
plan is prepared.
2.
2. Scope
AALRT ICT
This emergency plan applies to emergency repair of sudden fault of PMIS under the managing scope of
the ICT Division of AALRT Maintenance Center.
3.
3. References
Rules for Organization of Train Operation

4.
4. Definitions and Terms
4.1
4.1 Definition of sudden fault
A sudden fault refers to an unforeseeable or uncontrollable fault of the information system that may
cause the following one or more results and must be handled immediately:
1
A
System (Rev. A)
E/AALRT 1030-2015
3 34
Page 3 of 34
AALRT
1) The important PMIS software doesnt work properly;
2
2) The hardware running important PMIS doesnt work properly;
3
3) The backbone network doesnt work properly;
4
4) The software doesnt work properly for error;
5
5) The hardware doesnt work properly for fault;
6
6) Other faults that severely impact the normal functioning of the information system
4.2
4.2 Terms
Oracle oracle database, oracle RDBMS, oracle,
Oracle database: Oracle database, which is also called oracle RDBMS or oracle for short, is a
relationship information management system developed by Oracle Corporation.
Internet
Internet: The Internet is a logical global system of interconnected computer networks that use the
universal protocol suite to link several billion devices worldwide.
Intranet
Intranet: As a computer network using the same technologies with the Internet, an intranet is usually
established inside an enterprise or organization to provide services such as information sharing and
exchanging for its staff.
VPNVirtual Private Network
VPNVirtual Private Network: A VPN extends a private network across a public network. It enables
secure communication and remote access through packet encryption and the translation of packet
destination address.
5.
5. PMIS emergency organization and responsibility
5.1
5.1 Organizational structure of emergency repair group
ICT
ICT Division shall be responsible for organizing the PMIS emergency treatment.
Group leader: Director of the Maintenance Center

A
System (Rev. A)
E/AALRT 1030-2015
4 34
Page 4 of 34
AALRT
ICT
Deputy group leader: Manager of ICT Division
ICT
Members of the emergency repair group: Members of ICT Division
5.2
5.2 Responsibility
5.2.1
5.2.1 Responsibilities of the leader of the emergency repair group
1
Be responsible for emergency repair, technical support as well as organizing, command and decisionmaking of the emergency treatment;
2
Be responsible for cross-departmental communication and coordination in the emergency treatment;
3
Be responsible for decision-making for important issues in the emergency treatment;
4
Be responsible for reporting the emergency treatment process to the leaders of the department and the
Center;
5
Be responsible for organizing analysis of the emergency as well as implementing relevant
responsibilities and corrective measures.
5.2.2
5.2.2 Responsibilities of the members of the emergency repair group
1
Be responsible for emergency repair, to recover the fault as soon as possible and make the system
work properly;
2
Timely report the progress to the leader of the emergency repair group;
3
Be responsible for predict the influence of the fault and carry out according treatment;
Be responsible for fault recovery test to recover the system to normal state;
Take part in analysis of the emergency;
Be responsible for the implementation of corrective measures.

5.3
5.3 Treatment principles
A
System (Rev. A)
E/AALRT 1030-2015
5 34
Page 5 of 34
AALRT
Treatment principles for emergency repair

1)
Stick to the principle of rapid response and decisive treatment. The emergency force shall rapidly arrive
at the site, quickly start the work and be decisive and accurate in the treatment so as to minimum the
influences of system fault on operation and production;
2
Stick to the principle of carefulness and leaving no hidden danger. After the emergency treatment,
professional staff shall be organized to carefully inspect the fault and conduct necessary functional tests
to guarantee the normal functioning of the system;
3
Guarantee data security. In the process of emergency treatment, complete security protection of data
shall be conducted to effectively prevent data from being destroyed;
4
Guarantee unimpeded network. In the process of emergency treatment, network fault shall be
controlled and isolated to ensure that other lines shall not be affected.
6.
6. Key guidelines
1 24
The emergency personnel shall keep their mobile phones available for 24 hours;
2 AALRT
When an emergency takes place, it must be reported timely and truly in accordance with the AALRT
information report procedure;
3ICT
ICT Division shall always prepare some spares that have important effect on equipment system in the
spares warehouse;
4 1.5
The emergency repair staff must arrive at the site within 1.5 hours after they receive the report of
emergency;
5
The emergency repair staff shall keep close contact with each other to master the progress of the whole
emergency treatment and keep record;
6
After the fault is recovered, the leader of the emergency repair group or relevant executives shall
conduct necessary confirmation and test to guarantee that that the system can work reliably and stably
after recovery.
A
System (Rev. A)
E/AALRT 1030-2015
6 34
Page 6 of 34
AALRT
7.
7. Emergency plan for PMIS
7.1 ORACLE
7.1 Fault treatment procedure for ORACLE database
7.1.1
7.1.1 Fault symptom
ORACLE ORACLE
ORACLE EAM
Software fault of ORACLE database refers to the phenomena caused by problems of abnormal system
breakdown of ORACLE database, abnormal startup instances and that application system cant
connect the database and so on. The concrete manifestation is: application systems based on
ORACLE database, such as EAM system, statistical analysis system and track area construction
operation system, cant be logged in normally, while other systems not based on the database (e.g.
websites, mailbox) can.
7.1.2
7.1.2 Influence of the fault
ORACLE
ORACLE database software fault mainly has the following influences:
1EAM
EAM system cant be normally logged in, the work order cant be entered and the statements cant be
correctly calculated;
2
The statistical analysis system cant be normally logged in and the data cant be calculated;
3
The track area construction operation system cant be logged in and the construction operation has to
stop.
7.1.3
7.1.3 Guidelines for fault treatment
1 ICT EAM
SQLPLUS ORACLE test
ORACLE
The user shall immediately inform ICT Division after discovering the fault and the professional engineer
shall immediately try to log in the EAM system and track area construction operation system and so on;
if the systems cant be logged in, try to log in the test user of the ORACLE database by SQLPLUS; and
if the database cant be connected normally, the professional engineer can preliminarily recognize that it
is ORACLE database fault and shall immediately inform the administrator of the database system.
2 2
The administrator of the database shall estimate the fault symptom. If the fault is recognized as
A
System (Rev. A)
E/AALRT 1030-2015
7 34
Page 7 of 34
AALRT
software crash of database system which cant be recovered within 2 hours, immediately launch the
emergency plan, start the standby system, report to the leader of the emergency repair group and tell
other group members to conduct emergency repair. If the fault is recognized as host hardware fault of
the database system which cant be recovered within 2 hours, immediately launch the emergency plan,
start the standby system, report to the leader of the emergency repair group and conduct emergency
repair toward the hardware. As for fault cant be recovered by replacing spare parts, immediately ask
the hardware supplier for support;
3 2
RMAN
After arriving at the site, the emergency repair group shall immediately conduct emergency repair to the
host of the database system. As for relatively serious fault which cant be repaired within 2 hours,
immediately report to the leader of the emergency repair group. After the discussion of the group
members, the group shall conduct RMAN repair or logical backup data recovery according to the fault
condition;
4 2 EAM
If the database fault cant be repaired within 2 hours, inform the dispatchers of various professional
centers and related functional departments that they shall apply the emergency plan of this department
to systems including EAM system, statistical analysis system, and track area construction operation
system and so on.
5
After the database system fault has been recovered, all departments shall add the data generated
during the emergency repair in various production management systems.
7.1.4
7.1.4 Action guidelines for posts
Posts
AALRT
AALRT
users
Action Guidelines
IT
IT
dispatching
console
Fault report:
AALRT EAM ICT
When the users of various AALRT departments cant normally log in EAM system or
statistical analysis system, immediately inform ICT Division;
Fault diagnosis:
EAM
SQLPLUS ORACLE test
ORACLE
The professional engineer shall immediately try to log in the EAM system and track area
construction operation system and so on; if the systems cant be logged in, try to log in
the test user of the ORACLE database by SQLPLUS; and if the database cant be
connected normally, the professional engineer can preliminarily recognize that it is
ORACLE database fault.
Fault report:
A
System (Rev. A)
E/AALRT 1030-2015
8 34
Page 8 of 34
AALRT

The professional engineer shall immediately inform the administrator of the database
system. After the database administrator recognizes the fault, the professional engineer
shall inform the leader of the emergency repair group, announce the launching of the
emergency plan according to the instruction of the group leader, and issue the
emergency repair instruction to the members of the emergency repair group.
Information transmission during the fault:

EAM
According to the instruction of the leader of the emergency repair group, inform relevant
functional departments and the dispatchers of various professional centers that they shall
deactivate EAM system, statistical analysis system and track area construction operation
system and that various departments shall apply the emergency plan of this department.
Information transmission after fault recovery:

1
Inform relevant functional departments and the dispatchers of various professional
centers that the fault has recovered;
2

centers that they shall enter the data generated during emergency repair in various
PMISs.
Fault record:
Emergency repairer, time of emergency repair, fault symptom, notification time and the
process of emergency repair and recovery.

/
Emergency
repair
group/group
leader
:
Fault treatment:
1)
The group leader takes change of organizing and coordinating the group members to
conduct specific emergency repair operations;
2)
After receiving the notification, the group members shall immediately rush to the central
apparatus room and tell the technicians of the group to rapidly leave for the destination
for joint treatment, and the group leader shall organize the members to complete the
emergency repair;
3) 2
The group members shall analyze the fault. If the fault is recognized as software crash of
database system which cant be recovered within 2 hours, immediately start the standby
database system and report to the leader of the emergency repair group; and if the fault
A
System (Rev. A)
E/AALRT 1030-2015
9 34
Page 9 of 34
AALRT
is recognized as host hardware fault of the database system which cant be recovered
within 2 hours, immediately start the standby database server, report to the leader of the
emergency repair group and conduct emergency repair toward the hardware. As for fault
cant be recovered by replacing spare parts, immediately ask the hardware supplier for
support.
Fault recovery:
1)
The leader of the emergency repair group shall arrange some members to check if the
host system software has all recovered and if the data is complete;
2)
If the logical backup recovery mechanism is applied, check if the data in the database
has recovered to the data before the system interrupt; if not, the leader of the emergency
repair group shall organize manpower to complete the data;
3)
If the backup host is applied, after the host fault has recovered, the data during the
suspension period needs to be renewed to the host;
4)
service software works properly;
5)
The group members shall inform the professional engineer that the system has
recovered.
Emergency repair completed:

1
Announce that the emergency repair is completed, send the message of emergency
repair completed to the members of the emergency repair group;
2)
The leader of the emergency repair group shall report the process to the superior.
3)
The leader of the emergency repair group shall organize the members to look up and
analyze the cause of the fault, prepare a correction plan and write an analysis report of
the event. The analysis report shall include the following contents: the original record of
the fault treatment process, the description of the fault and the fault treatment. Besides,
the report shall clearly state if the fault treatment is accurate and timely and if there is any
obvious fault or noncompliant behavior. The report shall be concise and to the point and
has a clear and logical description of the situation.
7.2 EAM
7.2 Fault treatment procedure for EAM system application server
7.2.1
7.2.1 Fault symptom
A
System (Rev. A)
E/AALRT 1030-2015
10 34
Page 10 of 34
AALRT
EAM EAM
SQLPLUS ORACLE test
EAM system application server fault means that the EAM system doesnt display the login interface and
cant be logged in normally, or cant be operated normally after login, and the data cant be read or
saved, but the professional engineer can log in the test user of ORACLE database by SQLPLUA.
7.2.2
EAM EAM
The EAM system cant be logged in, so the EAM application services like work order, supplier,
equipment and statement cant be used normally.
7.2.3
1 ICT EAM
SQLPLUS ORACLE test EAM
EAM
shall immediately try to log in the link of the EAM system application server; if the system cant be
logged in, try to log in the test user of the ORACLE database by SQLPLUS; and if the test user can be
connected normally, the professional engineer can preliminarily recognize that it is EAM system
application server fault and shall immediately inform the administrator of the EAM application system.
2EAM EAM
After the administrator of the EAM application system confirm that the EAM application server has
hardware fault or serious software fault, he shall inform the leader of the emergency repair group, and
tell other group members to conduct emergency repair;
3 1
EAM
EAM EAM
After the members of the emergency repair group arrive at the site, they shall immediately conduct
emergency repair. If the fault cant be repaired within 1 hour, they shall immediately report to the group
leader and inform the dispatchers of various professional centers and relevant functional departments
that they shall apply the emergency plan of this department to relevant EAM services during the repair.
The members of the emergency repair group shall immediately establish the EAM application system in
the virtual system so recover the EAM application in the shortest time.
4 EAM
After the recovery of the fault, various departments shall add the data generated during the repair in the
EAM system.
7.2.4
Posts
AALRT
Action Guidelines
Fault report:
A
System (Rev. A)
E/AALRT 1030-2015
11 34
Page 11 of 34
AALRT
AALRT
AALRT EAM
users
ICT
When the EAM system doesnt display the login interface and cant be logged in
normally, or cant be operated normally after login and the data cant be read or saved,
the AALRT user shall inform the ICT Division.
IT
IT
dispatching
center
Fault diagnosis:
EAM SQLPLUS
ORACLE test EAM
The professional engineer shall immediately try to log in the link of the EAM system
application server; if the system cant be logged in, try to log in the test user of the
ORACLE database by SQLPLUS; and if the test user can be connected normally, the
professional engineer can preliminarily recognize that it is EAM system application server
fault.
Fault report:
EAM EAM EAM
The professional engineer shall immediately inform the administrator of the EAM
application system. After the administrator of the EAM application system confirm that the
EAM application server has hardware fault or serious software fault, he shall inform the
leader of the emergency repair group and tell other group members to conduct
emergency repair;

EAM
functional departments and the dispatchers of various professional centers that they shall
apply the emergency plan of this department.
Information transmission after the fault recovery:

2
3

PMISs.
Fault record:
Repairer, time of repair, fault symptom, notification time and the process of emergency
repair and recovery.

/
Emergency
Fault treatment:
A
System (Rev. A)
E/AALRT 1030-2015
12 34
Page 12 of 34
AALRT
repair
1)
group/group The group leader takes change of organizing and coordinating the group members to
leader
2)
emergency repair;
3) EAM 1
EAM
EAM
EAM
The group members shall analyze the fault. If the fault is the EAM system application
server crash which cant be repaired within 1 hour, immediately report to the group leader
and inform the dispatchers of various professional centers and relevant functional
departments that they shall apply the emergency plan of this department in relevant EAM
services during the repair. The members of the emergency repair group shall immediately
establish the EAM application system in the virtual system so recover the EAM
application in the shortest time.
Fault recovery:
1)
The group leader arrange some members to check if the host system software has all
recovered;
2)
The group members shall inform the professional engineer that the system has
recovered.

1
2
The leader of the emergency repair group shall report the process to the superior;
3

the report shall clearly state if the fault treatment is accurate and timely and if there is any
obvious fault or noncompliant behavior. The report shall be concise and to the point and
has a clear and logical description of the situation.
7.3
7.3 Fault treatment procedure for server hardware
A
System (Rev. A)
E/AALRT 1030-2015
13 34
Page 13 of 34
AALRT
7.3.1
7.3.1 Fault symptom
AALRT AALRT EAM
Server hardware fault refers to the phenomenon caused by problems that the server hardware cant
work properly and the application system cant connect the database and so on. The concrete
manifestation is: the user cant normally log in the application systems of AALRT website, AALRT
mailbox system, EAM system, statistical analysis system, track area construction operation system and
so on.
7.3.2
The main influences of hardware fault of information system server are as follows:
AALRT
AALRT website cant be logged in normally;
AALRT
AALRT mailbox cant be logged in normally;
3EAM
EAM system cant be logged in normally;
4
Statistical analysis system cant be logged in normally;
5
Track area construction operation system cant be logged in normally;
6
The financial management system cant be logged in normally.
7.3.3
1 ICT
The user shall immediately inform ICT Division after discovering the fault, and the professional engineer
shall immediately try remote login of related server; if server cant be accessed, the professional
engineer can preliminarily recognize that it is server fault and shall immediately inform the network
system administrator;
2 1
The network system administrator shall estimate the fault symptom. If the fault is recognized as server
hardware fault which cant be recovered in 1 hour, immediately launch the emergency plan, start using
the spare parts, report to the leader of the emergency repair group and conduct emergency repair
toward the hardware. As for fault cant be recovered by replacing spare parts, immediately ask the
hardware supplier for support
A
System (Rev. A)
E/AALRT 1030-2015
14 34
Page 14 of 34
AALRT
3 1
After arriving at the site, the members of the emergency repair group shall immediately conduct
emergency repair to the server hardware. As for relatively serious fault which cant be repaired within 1
hour, immediately report to the leader of the emergency repair group. After the discussion of the group
members, the group shall repair the server hardware according to the fault condition;
4 1
As for server hardware fault that cant be repaired within 1 hour, inform the dispatchers of various
professional centers and relevant functional departments that they shall apply the emergency plan of
this department to relevant information management system.
5
After the database system fault has been recovered, all departments shall add the data generated
during the emergency repair in various production management systems.
7.3.4
Posts
Action Guidelines
AALRT
AALRT
users
Fault report:
AALRT ICT
When the AALRT departments cant log in the information management systems
normally, inform ICT Division.
IT
IT
dispatching
console
Fault diagnosis:
When the professional engineer shall immediately try to log in relevant server but the
system cant be logged in normally, he can preliminarily recognize that it is server
hardware fault;
Fault report:
IT IT
The professional engineer shall immediately inform the IT administrator. After the IT
administrator recognizes the fault, the professional engineer shall inform the leader of
the emergency repair group, announce the launching of the emergency plan according
to the instruction of the group leader, and issue the emergency repair instruction to the
members of the emergency repair group.
functional departments and the dispatchers of various professional centers that they
shall deactivate the information management system for fault and that various
departments shall apply the emergency plan of this department.
A
System (Rev. A)
E/AALRT 1030-2015
15 34
Page 15 of 34
AALRT

1
2

PMISs.
Fault record:
Emergency repairer, time of emergency repair, fault symptom, notification time and the
process of emergency repair and recovery.
/
Emergency
repair
group/group
leader
Fault treatment:
1)
2)
for joint treatment, and the group leader shall organize the members to finish the
emergency repair;
3) 2
The group members shall analyze the fault. If the fault is recognized as server hardware
fault which cant be recovered within 2 hours, immediately start the standby equipment
and report to the leader of the emergency repair group and conduct emergency repair
toward the hardware. As for fault cant be recovered by replacing spare parts,
immediately ask the hardware supplier for support.
Fault recovery:
1)
server hardware has all recovered and if the data is complete;
2)
If the hardware recovery mechanism is applied, check if the data in the server has
recovered to the data before the system interrupt; if not, the leader of the emergency
repair group shall organize manpower to complete the data;
3)
If a standby server is applied, after the replacement of the server, import the backup
data to the host;
4)
A
System (Rev. A)
E/AALRT 1030-2015
16 34
Page 16 of 34
AALRT
hardware works properly;
5)
The group members shall inform the professional engineer that the system server has
recovered.
1
2)
The leader of the emergency repair group shall report the process to the superior;
3)
the report shall clearly state if the fault treatment is accurate and timely and if there is
any obvious fault or noncompliant behavior. The report shall be concise and to the point
and has a clear and logical description of the situation.
7.4
7.4 Fault treatment procedure for extranet access device of office network system
7.4.1
7.4.1 Fault symptom
VPN
AALRT
Extranet access device fault of office network system refers to network fault caused by hardware errors,
software vulnerabilities, network intrusion and environment factors and so on. The concrete
manifestation is: the extranet cant be accessed normally (e.g. the users cant access extranet websites
or receive the external mails and the VPN remote users cant log in the AALRT intranet, etc.).
7.4.2
Extranet access device fault of office network system mainly has the following influences:
1
The egress firewall breaks down, so the users cant access the extranet normally;
2
The Internet access authentication system breaks down, so the users are unable to be authenticated to
access the extranet;
3VPN AALRT
VPN remote users cant log in AALRT intranet;
4
A
System (Rev. A)
E/AALRT 1030-2015
17 34
Page 17 of 34
AALRT
The intranet users cant receive extranet mails.
7.4.3
1 ICT
shall immediately try to access the extranet; if the extranet cant be accessed, he shall try to log in the
device via remote tools; and if he fails again, the professional engineer can preliminarily recognize that
it is extra device fault and shall immediately inform the network system administrator.
2
1
1
The network system administrator shall estimate the fault symptom. If the fault is recognized as
software program error of extranet device (firewall and Internet access authentication system) which
cant be recovered within 1 hour, immediately launch the emergency plan, start the standby device,
report to the leader of the emergency repair group and tell other group members to conduct emergency
repair. If the fault is recognized as hardware fault or crash of the extranet device which cant be
recovered within 1 hour, immediately launch the emergency plan, start the standby device, report to the
leader of the emergency repair group and conduct emergency repair toward the hardware. As for fault
cant be recovered by replacing spare parts, immediately ask the device supplier for support;
3 1
extranet device. As for relatively serious fault which cant be repaired within 1 hour, immediately report
to the leader of the emergency repair group. After the discussion of the group members, the group shall
conduct troubleshooting or configuration data backup recovery according to the fault condition;
4 1
If the extranet access device fault cant be repaired within 1 hour, tell the dispatcher of various
production departments and relevant functional departments to apply the emergency plan of this
department;
5
After the extranet access device fault is recovered, inform the dispatchers of various production
departments and relevant functional departments that the extranet access has recovered.
7.4.4
Posts
Action Guidelines
AALRT
AALRT
users
Fault report:
AALRT ICT
When the AALRT centers cant access the extranet normally, inform ICT Division.
A
System (Rev. A)
E/AALRT 1030-2015
18 34
Page 18 of 34
AALRT
IT
IT
Fault diagnosis:
dispatching

console
The professional engineer shall immediately try to access the extranet but fails, and then
he try to log in the extranet device (egress firewall and Internet access authentication
system) via remote tools:
If the device cant be accessed normally, the fault can be preliminarily recognized as
extranet device hardware fault;
If the device can be logged in normally, the fault can be preliminarily recognized as link
failure or configuration protocol error.
Fault report:
The professional engineer shall immediately inform the network system administrator.
After the network system administrator recognizes the fault, the professional engineer
shall inform the group leader of the emergency repair group, announce the launching of
the emergency plan according to the instruction of the group leader, and issue the
functional departments and the dispatchers of various professional centers that the
extranet cant be accessed and that they shall apply the emergency plan of this
department.

centers that the fault has been eliminated and the access to extranet has recovered.
Fault record:
/
Emergency
repair
group/group
leader
Fault treatment:
1)
2)
A
System (Rev. A)
E/AALRT 1030-2015
19 34
Page 19 of 34
AALRT
After receiving the notification, the group members shall immediately rush to the network
computer room and tell the technicians of the group to rapidly leave for the destination
emergency repair;
3)
1
1
The group members shall analyze the fault symptom. If the fault is recognized as
software program error of extranet device (egress firewall and Internet access
authentication system) which cant be recovered within 1 hour, immediately start the
standby device and report to the leader of the emergency repair group; if the fault is
recognized as hardware fault or crash of the extranet device which cant be recovered
within 1 hour, immediately start the standby device, report to the leader of the
emergency repair group and conduct emergency repair toward the hardware. As for fault
cant be repaired by replacing spare parts, immediately ask the device supplier for
support;
Fault recovery:
1)
extranet device has all recovered and if link communications have recovered;
2)
If the configuration backup recovery mechanism is applied, check if the configuration

data has recovered to the configuration before the system interrupt; if not, the leader of
the emergency repair group shall organize manpower to complete the configuration;
3)
If the backup device is applied, after the primary device fault has recovered, the
configuration files need to be imported to the primary device;
4)
service modules work properly;
5)
The group members shall inform the professional engineer that the network has
recovered.

1)
2)
The leader of the emergency repair group shall report the process to the superior
3)
A
System (Rev. A)
E/AALRT 1030-2015
20 34
Page 20 of 34
AALRT
7.5
7.5 Fault treatment procedure for core layer device of office network system
7.5.1
7.5.1 Fault symptom
AALRT AALRT
Fault in core layer device of office network system refers to network fault caused by hardware errors,
software vulnerabilities, network intrusion and environment factors and so on. The concrete
manifestation is: the extranet cant be accessed normally and the intranet cant be accessed normally
(e.g. Group information management system, AALRT website, AALRT mailbox system, information
management system and salary query system and so on).
7.5.2
Fault in core layer device of office network system mainly has the following influences:
1
The users cant access the extranet normally;
2
The users cant log in the Group information management system normally;
3
The users cant log in information management system normally;
4 AALRT
The users cant log in AALRT mailbox system normally;
5VPN AALRT
The VPN remote users cant log in AALRT intranet.
7.5.3
1 ICT
OA SecureCRT
shall immediately try to log in the intranet, extranet, Group information management system (OA
system) and the information management system and so on; if all of the systems cant be logged in
normally, try to log in the core device by remote tools of SecureCRT and hyperterminal, etc.; and if the
core device cant be connected normally, the professional engineer can preliminarily recognize that it is
extranet device fault and shall immediately inform the network system administrator.
A
System (Rev. A)
E/AALRT 1030-2015
21 34
Page 21 of 34
AALRT
2
1
1
software program error of core device (router and switchboard) which cant be recovered within 1 hour,
immediately launch the emergency plan, start the standby device, report to the leader of the emergency
repair group and tell other group members to conduct emergency repair; if the fault is recognized as
hardware fault or crash of core device which cant be recovered within 1 hour, immediately launch the
emergency plan, start the standby device, report to the leader of the emergency repair group and
conduct emergency repair toward the hardware. As for fault cant be repaired by replacing spare parts,
immediately ask the device supplier for support;
3 1
core device. As for relatively serious fault which cant be repaired within 1 hour, immediately report to
the leader of the emergency repair group. After the discussion of the group members, the group shall
conduct troubleshooting or data configuration backup recovery of the core device according to the fault
condition;
4 1
If the core device fault cant be repaired within 1 hour, inform the dispatchers in various professional
centers and relevant functional departments that the information management system shall apply the
emergency plan of this department.
5
After the core device fault is recovered, inform the dispatchers of various production departments and
relevant functional departments that the network has recovered to service.
7.5.4
Posts
Action Guidelines
AALRT
AALRT
users
Fault report:
AALRT IT
When the users cant access the intranets of various AALRT centers and the extranet,
inform IT dispatching console.
IT
IT
dispatching
console
Fault diagnosis:
OA
SecureCRT
The professional engineer shall immediately try to log in the intranet, extranet, Group
information management system (OA system) and the information management system,
A
System (Rev. A)
E/AALRT 1030-2015
22 34
Page 22 of 34
AALRT
and all of the systems cant be logged in normally. Then he shall try to log in the core
device (router and switchboard) by remote tools of SecureCRT and hyperterminal, etc.;
If the core device can be connected, the fault can be preliminarily recognized as
hardware fault of core device;
If the core device can be logged in normally, the fault can be preliminarily recognized as
link failure or configuration protocol error.
Fault report:
After the network system administrator recognizes the fault, the professional engineer
shall inform the group leader of the emergency repair group, announce the launching of
the emergency plan according to the instruction of the group leader, and issue the
According to the instruction of the leader of the emergency repair group, inform the
dispatchers of various production departments and relevant functional departments that
the intranet and extranet cant be accessed and that the information management
system shall apply the emergency plan of this department.

1
Inform relevant functional departments and the dispatchers in various professional
centers that the fault has been eliminated and the network has recovered;
2
Inform relevant functional departments and the dispatchers in various professional

PMISs
Fault record:
/
Emergency
repair
group/group
leader
Fault treatment:
1)
2)
A
System (Rev. A)
E/AALRT 1030-2015
23 34
Page 23 of 34
AALRT
computer room and tell the technicians of the group to rapidly leave for the destination
for joint treatment, and the group leader shall organize the members to finish the
emergency repair;
3) 1
The group members shall analyze the fault. If the fault is recognized as software
program error of core device which cant be recovered within 1 hour, immediately start
the standby device and report to the leader of the emergency repair group; and if the
fault is recognized as hardware fault or crash of core device (router and switchboard)
which cant be recovered within 1 hour, immediately start the standby device, report to
the leader of the emergency repair group and conduct emergency repair toward the
hardware. As for fault cant be repaired by replacing spare parts, immediately ask the
device supplier for support.
Fault recovery:
1)
The leader of the emergency repair group arrange some members to check if the core
device has all recovered and if the link communications have recovered;
2)

3)
If the backup device is applied, after the primary device has recovered, the configuration
files needs to be imported to the primary device;
4)
service module works properly;
5)
recovered.

1
2)
3)
A
System (Rev. A)
E/AALRT 1030-2015
24 34
Page 24 of 34
AALRT
7.6
7.6 Fault treatment procedure for convergence layer (branch) device of office network system
7.6.1
7.6.1 Fault symptom
AALRT AALRT
Fault in convergence layer device of office network system refers to network fault caused by hardware
errors, software vulnerabilities, link loop and environment factors, etc. The concrete manifestation is:
the branch users cant access the intranet and extranet normally, or log in information management
system normally (e.g. the Group information management system, AALRT website, AALRT mailbox
system, information management and salary query system, etc.).
7.6.2
Fault in convergence layer device of office network system mainly has the following influences:
1
The branch users cant access extranet normally;
2OA
The branch users cant log in the Group information management system (OA system) normally;
3
The branch users cant log in the information management system;
4) AALRT
The branch users cant log in the AALRT mailbox system.
7.6.3
1 ICT SecureCRT
The branch users shall immediately inform ICT Division after discovering the fault and the professional
engineer shall immediately try to access the branch device and log in the device via remote tools of
SecureCRT and hyperterminal and so on, but the device cant be connected anyway, then the
professional engineer can preliminarily recognize that it is convergence device fault and shall
immediately inform the network system administrators.
2 1.5
1.5
A
System (Rev. A)
E/AALRT 1030-2015
25 34
Page 25 of 34
AALRT
software program error of convergence device which cant be recovered within 1.5 hours, immediately
launch the emergency plan, start the standby device, report to the leader of the emergency repair group
and tell other group members to conduct emergency repair. If the fault is recognized as hardware fault
or crash of convergence device which cant be recovered within 1.5 hours, immediately launch the
emergency plan, start the standby device, report to the leader of the emergency repair group and
conduct emergency repair toward the hardware. As for fault cant be repaired by replacing spare parts,
3 1.5
convergence device. As for relatively serious fault which cant be repaired within 1.5 hours, immediately
report to the leader of the emergency repair group. After the discussion of the group members, the
group shall conduct troubleshooting or data configuration backup recovery of the convergence device
according to the fault condition;
4 1.5
If the convergence device fault cant be repaired within 1.5 hours, inform the dispatchers of various
professional centers and relevant functional departments of the branching that the information
management system shall apply the emergency plan of this department;
5
After the convergence device fault has recovered, inform dispatchers of various professional centers
and relevant functional departments of the branch that the intranet, extranet as well as the information
management system has recovered for service.
7.6.4
Posts
Action Guidelines
AALRT
AALRT
users
AALRT IT
When the intranet, extranet as well as the information management system of a certain
branch of AALRT cant be accessed normally, inform the IT dispatching console.
IT
IT
dispatching
console
Fault diagnosis:
SecureCRT
The professional engineer shall immediately try to access the branch device and try to
log in the convergence device via remote tools such as SecureCRT and hyperterminal:
If the device cant be connected normally, the fault can be preliminarily recognized as
hardware fault of convergence device;
If the device can be connected normally, the fault can be preliminarily recognized as link
Fault report:
A
System (Rev. A)
E/AALRT 1030-2015
26 34
Page 26 of 34
AALRT
After the network system administrator recognizes the fault, the special engineer shall
inform the group leader of the emergency repair group, announce the launching of the

According to the instruction of the leader of the emergency repair group, inform the
dispatchers of various production departments and relevant functional departments of
the branch that the intranet and extranet cant be accessed normally and that the
information management system shall apply the emergency plan of this department.

1
Inform relevant functional departments and dispatchers of various professional centers
of the branch that the fault has been eliminated and the network has recovered;
2
Inform relevant functional departments and dispatchers of various professional centers

of the branch that they shall enter the data generated during emergency repair in various
PMISs.
Fault record:
/
Emergency
repair
group/group
leader
Fault treatment:
1)
2)
computer room of the branch and tell the technicians of the group to rapidly leave for the
destination for joint treatment, and the group leader shall organize the members to
complete the emergency repair;
3) 1.5
1.5
program error of convergence device which cant be recovered within 1.5 hours,
immediately start the standby device and report to the leader of the emergency repair
group; and if the fault is recognized as hardware fault or crash of the convergence
A
System (Rev. A)
E/AALRT 1030-2015
27 34
Page 27 of 34
AALRT
device which cant be recovered within 1.5 hours, immediately start the standby device,
report to the leader of the emergency repair group and conduct emergency repair toward
the hardware. As for fault cant be repaired by replacing spare parts, immediately ask the
device supplier for support.
Fault recovery:
1)
convergence device has all recovered and if the link communications have recovered;
2)

3)
If the backup device is applied, after the fault of the primary device has recovered, the
configuration files need to imported to the primary device;
4)
service module works properly;
5)
recovered.

1
2)
3)
7.7
7.7 Fault treatment procedure for access stratum (important node user) of office network system
7.7.1
7.7.1 Fault symptom
A
System (Rev. A)
E/AALRT 1030-2015
28 34
Page 28 of 34
AALRT
AALRT AALRT
Fault in the access stratum of office network system refers to network fault caused by hardware errors,
software vulnerabilities, virus intrusion and environment factors and so on. The concrete manifestation
is: the node users cant normally access the intranet and extranet; the node users cant normally access
the information management system (e.g. Group information management system, AALRT website,
AALRT mailbox system, information management system and salary query system, etc.); and the node
users cant realize normal resource sharing of local area network.
7.7.2
Fault in the access stratum of office network system mainly has the following influences:
1
The node users cant access the extranet normally;
2 AALRT
The node users cant access the AALRT intranet normally;
3 AALRT
The node users cant log in the AALRT internal information management system;
4(FTP)
The node users cant realize resource sharing of the local area network (FTP, application sharing and
printing sharing, etc.)
7.7.3
1 ICT
shall immediately try to access the node device but the device cant be accessed normally; and he shall
try to connect the access device via remote tools but the device cant be connected, then the
professional engineer can preliminarily recognize that it is fault in the access device and shall
immediately inform the network system administrators.
2 2
software program error of access device which cant be recovered within 2 hours, immediately launch
the emergency plan, start the standby device, report to the leader of the emergency repair group and
tell other group members to conduct emergency repair. If the fault is recognized as hardware fault or
crash of access device which cant be recovered within 2 hours, immediately launch the emergency
plan, start the standby device, report to the leader of the emergency repair group and conduct
emergency repair toward the hardware. As for fault cant be repaired by replacing spare parts,
3 2
A
System (Rev. A)
E/AALRT 1030-2015
29 34
Page 29 of 34
AALRT
After arriving at the site, the emergency repair group shall immediately conduct emergency repair to
access device. As for relatively serious fault which cant be repaired within 2 hours, immediately report
to the leader of the emergency repair group. After the discussion of the group members, the group shall
conduct troubleshooting or data configuration backup recovery of the access device according to the
fault condition;
4 2
If the access device cant be repaired within 2 hours, tell relevant uses of various nodes to apply the
emergency plan of this department;
5
After the access device fault has recovered, inform relevant users of various nodes that the network
has recovered.
7.7.4
Posts
Action Guidelines
AALRT
AALRT
users
Fault report:
AALRT IT
When the intranet and extranet of an AALRT node cant be accessed, inform the IT
dispatching console.
IT
IT
dispatching
console
Fault diagnosis:
The professional engineer shall immediately try to access the node device but the device
cant be accessed normally. Then he tries to log in the access device via remote tools:
If the device cant be connected, the fault can be preliminarily recognized as hardware
fault of access device;
If the device can be logged in normally, the fault can be preliminarily recognized as link
Fault report:
After the network system administrator recognizes the fault, the special engineer shall
inform the group leader of the emergency repair group, announce the launching of the
A
System (Rev. A)
E/AALRT 1030-2015
30 34
Page 30 of 34
AALRT
According to the instruction of the leader of the emergency repair group, tell relevant
users of the node to use the emergency plan of this department.
Inform relevant users of the node that the fault is excluded and the network has
recovered.
Fault record:
/
Emergency
repair
group/group
leader
Fault treatment:
1)
2)
distributing room (house) and tell the technicians of the group to rapidly leave for the
destination for joint treatment, and the group leader shall organize the members to
complete the emergency repair;
3) 2
program error of access device which cant be recovered within 2 hours, immediately
start the standby device, report to the leader of the emergency repair group and tell
other group members to conduct emergency repair; and if the fault is recognized as
hardware fault or crash of the access device which cant be recovered within 2 hours,
immediately start the standby device, report to the leader of the emergency repair group
and conduct emergency repair toward the hardware. As for fault cant be repaired by
replacing spare parts, immediately ask the hardware supplier for support.
Fault recovery:
1)
access device has all recovered and if the port communication has recovered;
2)

3)
If the backup device is applied, after the primary device has recovered, the configuration
files need to be imported to the primary device;
A
System (Rev. A)
E/AALRT 1030-2015
31 34
Page 31 of 34
AALRT
4)
service module and port work normally;
5)
recovered.

1)
2)
3)
8.
8. Appendix
Time Table of Emergency Plan for Production Management Information System Fault
S/N
Name of Fault Treatment Procedure
Time to Launch
Emergency Plan
ORACLE
Fault treatment procedure for ORACLE database
2
2 hours
EAM
Fault treatment procedure for EAM system application server
1
1 hour
Fault treatment procedure for server hardware
2
2 hours
Fault treatment procedure for extranet access device of office network

system
2
2 hours
A
System (Rev. A)
E/AALRT 1030-2015
32 34
Page 32 of 34
AALRT
Fault treatment procedure for core layer device of office network system
1 hour
Fault treatment procedure for convergence layer (branch) device of office

network system
2
2 hours
Fault treatment procedure for access stratum (important node user) of

office network system
2
2 hours
Emergency Repair Plan

for Automation System
S/N
Standard name
Revisio
n
Prepared by
Wu Hailong
/
/
Prepared/Revised
Preparation/Revision
by
2015.7.30
Date:July
30, 2015
Issued on
Standard history tracking
A
System (Rev. A)
E/AALRT 1030-2015
33 34
Page 33 of 34

30 Emergency Repair Plan for Automation System 自动化系统应急抢修预案

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

30 Emergency Repair Plan for Automation System 自动化系统应急抢修预案

Transféré par

Droits d'auteur :

Formats disponibles

AALRT

AALRT Enterprise Standard

Special Response Plan

Reviewed by: Sun Hongyu

Prepared by: Wu Hailong

Dept.: Maintenance Center

Controlled State: Internal

4. DEFINITIONS AND TERMS

7. EMERGENCY PLAN FOR PMIS

Emergency Repair Plan for Automation System

Rules for Organization of Train Operation

Group leader: Director of the Maintenance Center

Take part in analysis of the emergency;

Be responsible for the implementation of corrective measures.

Treatment principles for emergency repair

Information transmission during the fault:

Information transmission after fault recovery:

Inform relevant functional departments and the dispatchers of various professional

Emergency repair completed:

Information transmission during the fault:

Information transmission after the fault recovery:

Inform relevant functional departments and the dispatchers of various professional

Emergency repair completed:

AALRT AALRT EAM

Information transmission during the fault:

Information transmission after fault recovery:

Inform relevant functional departments and the dispatchers of various professional

Information transmission during the fault:

Information transmission after fault recovery:

Inform relevant functional departments and the dispatchers of various professional

If the configuration backup recovery mechanism is applied, check if the configuration

Emergency repair completed:

Information transmission during the fault:

Information transmission after fault recovery:

Inform relevant functional departments and the dispatchers in various professional

If the configuration backup recovery mechanism is applied, check if the configuration

Emergency repair completed:

Information transmission during the fault:

Information transmission after fault recovery:

Inform relevant functional departments and dispatchers of various professional centers

If the configuration backup recovery mechanism is applied, check if the configuration

Emergency repair completed:

Information transmission during the fault:

Information transmission after fault recovery:

If the configuration backup recovery mechanism is applied, check if the configuration

Emergency repair completed:

Name of Fault Treatment Procedure

Fault treatment procedure for server hardware

Fault treatment procedure for extranet access device of office network

Fault treatment procedure for convergence layer (branch) device of office

Fault treatment procedure for access stratum (important node user) of

Emergency Repair Plan

Standard history tracking

Vous aimerez peut-être aussi