IBM Power System System E950 Manual 

page v, “Notices” on page 67, the IBM Systems Safety Notices manual, G229-9054, and the IBM Environmental Notices and User Guide, Z125–5823. This edition applies to IBM® Power Systems servers that contain the POWER9™ processor and to all associated models. © Copyright International Business Machines Corporation 2018, 2019. US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

Installing PCIe adapters in the system....................................................................................................... 1 Preparing the 9040-MR9 system to install a PCIe adapter ................................................................. 1 Installing a PCIe adapter in the system.................................................................................................9 Preparing the 9040-MR9 system for operation after installing a PCIe adapter................................ 15 Removing and replacing PCIe adapters....................................................................................................18 Preparing the system to remove and replace a PCIe adapter............................................................ 18 Removing a PCIe adapter from the 9040-MR9 system...................................................................... 27 Replacing a PCIe adapter in the 9040-MR9 system...........................................................................30 Preparing the system for operation after removing and replacing a PCIe adapter............................35 Permanently removing a PCIe adapter from the system......................................................................... 38 Preparing the system to permanently remove a PCIe adapter from the system...............................38 Permanently removing a PCIe adapter from the 9040-MR9 system................................................. 43 Preparing the 9040-MR9 system for operation after permanently removing a PCIe adapter.......... 47 Related procedures for installing and removing PCIe adapters.............................................................. 49 Avoiding electric shock........................................................................................................................ 49 Handling static-sensitive devices........................................................................................................ 51 Updating the worldwide port name..................................................................................................... 51 Removing and replacing the tailstock on a PCIe adapter .................................................................. 51 PCIe hot-plug manager access for AIX............................................................................................... 56 Installing the AIX device driver software............................................................................................ 58 Installing or replacing a PCIe adapter with the system power turned on in a Virtual I/O Server......60 Verifying that the hot-plug PCI tools are installed for Linux...............................................................63 Checking the amount of remaining life in NVMe devices....................................................................64 Notices................................................................................................................67 Accessibility features for IBM Power Systems servers............................................................................ 68 Privacy policy considerations ................................................................................................................... 69 Trademarks................................................................................................................................................69 Electronic emission notices.......................................................................................................................69 Class A Notices.....................................................................................................................................69 Class B Notices.....................................................................................................................................73 Terms and conditions................................................................................................................................ 75 iii

• DANGER notices call attention to a situation that is potentially lethal or extremely hazardous to people. • CAUTION notices call attention to a situation that is potentially hazardous to people because of some existing condition. • Attention notices call attention to the possibility of damage to a program, device, system, or data. World Trade safety information Several countries require the safety information contained in product publications to be presented in their national languages. If this requirement applies to your country, safety information documentation is included in the publications package (such as in printed documentation, on DVD, or as part of the product) shipped with the product. The documentation contains the safety information in your national language with references to the U.S. English source. Before using a U.S. English publication to install, operate, or service this product, you must first become familiar with the related safety information documentation. You should also refer to the safety information documentation any time you do not clearly understand any safety information in the U.S. English publications. Replacement or additional copies of safety information documentation can be obtained by calling the IBM Hotline at 1-800-300-8751. German safety information Das Produkt ist nicht für den Einsatz an Bildschirmarbeitsplätzen im Sinne § 2 der Bildschirmarbeitsverordnung geeignet. Laser safety information IBM servers can use I/O cards or features that are fiber-optic based and that utilize lasers or LEDs. Laser compliance IBM servers may be installed inside or outside of an IT equipment rack. DANGER: When working on or around the system, observe the following precautions: Electrical voltage and current from power, telephone, and communication cables are hazardous. To avoid a shock hazard: • If IBM supplied the power cord(s), connect power to this unit only with the IBM provided power cord. Do not use the IBM provided power cord for any other product. • Do not open or service any power supply assembly. • Do not connect or disconnect any cables or perform installation, maintenance, or reconfiguration of this product during an electrical storm. • The product might be equipped with multiple power cords. To remove all hazardous voltages, disconnect all power cords. – For AC power, disconnect all power cords from their AC power source. – For racks with a DC power distribution panel (PDP), disconnect the customer’s DC power source to the PDP. • When connecting power to the product ensure all power cables are properly connected. – For racks with AC power, connect all power cords to a properly wired and grounded electrical outlet. Ensure that the outlet supplies proper voltage and phase rotation according to the system rating plate. © Copyright IBM Corp. 2018, 2019 v

to the PDP. Ensure that the proper polarity is used when attaching the DC power and DC power return wiring. • Connect any equipment that will be attached to this product to properly wired outlets. • When possible, use one hand only to connect or disconnect signal cables. • Never turn on any equipment when there is evidence of fire, water, or structural damage. • Do not attempt to switch on power to the machine until all possible unsafe conditions are corrected. • Assume that an electrical safety hazard is present. Perform all continuity, grounding, and power checks specified during the subsystem installation procedures to ensure that the machine meets safety requirements. • Do not continue with the inspection if any unsafe conditions are present. • Before you open the device covers, unless instructed otherwise in the installation and configuration procedures: Disconnect the attached AC power cords, turn off the applicable circuit breakers located in the rack power distribution panel (PDP), and disconnect any telecommunications systems, networks, and modems. DANGER: • Connect and disconnect cables as described in the following procedures when installing, moving, or opening covers on this product or attached devices. To Disconnect: 1. Turn off everything (unless instructed otherwise). 2. For AC power, remove the power cords from the outlets. 3. For racks with a DC power distribution panel (PDP), turn off the circuit breakers located in the PDP and remove the power from the Customer's DC power source. 4. Remove the signal cables from the connectors. 5. Remove all cables from the devices. To Connect: 1. Turn off everything (unless instructed otherwise). 2. Attach all cables to the devices. 3. Attach the signal cables to the connectors. 4. For AC power, attach the power cords to the outlets. 5. For racks with a DC power distribution panel (PDP), restore the power from the Customer's DC power source and turn on the circuit breakers located in the PDP. 6. Turn on the devices. Sharp edges, corners and joints may be present in and around the system. Use care when handling equipment to avoid cuts, scrapes and pinching. (D005) (R001 part 1 of 2): DANGER: Observe the following precautions when working on or around your IT rack system: • Heavy equipment–personal injury or equipment damage might result if mishandled. • Always lower the leveling pads on the rack cabinet. • Always install stabilizer brackets on the rack cabinet unless the earthquake option is to be installed. • To avoid hazardous conditions due to uneven mechanical loading, always install the heaviest devices in the bottom of the rack cabinet. Always install servers and optional devices starting from the bottom of the rack cabinet. vi Power Systems: PCIe adapters for the 9040-MR9

of rack-mounted devices. In addition, do not lean on rack mounted devices and do not use them to stabilize your body position (for example, when working from a ladder). • Stability hazard: – The rack may tip over causing serious personal injury. – Before extending the rack to the installation position, read the installation instructions. – Do not put any load on the slide-rail mounted equipment mounted in the installation position. – Do not leave the slide-rail mounted equipment in the installation position. • Each rack cabinet might have more than one power cord. – For AC powered racks, be sure to disconnect all power cords in the rack cabinet when directed to disconnect power during servicing. – For racks with a DC power distribution panel (PDP), turn off the circuit breaker that controls the power to the system unit(s), or disconnect the customer’s DC power source, when directed to disconnect power during servicing. • Connect all devices installed in a rack cabinet to power devices installed in the same rack cabinet. Do not plug a power cord from a device installed in one rack cabinet into a power device installed in a different rack cabinet. • An electrical outlet that is not correctly wired could place hazardous voltage on the metal parts of the system or the devices that attach to the system. It is the responsibility of the customer to ensure that the outlet is correctly wired and grounded to prevent an electrical shock. (R001 part 1 of 2) (R001 part 2 of 2): CAUTION: • Do not install a unit in a rack where the internal rack ambient temperatures will exceed the manufacturer's recommended ambient temperature for all your rack-mounted devices. • Do not install a unit in a rack where the air flow is compromised. Ensure that air flow is not blocked or reduced on any side, front, or back of a unit used for air flow through the unit. • Consideration should be given to the connection of the equipment to the supply circuit so that overloading of the circuits does not compromise the supply wiring or overcurrent protection. To provide the correct power connection to a rack, refer to the rating labels located on the equipment in the rack to determine the total power requirement of the supply circuit. • (For sliding drawers.) Do not pull out or install any drawer or feature if the rack stabilizer brackets are not attached to the rack or if the rack is not bolted to the floor. Do not pull out more than one drawer at a time. The rack might become unstable if you pull out more than one drawer at a time. • (For fixed drawers.) This drawer is a fixed drawer and must not be moved for servicing unless specified by the manufacturer. Attempting to move the drawer partially or completely out of the rack might cause the rack to become unstable or cause the drawer to fall out of the rack. (R001 part 2 of 2) Safety notices vii

stability during relocation. Follow these general guidelines whenever you relocate a populated rack cabinet within a room or building. • Reduce the weight of the rack cabinet by removing equipment starting at the top of the rack cabinet. When possible, restore the rack cabinet to the configuration of the rack cabinet as you received it. If this configuration is not known, you must observe the following precautions: – Remove all devices in the 32U position (compliance ID RACK-001 or 22U (compliance ID RR001) and above. – Ensure that the heaviest devices are installed in the bottom of the rack cabinet. – Ensure that there are little-to-no empty U-levels between devices installed in the rack cabinet below the 32U (compliance ID RACK-001 or 22U (compliance ID RR001) level, unless the received configuration specifically allowed it. • If the rack cabinet you are relocating is part of a suite of rack cabinets, detach the rack cabinet from the suite. • If the rack cabinet you are relocating was supplied with removable outriggers they must be reinstalled before the cabinet is relocated. • Inspect the route that you plan to take to eliminate potential hazards. • Verify that the route that you choose can support the weight of the loaded rack cabinet. Refer to the documentation that comes with your rack cabinet for the weight of a loaded rack cabinet. • Verify that all door openings are at least 760 x 230 mm (30 x 80 in.). • Ensure that all devices, shelves, drawers, doors, and cables are secure. • Ensure that the four leveling pads are raised to their highest position. • Ensure that there is no stabilizer bracket installed on the rack cabinet during movement. • Do not use a ramp inclined at more than 10 degrees. • When the rack cabinet is in the new location, complete the following steps: – Lower the four leveling pads. – Install stabilizer brackets on the rack cabinet or in an earthquake environment bolt the rack to the floor. – If you removed any devices from the rack cabinet, repopulate the rack cabinet from the lowest position to the highest position. • If a long-distance relocation is required, restore the rack cabinet to the configuration of the rack cabinet as you received it. Pack the rack cabinet in the original packaging material, or equivalent. Also lower the leveling pads to raise the casters off of the pallet and bolt the rack cabinet to the pallet. (R002) (L001) DANGER: Hazardous voltage, current, or energy levels are present inside any component that has this label attached. Do not open any cover or barrier that contains this label. (L001) (L002) viii Power Systems: PCIe adapters for the 9040-MR9

objects on top of rack-mounted devices. In addition, do not lean on rack-mounted devices and do not use them to stabilize your body position (for example, when working from a ladder). Stability hazard: • The rack may tip over causing serious personal injury. • Before extending the rack to the installation position, read the installation instructions. • Do not put any load on the slide-rail mounted equipment mounted in the installation position. • Do not leave the slide-rail mounted equipment in the installation position. (L002) (L003) or or or Safety notices ix

multiple DC power cables. To remove all hazardous voltages, disconnect all power cords and power cables. (L003) (L007) CAUTION: A hot surface nearby. (L007) x Power Systems: PCIe adapters for the 9040-MR9

All lasers are certified in the U.S. to conform to the requirements of DHHS 21 CFR Subchapter J for class 1 laser products. Outside the U.S., they are certified to be in compliance with IEC 60825 as a class 1 laser product. Consult the label on each part for laser certification numbers and approval information. CAUTION: This product might contain one or more of the following devices: CD-ROM drive, DVD- ROM drive, DVD-RAM drive, or laser module, which are Class 1 laser products. Note the following information: • Do not remove the covers. Removing the covers of the laser product could result in exposure to hazardous laser radiation. There are no serviceable parts inside the device. • Use of the controls or adjustments or performance of procedures other than those specified herein might result in hazardous radiation exposure. (C026) CAUTION: Data processing environments can contain equipment transmitting on system links with laser modules that operate at greater than Class 1 power levels. For this reason, never look into the end of an optical fiber cable or open receptacle. Although shining light into one end and looking into the other end of a disconnected optical fiber to verify the continuity of optic fibers may not injure the eye, this procedure is potentially dangerous. Therefore, verifying the continuity of optical fibers by shining light into one end and looking at the other end is not recommended. To verify continuity of a fiber optic cable, use an optical light source and power meter. (C027) CAUTION: This product contains a Class 1M laser. Do not view directly with optical instruments. (C028) CAUTION: Some laser products contain an embedded Class 3A or Class 3B laser diode. Note the following information: • Laser radiation when open. • Do not stare into the beam, do not view directly with optical instruments, and avoid direct exposure to the beam. (C030) (C030) CAUTION: The battery contains lithium. To avoid possible explosion, do not burn or charge the battery. Do Not: • Throw or immerse into water • Heat to more than 100 degrees C (212 degrees F) • Repair or disassemble Exchange only with the IBM-approved part. Recycle or discard the battery as instructed by local regulations. In the United States, IBM has a process for the collection of this battery. For information, call 1-800-426-4333. Have the IBM part number for the battery unit available when you call. (C003) CAUTION: Regarding IBM provided VENDOR LIFT TOOL: • Operation of LIFT TOOL by authorized personnel only. Safety notices xi

not to be used loaded transporting over major ramps nor as a replacement for such designated tools like pallet jacks, walkies, fork trucks and such related relocation practices. When this is not practicable, specially trained persons or services must be used (for instance, riggers or movers). • Read and completely understand the contents of LIFT TOOL operator's manual before using. Failure to read, understand, obey safety rules, and follow instructions may result in property damage and/or personal injury. If there are questions, contact the vendor's service and support. Local paper manual must remain with machine in provided storage sleeve area. Latest revision manual available on vendor's web site. • Test verify stabilizer brake function before each use. Do not over-force moving or rolling the LIFT TOOL with stabilizer brake engaged. • Do not raise, lower or slide platform load shelf unless stabilizer (brake pedal jack) is fully engaged. Keep stabilizer brake engaged when not in use or motion. • Do not move LIFT TOOL while platform is raised, except for minor positioning. • Do not exceed rated load capacity. See LOAD CAPACITY CHART regarding maximum loads at center versus edge of extended platform. • Only raise load if properly centered on platform. Do not place more than 200 lb (91 kg) on edge of sliding platform shelf also considering the load's center of mass/gravity (CoG). • Do not corner load the platforms, tilt riser, angled unit install wedge or other such accessory options. Secure such platforms -- riser tilt, wedge, etc options to main lift shelf or forks in all four (4x or all other provisioned mounting) locations with provided hardware only, prior to use. Load objects are designed to slide on/off smooth platforms without appreciable force, so take care not to push or lean. Keep riser tilt [adjustable angling platform] option flat at all times except for final minor angle adjustment when needed. • Do not stand under overhanging load. • Do not use on uneven surface, incline or decline (major ramps). • Do not stack loads. • Do not operate while under the influence of drugs or alcohol. • Do not support ladder against LIFT TOOL (unless the specific allowance is provided for one following qualified procedures for working at elevations with this TOOL). • Tipping hazard. Do not push or lean against load with raised platform. • Do not use as a personnel lifting platform or step. No riders. • Do not stand on any part of lift. Not a step. • Do not climb on mast. • Do not operate a damaged or malfunctioning LIFT TOOL machine. • Crush and pinch point hazard below platform. Only lower load in areas clear of personnel and obstructions. Keep hands and feet clear during operation. • No Forks. Never lift or move bare LIFT TOOL MACHINE with pallet truck, jack or fork lift. • Mast extends higher than platform. Be aware of ceiling height, cable trays, sprinklers, lights, and other overhead objects. • Do not leave LIFT TOOL machine unattended with an elevated load. • Watch and keep hands, fingers, and clothing clear when equipment is in motion. • Turn Winch with hand power only. If winch handle cannot be cranked easily with one hand, it is probably over-loaded. Do not continue to turn winch past top or bottom of platform travel. Excessive unwinding will detach handle and damage cable. Always hold handle when lowering, unwinding. Always assure self that winch is holding load before releasing winch handle. • A winch accident could cause serious injury. Not for moving humans. Make certain clicking sound is heard as the equipment is being raised. Be sure winch is locked in position before releasing handle. Read instruction page before operating this winch. Never allow winch to unwind freely. xii Power Systems: PCIe adapters for the 9040-MR9

cause serious injury. • This TOOL must be maintained correctly for IBM Service personnel to use it. IBM shall inspect condition and verify maintenance history before operation. Personnel reserve the right not to use TOOL if inadequate. (C048) Power and cabling information for NEBS (Network Equipment-Building System) GR-1089-CORE The following comments apply to the IBM servers that have been designated as conforming to NEBS (Network Equipment-Building System) GR-1089-CORE: The equipment is suitable for installation in the following: • Network telecommunications facilities • Locations where the NEC (National Electrical Code) applies The intrabuilding ports of this equipment are suitable for connection to intrabuilding or unexposed wiring or cabling only. The intrabuilding ports of this equipment must not be metallically connected to the interfaces that connect to the OSP (outside plant) or its wiring. These interfaces are designed for use as intrabuilding interfaces only (Type 2 or Type 4 ports as described in GR-1089-CORE) and require isolation from the exposed OSP cabling. The addition of primary protectors is not sufficient protection to connect these interfaces metallically to OSP wiring. Note: All Ethernet cables must be shielded and grounded at both ends. The ac-powered system does not require the use of an external surge protection device (SPD). The dc-powered system employs an isolated DC return (DC-I) design. The DC battery return terminal shall not be connected to the chassis or frame ground. The dc-powered system is intended to be installed in a common bonding network (CBN) as described in GR-1089-CORE. Safety notices xiii

the 9040-MR9 system Find information about installing, removing, and replacing Peripheral Component Interconnect Express (PCIe) adapters in the IBM Power® System E950 (9040-MR9) server. Installing PCIe adapters in the 9040-MR9 Find information about installing Peripheral Component Interconnect Express (PCIe) adapters in the system. About this task Note: Installing this feature is a customer task. You can complete this task yourself, or contact a service provider to complete the task for you. You might be charged a fee by the service provider for this service. If your system is managed by the Hardware Management Console (HMC), use the HMC to complete the steps for installing a PCIe adapter in the system. For instructions, see Installing a part by using the HMC (www.ibm.com/support/knowledgecenter/POWER9/p9haj/hmcinstall.htm). If your system is not managed by an HMC, complete the steps in the following procedures to install a PCIe adapter in the system. Preparing the 9040-MR9 system to install a PCIe adapter To prepare the system to install a PCIe adapter, complete the steps in this procedure. Before you begin Note: If you are installing a PCIe3 cable adapter to attach your server to an EMX0 PCIe3 expansion drawer, you must ensure that you have the correct PCIe3 cable adapter and expansion drawer cable that works with the PCIe3 6-slot fanout module that you have in your EMX0 PCIe3 expansion drawer. • If you have a CCIN 50CB PCIe3 6-slot fanout module: – You must use one of the following PCIe3 cable adapters: - FC EJ08 (CCIN 2CE2) - FC EJ07 (CCIN 6B52) - FC EJ05 (CCIN 2B1C) – You must use one of the following expansion drawer cables: FC ECC6, FC ECC7, FC ECC8, FC ECC9, FC ECCR, FC ECCS, FC ECCX, FC ECCY, or FC ECCZ. • If you have a CCIN 50CD PCIe3 6-slot fanout module: – You must install one of the following PCIe3 cable adapters: - FC EJ19 (CCIN 6B53) - FC EJ1R (CCIN 58FF) - FC EJ20 (CCIN 2CF5) – You must use one of the following expansion drawer cables: FC ECCR, FC ECCX, FC ECCY, or FC ECCZ. Procedure 1. Ensure that the required software is installed to support the new feature. See the Power Systems Prerequisites website. © Copyright IBM Corp. 2018, 2019 1

drawer via the HMC, you must have the HMC driver 9.1.921.0 or later installed. 2. Take appropriate precautions for avoiding electric shock and for handling static-sensitive devices. For more information, see “Avoiding electric shock” on page 49 and “Handling static-sensitive devices” on page 51. 3. If you are installing a PCIe3 cable adapter, ensure that the target slot is Unassigned in the I/O properties field for the managed server you are working on. 4. If applicable, open the rack door at the rear of the system. 5. Review the PCIe adapter slot locations. The PCIe adapter slots are at the rear of a system. The number of the PCIe adapter slots that are available in the system depends on the number of system processors in the system. For the PCIe adapter placement information for the available slots in this system, see PCIe adapter placement rules and slot priorities for the 9040-MR9. The PCIe adapter has two LEDs that indicate the following status: • A power/activity LED (green) • An error and identify function LED (amber) The status of the LEDs are as follows: • (B) Indicates that the PCIe adapter is operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is off. • (C) Indicates that the PCIe adapter is not operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is lit solid. • (D) Indicates the faulty or failed PCIe adapter was selected by using the identify function. The green power LED (top LED) might or might not be lit and the amber fault LED (bottom LED) is flashing. 2 Power Systems: PCIe adapters for the 9040-MR9

6. Choose from the following options: • If you want to install a PCIe adapter when the system power is turned off, continue with step “7” on page 3. • If the system power is turned on and if the AIX® operating system controls the slot, continue with step “9” on page 6. • If the system power is turned on and if the Linux operating system controls the slot, continue with step “10” on page 6. 7. To prepare the system to install a PCIe adapter when the system is powered off, complete the following steps: a) Activate the identify function. For instructions, see Identifying a part (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/sal.htm). b) Physically verify that the slot you identified is where you want to install the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for a flashing amber LED, which identifies the unoccupied slot that was selected by using the identify function. c) Stop the system. For instructions, see Stopping a system (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/crustopsys.htm). Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 3

a) If applicable, open the rack door at the rear of the system. b) Label and disconnect the power cords from the system unit as shown in the following figure. Notes: • This system might be equipped with two or more power supplies. If the removing and replacing procedures require the system power to be turned off, ensure that all power sources to the system are disconnected. • The power cord (B) is fastened to the system with a hook-and-loop fastener (A). If you are placing the system in a service position after you disconnect the power cords, ensure that you unstrap the fastener. Figure 2. Removing the power cords (L003) or 4 Power Systems: PCIe adapters for the 9040-MR9

cords or multiple DC power cables. To remove all hazardous voltages, disconnect all power cords and power cables. (L003) c) Continue with step “11” on page 7. 9. To prepare the system to install a PCIe adapter by using the AIX operating system, complete the following steps: a) Log in to the console as root user to access the Hot Plug Manager. b) At the command line, type smitty. c) Select Devices > PCI Hot Plug Manager and press Enter. The PCI Hot-Plug Manager menu displays. For more information about the menu options in the PCI Hot-Plug Manager screen, see PCIe hot- plug manager menu (www.ibm.com/support/knowledgecenter/POWER9/p9hak/ p9hak_hotplug_menu.htm). d) From the PCI Hot-Plug Manager menu, select Add a PCI Hot-Plug Adapter, and then press Enter. e) Select the appropriate PCIe slot from the list that is displayed on the screen, and then press Enter. f) Physically verify that the slot you identified is where you want to install the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for a flashing amber LED, which identifies the unoccupied slot that was selected by using the identify function. g) Continue with step “11” on page 7. 10. To prepare the system to install a PCIe adapter by using the Linux operating system, complete the following steps: a) Log in to the system console as the root user. b) Run the following command to list the available slots: lsslot -c pci -a The following screen is an example of the information that is displayed by this command: # Slot Description Device(s) U7879.001.DQD014E-P1-C1 PCI-X capable, 64 bit, 133MHz slot Empty U7879.001.DQD014E-P1-C4 PCI-X capable, 64 bit, 133MHz slot Empty U7879.001.DQD014E-P1-C5 PCI-X capable, 64 bit, 133MHz slot Empty c) Record the empty PCIe slot where you want to install the adapter. d) Complete the following steps to prepare the slot to accept a PCIe adapter: 1) Type the following command: drmgr -c pci -r -s locationcode 6 Power Systems: PCIe adapters for the 9040-MR9

U7879.001.DQD014E-P1-C3. 2) Press Enter. A fast-flashing amber LED at the rear of the system near the adapter indicates that the slot is identified. 3) Physically verify that the slot you identified is where you want to install the adapter. Notes: • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for the flashing amber LED, which identifies the slot that was selected by using the identify function. e) Physically verify that the slot you identified is where you want to install the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for a flashing amber LED, which identifies the unoccupied slot that was selected by using the identify function. 11. At the rear of the system, put the cable management bracket in the service position. See the following figure. a) Pull out the quarter-turn fasteners (B) and turn them to disengage them while you lift the cable management bracket (A) to its raised position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 7

12. Attach the electrostatic discharge (ESD) wrist strap. The ESD wrist strap must be connected to an unpainted metal surface until the service procedure is completed, and if applicable, until the service access cover is replaced. Attention: • Attach an electrostatic discharge (ESD) wrist strap to the front ESD jack, to the rear ESD jack, or to an unpainted metal surface of your hardware to prevent the electrostatic discharge from damaging your hardware. • When you use an ESD wrist strap, follow all electrical safety procedures. An ESD wrist strap is used for static control. It does not increase or decrease your risk of receiving electric shock when using or working on electrical equipment. • If you do not have an ESD wrist strap, just prior to removing the product from ESD packaging and installing or replacing hardware, touch an unpainted metal surface of the system for a minimum of 5 seconds. If at any point in this service process you move away from the system, it is important to again discharge yourself by touching an unpainted metal surface for at least 5 seconds before you continue with the service process. 8 Power Systems: PCIe adapters for the 9040-MR9

To install a PCIe adapter in the system, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. If necessary, remove the adapter from the antistatic package. Attention: Avoid touching the components and gold connectors on the adapter. 3. Place the adapter, component-side up, on a flat, antistatic surface. Attention: A pin on the tailstock of the adapter resembles a removable screw. Do not remove this pin. It is required for correct alignment and seating. 4. Remove the PCIe cassette from the slot: a) Squeeze the latch lever (A) and press down on the latch (B) in the direction shown. This action pivots the cassette forward and releases the cassette from the slot. b) Pull out the cassette from its slot. Figure 4. Removing a PCIe adapter cassette from the 9040-MR9 system 5. Remove the PCIe tailstock filler from the cassette. a) Open the tailstock clamp (A) by rotating the clamp in the direction that is shown. b) Press and slide the release bar (B) in the direction that is shown to open the cassette. c) Grasp the tailstock filler and pull the filler away from the cassette in the direction that is shown. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 9

Note: Your cassette might not have the two adapter retainers (C) and strips. d) Save the tailstock filler for a future use. You must have a tailstock filler in an empty PCIe cassette in a system to ensure proper airflow. 6. Choose from the following options: • To install a PCIe adapter into a cassette that has adapter retainers and strips, continue with step “7” on page 10. • To install a PCIe adapter into a cassette that doesn't have adapter retainers and strips, continue with step 8. 7. If your cassette has two adapter retainers with strips, complete the following steps to install or replace a PCIe adapter into the cassette: a) If the adapter retainers (A) haven't already been removed, remove the adapter retainers (A) by putting them in the open position and moving them all the way off the strips. b) Press and slide the release bar (B) to open the cassette. c) Place an ESD safety sheet (C) over the strips. Note: You can use the ESD package that the new adapter was in. If necessary, cut it to size. 10 Power Systems: PCIe adapters for the 9040-MR9

d) Insert the adapter into the cassette. e) Press and slide the release bar (A) to close the cassette. f) Close the tailstock clamp (B) by rotating the clamp in the direction that is shown. g) Pull the ESD safety film (C) out. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 11

h) With the adapter retainers (D) in the open position, slide the retainers onto the strips. i) Position the adapter retainers (D) so that they support the adapter and put them in the locked position by rotating them in the direction that is shown. Note: Ensure that the adapter is secured in the adapter retainer channel. 12 Power Systems: PCIe adapters for the 9040-MR9

8. If your cassette does not have the two adapter retainers and strips, complete the following steps to install or replace a PCIe adapter into the cassette: a) Press and slide the release bar (A) to open the cassette. b) Insert the adapter into the cassette. c) Press and slide the release bar (A) to close the cassette. d) Close the tailstock clamp (B) by rotating the clamp in the direction that is shown. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 13

9. Slide the PCIe cassette into the slot by completing the following steps: Attention: Ensure proper alignment when you insert a PCI adapter cassette into the system. a) Align the cassette on the cassette rail in the cassette slot. b) Slide the cassette forward until the cassette is fully seated. c) To lock the cassette in its slot, squeeze the latch lever (A) toward the latch (B) and rotate the latch (B) in the direction that is shown. 14 Power Systems: PCIe adapters for the 9040-MR9

Preparing the 9040-MR9 system for operation after installing a PCIe adapter To prepare the system for operation after permanently removing a PCIe adapter, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. If you installed a PCIe3 cable adapter, and want to connect an EMX0 PCIe3 expansion drawer to your system, continue with Connecting an EMX0 PCIe Gen3 I/O expansion drawer to your system (http:// www.ibm.com/support/knowledgecenter/POWER9/p9egp/p9egp_connect_kickoff.htm). If you have an HMC-managed 9040-MR9 or 9080-M9S system, you can connect and activate the expansion drawer cable when the system power is turned on. Otherwise, you must connect the cable with the system power turned off. The cable is activated when you start the system. 3. If removed, connect the cables or transceivers to the adapter. 4. Route the cables through the cable management bracket. 5. If not already connected, reconnect the power cords (A) to the system unit. Fasten the power cords to the system using the hook-and-loop fasteners (B) as shown in the following figure. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 15

6. Put the cable management bracket in the operating position. See the following figure. a) Pull out the quarter-turn fasteners (B) and turn them to disengage them while moving the cable management bracket (A) to its lower position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. 16 Power Systems: PCIe adapters for the 9040-MR9

7. Close the rack door at the rear of the system. 8. Choose from the following options: • If the system power is turned off, continue with step “9” on page 17. • If you are using the AIX operating system, continue with step “10” on page 17. • If you are using the Linux operating system, continue with step “11” on page 17. 9. If the system power is turned off, complete the following steps: a) Start the system. For instructions, see Starting a system (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/crustartsys.htm). b) Continue with step “12” on page 18. 10. If you are using the AIX operating system, you must configure the adapter by completing the following steps: a) Install the AIX device drivers for the installed adapter. For instructions, see Installing the AIX device driver software. b) On the console, enter cfgmgr to configure the adapter. c) Continue with step “12” on page 18. 11. If you are using the Linux operating system, complete the following steps: Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 17

slot into the action state. b) Enter the slot information with the lsslot command, as shown in the following example. For example, if the slot in which you installed the PCIe adapter was U7879.001.DQD014E-P1-C3. Enter lsslot -c pci -s U7879.001.DQD014E-P1-C3 The following screen is an example of the information that is displayed by this command: # Slot Description Device(s) U7879.001.DQD014E-P1-C3 PCI-X capable, 64 bit, 133MHz slot 0001:40:01.0 12. Verify the installed part. • If you replaced the part because of a service action, verify the installed part. For instructions, see https://www.ibm.com/support/knowledgecenter/POWER9/p9ect/ pxect_verifyrepair.htm (www.ibm.com/support/knowledgecenter/POWER9/p9ect/ pxect_verifyrepair.htm). • If you installed the part for any other reason, verify the installed part. For instructions, see https:// www.ibm.com/support/knowledgecenter/POWER9/p9haj/pxhaj_hsmverify.htm (www.ibm.com/ support/knowledgecenter/POWER9/p9haj/pxhaj_hsmverify.htm). 13. Turn off the identify LED. For instructions, see Deactivating an identify LED (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/p9haj_turn_off_identify_led.htm). Removing and replacing PCIe adapters in the 9040-MR9 Find information about removing and replacing a Peripheral Component Interconnect Express (PCIe) adapter in the IBM Power System E950 (9040-MR9) server. About this task Note: Removing or replacing this feature is a customer task. You can complete this task yourself, or contact a service provider to complete the task for you. You might be charged a fee by the service provider for this service. If your system is managed by the Hardware Management Console (HMC), use the HMC to repair a part in the system. For instructions, see Repairing a part by using the HMC (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/p9haj_hmc_repair.htm). If your system is not managed by an HMC, complete the steps in the following procedures to remove and replace a PCIe adapter in the system. Preparing the 9040-MR9 system to remove and replace a PCIe adapter To prepare the system to remove and replace a PCIe adapter in the system, complete the steps in this procedure. Before you begin Note: If you are replacing a PCIe3 cable adapter that you use to attach your server to an EMX0 PCIe3 expansion drawer, you must ensure that you have the correct PCIe3 cable adapter and expansion drawer cable to work with the PCIe3 6-slot fanout module that you have in your EMX0 PCIe3 expansion drawer. • If you have a CCIN 50CB PCIe3 6-slot fanout module: – You must use one of the following PCIe3 cable adapters: - FC EJ08 (CCIN 2CE2) - FC EJ07 (CCIN 6B52) 18 Power Systems: PCIe adapters for the 9040-MR9

– You must use one of the following expansion drawer cables: FC ECC6, FC ECC7, FC ECC8, FC ECC9, FC ECCR, FC ECCS, FC ECCX, FC ECCY, or FC ECCZ. • If you have a CCIN 50CD PCIe3 6-slot fanout module: – You must install one of the following PCIe3 cable adapters: - FC EJ19 (CCIN 6B53) - FC EJ1R (CCIN 58FF) - FC EJ20 (CCIN 2CF5) – You must use one of the following expansion drawer cables: FC ECCR, FC ECCX, FC ECCY, or FC ECCZ. Procedure 1. Take appropriate precautions for avoiding electric shock and for handling static-sensitive devices. For more information, see “Avoiding electric shock” on page 49 and “Handling static-sensitive devices” on page 51. 2. If applicable, open the rack door at the rear of the system. 3. Review the locations of the PCIe adapter slots and the service indicators for the PCIe adapter that you want to service. The PCIe adapters are installed in the rear of the system. Figure 13 on page 20 shows the locations of the PCIe adapters in a 9040-MR9 system and the locations of the light- emitting diodes (LEDs) for the PCIe adapter. The number of the PCIe adapter slots that are available in the system depends on the number of system processors in the system. For the PCIe adapter placement information for the available slots in this system, see PCIe adapter placement rules and slot priorities for the 9040-MR9. The PCIe adapter has two LEDs that indicate the status: • A power/activity LED (green) • An error and identify function LED (amber) The status of the LEDs are as follows: • (B) Indicates that the PCIe adapter is operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is off. • (C) Indicates that the PCIe adapter is not operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is lit solid. • (D) Indicates the faulty or failed PCIe adapter was selected by using the identify function. The green power LED (top LED) is lit and the amber fault LED (bottom LED) is flashing. Attention: If the PCIe adapter is operating correctly, then the Activity LED is ON (green) and the Error amber LED is OFF, as shown in (B). Do not attempt to remove a working PCIe adapter. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 19

4. Choose from the following options: • If you want to remove a adapter when the system power is turned off, continue with step “5” on page 20. • If the system power is turned on and if the AIX operating system controls the slot, continue with step “7” on page 23. • If the system power is turned on and if the Linux operating system controls the slot, continue with step “8” on page 25. 5. To prepare the system to remove a PCIe adapter when the system is powered off, complete the following steps: a) Turn on the identify LED for the failing adapter. For instructions, see Identifying a part (www.ibm.com/support/knowledgecenter/POWER9/p9haj/ sal.htm). Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. b) Physically verify that the slot you identified is where you want to install the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. 20 Power Systems: PCIe adapters for the 9040-MR9

the identify function. c) If you have not yet recorded the PCIe slot details, record the slot number and location of each adapter that is being removed. Note: Adapter slots are numbered on the rear of the system. d) Stop the system. For instructions, see Stopping a system (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/crustopsys.htm). 6. If you powered the system off, complete the following steps: a) If applicable, open the rack door at the rear of the system. b) Label and disconnect the power cords from the system unit as shown in the following figure. Notes: • This system might be equipped with two or more power supplies. If the removing and replacing procedures require the system power to be turned off, ensure that all power sources to the system are disconnected. • The power cord (B) is fastened to the system with a hook-and-loop fastener (A). If you are placing the system in a service position after you disconnect the power cords, ensure that you unstrap the fastener. Figure 14. Removing the power cords Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 21

cords or multiple DC power cables. To remove all hazardous voltages, disconnect all power cords and power cables. (L003) c) Continue with step “10” on page 26. 7. To prepare to remove a PCIe adapter by using the AIX operating system, complete the following steps: a) You must take the PCIe adapter offline before you remove and replace a PCIe adapter when the system power is turned on (hot-plugging). Before you take an adapter offline, the devices that are attached to the adapter must be taken offline as well. This action must be done by a system administrator. Taking the PCIe adapter offline prevents a service representative or user from causing an unexpected outage for other users of the system. b) Before hot-plugging adapters or storage devices, ensure that the file systems on those devices are unmounted. c) Ensure that any processes or applications that might use the adapter are stopped. d) To identify the failed PCIe adapter, complete the steps in this procedure on the console: 1) Log in as root user. 2) At the command line, type diag and then press Enter. 3) From the Function Selection menu, select Task Selection > Hot Plug Task > PCI Hot Plug Manager > Unconfigure a Device. 4) Press F4 (or Esc +4) to display the Device Names menu. 5) Select the adapter that you are removing in the Device Names menu. 6) Use the Tab key to answer No to Keep Definition. Use the Tab key again to answer Yes to Unconfigure Child Devices, and then press Enter. Press Enter to confirm the operation. The OK message next to the Command field confirms a successful unconfiguration. 7) Press F3 (or Esc +3) twice to return to the Hot Plug Manager menu. 8) Select replace/remove PCI Hot Plug adapter and then select the slot that contains the adapter to be removed from the system. 9) Select remove. The amber LED for the PCI adapter flashes for the slot that is identified. 10) Physically verify that the slot you identified is where you want to remove the adapter. Notes: • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 23

identify function. 11) Press Enter. The adapter is placed in the action state and is ready to be removed from the system. e) If necessary to view the identify LEDs or to manage cables at the rear of a rack-mounted system, open the cable management arm by completing the following steps: 1) Release the mounting bracket from the cable management arm assembly by pressing the indentation (A) on the inner cable management arm tab. The cable management arm releases from the outer mounting bracket in the direction shown. Figure 15. Releasing the mounting bracket 2) Swing the cable management arm away from the system. In the image, the inner cable management arm tab is shown on the left side as you are facing the system from the rear. However, the cable management arm can also be installed so that the inner cable management arm tab is on the right side of the system. CAUTION: To avoid causing damage to the arm, open the cable management arm just long enough to complete these repairs. f) Physically verify that the slot you identified is where you want to remove the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. 24 Power Systems: PCIe adapters for the 9040-MR9

function. g) Continue with step “10” on page 26. 8. To prepare to remove a PCIe adapter by using the Linux operating system, complete the following steps: a) Before you hot-plug an adapter, ensure that the server or partition is at the correct level of the Linux operating system. b) Verify that the Linux, hot-plug PCI tools are installed. For instructions, see PCIe hot-plug manager menu (www.ibm.com/support/knowledgecenter/ POWER9/p9hak/pxhak_linuxhotplugverify.htm). c) Ensure that you have the POWER® Linux Service Aids installed on your system. These service aids enable system serviceability, as well to improve system management. If you are using a Linux on POWER distribution with Linux kernel version 2.6 or later, you can install the Service Aids that gives you access to more capabilities, which can help you diagnose problems on your system. This software is available at the Service and productivity tools for Linux on POWER website (http:// www14.software.ibm.com/webapp/set2/sas/f/lopdiags/home.html). d) You must take the PCIe adapter offline before you remove and replace a PCIe adapter when the system power is turned on (hot-plugging). Before you take an adapter offline, the devices that are attached to the adapter must be taken offline as well. This action must be done by a system administrator. Taking the PCIe adapter offline prevents a service representative or user from causing an unexpected outage for other users of the system. e) Before hot-plugging adapters or storage devices, ensure that the file systems on those devices are unmounted. f) Ensure that any processes or applications that might use the adapter are stopped. g) Use the service indicator light-emitting diodes (LEDs) to help identify the part. For instructions, see Identifying a part (www.ibm.com/support/knowledgecenter/POWER9/p9haj/sal.htm). h) Physically verify that the slot you identified is where you want to install the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for a flashing amber LED, which identifies the unoccupied slot that was selected by using the identify function. i) Use the Linux operating system drmgr command to prepare the adapter to be removed when the system power is turned on. 1) Type the following command: drmgr -c pci -r -s locationcode Where locationcode is replaced with the location you identified earlier. For example, U7879.001.DQD014E-P1-C3. The following screen is displayed. The visual indicator for the specified PCI slot has been set to the identify state. Press Enter to continue or enter x to exit. 2) Physically verify that the slot you identified is where you want to remove the adapter. Notes: • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for the flashing amber LED, which identifies the slot that was selected by using the identify function. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 25

instructions until you are told to remove the adapter. 9. If applicable, open the rack door at the rear of the system. 10. At the rear of the system, put the cable management bracket in the service position. See the following figure. a) Pull out the quarter-turn fasteners (B) and turn them to disengage them while you lift the cable management bracket (A) to its raised position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. Figure 16. Placing the cable management bracket in the raised service position 11. Label and disconnect all of the cables or transceivers that are attached to the adapter you plan to remove. Use a hook-and-loop fastener to hold the cables in place. Important: If you are disconnecting the SAS AZ cable which controls the internal disk bays, use extra care when removing cables from the adapter connectors. Gently pull the blue tab straight back from the system to release the latches that secure the SAS cable to the PCIe adapter. To prevent possible tab breakage, Do not pull the blue tab to the side; instead, pull the blue tab straight back. While gently pulling the blue tab, unplug and remove the SAS cable from the PCIe adapter by pulling on the black SAS cable itself. Do not apply extra pulling force on the blue tab to remove cable. 12. Attach the electrostatic discharge (ESD) wrist strap. 26 Power Systems: PCIe adapters for the 9040-MR9

completed, and if applicable, until the service access cover is replaced. Attention: • Attach an electrostatic discharge (ESD) wrist strap to the front ESD jack, to the rear ESD jack, or to an unpainted metal surface of your hardware to prevent the electrostatic discharge from damaging your hardware. • When you use an ESD wrist strap, follow all electrical safety procedures. An ESD wrist strap is used for static control. It does not increase or decrease your risk of receiving electric shock when using or working on electrical equipment. • If you do not have an ESD wrist strap, just prior to removing the product from ESD packaging and installing or replacing hardware, touch an unpainted metal surface of the system for a minimum of 5 seconds. If at any point in this service process you move away from the system, it is important to again discharge yourself by touching an unpainted metal surface for at least 5 seconds before you continue with the service process. Removing a PCIe adapter from the 9040-MR9 system To remove a PCIe adapter from the system, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Remove the PCIe cassette from the slot: a) Squeeze the latch lever (A) and press down on the latch (B) in the direction shown. This action pivots the cassette forward and releases the cassette from the slot. b) Pull out the cassette from its slot. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 27

3. Choose from the following options: • To remove a PCIe adapter from a cassette that has adapter retainers and strips, continue with step “4” on page 28. • To remove a PCIe adapter from a cassette that doesn't have adapter retainers and strips, continue with step “5” on page 29. 4. If your cassette has two adapter retainers and strips, complete the following steps to remove a PCIe adapter from a cassette. a) Open the tailstock clamp (A) by rotating the clamp in the direction that is shown. b) Remove the adapter retainers (B) by rotating them in the direction that is shown and moving them all the way off the strips. c) Open the cassette by pressing and sliding the release bar (C) in the direction that is shown. d) Remove the PCIe adapter. 28 Power Systems: PCIe adapters for the 9040-MR9

5. If your cassette doesn't have two adapter retainers and strips, complete the following steps to remove the PCIe adapter from the cassette. a) Open the tailstock clamp (A) by rotating the clamp in the direction that is shown. b) Open the cassette by pressing and sliding the release bar (B) in the direction that is shown. c) Remove the PCIe adapter. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 29

6. Place the removed adapter on an approved ESD surface. Replacing a PCIe adapter in the 9040-MR9 system To replace a PCIe adapter in the system, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Ensure that the target slot is empty. 3. If necessary, remove the adapter from the antistatic package. Attention: Avoid touching the components and gold connectors on the adapter. 4. Place the adapter, component-side up, on a flat, antistatic surface. Attention: A pin on the tailstock of the adapter resembles a removable screw. Do not remove this pin. It is required for correct alignment and seating. 5. Choose from the following options: • To replace a PCIe adapter into a cassette that has adapter retainers and strips, continue with step “6” on page 30. • To replace a PCIe adapter into a cassette that does not have adapter retainers and strips, continue with step “8” on page 33. 6. If your cassette has two adapter retainers with strips, complete the following steps to install or replace a PCIe adapter into the cassette: a) If the adapter retainers (A) haven't already been removed, remove the adapter retainers (A) by putting them in the open position and moving them all the way off the strips. 30 Power Systems: PCIe adapters for the 9040-MR9

c) Place an ESD safety sheet (C) over the strips. Note: You can use the ESD package that the new adapter was in. If necessary, cut it to size. Figure 20. Placing an ESD safety sheet over the strips on the 9040-MR9 cassette d) Insert the adapter into the cassette. e) Press and slide the release bar (A) to close the cassette. f) Close the tailstock clamp (B) by rotating the clamp in the direction that is shown. g) Pull the ESD safety film (C) out. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 31

h) With the adapter retainers (D) in the open position, slide the retainers onto the strips. i) Position the adapter retainers (D) so that they support the adapter and put them in the locked position by rotating them in the direction that is shown. Note: Ensure that the adapter is secured in the adapter retainer channel. 32 Power Systems: PCIe adapters for the 9040-MR9

7. Continue with step “9” on page 34. 8. If your cassette does not have the two adapter retainers and strips, complete the following steps to install or replace a PCIe adapter into the cassette: a) Press and slide the release bar (A) to open the cassette. b) Insert the adapter into the cassette. c) Press and slide the release bar (A) to close the cassette. d) Close the tailstock clamp (B) by rotating the clamp in the direction that is shown. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 33

9. Place the cassette with the adapter in it on an antistatic surface. 10. Choose from the followings options: a) If the system power is turned off, continue with step “14” on page 34. b) If the system power is turned on and you are using the AIX operating system, continue with step “11” on page 34. c) If the system power is turned on and you are using the Linux operating system, continue with step “13” on page 34. 11. If you are using the AIX operating system, use the diag command on the console to prepare the slot to accept a PCIe adapter. a) Select Add a PCI Hot Plug adapter from the Hot Plug Manager menu. b) Select the slot from where you removed the adapter. c) Press Enter again to put the slot into the action state. A fast-flashing amber LED at the rear of the system near the adapter indicates that the slot is identified and is ready to receive the adapter. 12. Continue with step “14” on page 34. 13. If you are using the Linux operating system, use the drmgr command on the console to prepare the slot to accept a PCIe adapter. For example, to install the PCIe adapter in slot U7879.001.DQD014E-P1-C3, type: drmgr -c pci -r -s locationcode In this example, you would replace locationcode with U7879.001.DQD014E-P1-C3. Follow the instructions on the display to complete the task. 14. Slide the PCIe cassette into the slot by completing the following steps: 34 Power Systems: PCIe adapters for the 9040-MR9

a) Align the cassette on the cassette rail in the cassette slot. b) Slide the cassette forward until the cassette is fully seated. c) To lock the cassette in its slot, squeeze the latch lever (A) toward the latch (B) and rotate the latch (B) in the direction that is shown. Figure 24. Installing a cassette in the 9040-MR9 system Preparing the 9040-MR9 system for operation after removing and replacing a PCIe adapter To prepare the system for operation after permanently removing a PCIe adapter, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Connect any cables or transceivers to the adapter. 3. Route the cables through the cable management bracket. 4. If you removed the power cords, connect all power cords to the system that you are servicing. 5. Put the cable management bracket in the operating position. See the following figure. a) Pull out the quarter-turn fasteners (B) and turn them to disengage them while moving the cable management bracket (A) to its lower position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 35

6. Close the rack door at the rear of the system. 7. Choose from the following options: • If you completed this procedure with the system power turned off, continue with step “8” on page 36. • If you completed the procedure with the system power turned on and used the AIX operating system, continue with step “9” on page 36. • If you completed the procedure with the system power turned on and used the Linux operating system, continue with step “11” on page 37. 8. To prepare the system for operation when the system is powered off, complete the following steps: a) Start the system. For instructions, see Starting a system (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/crustartsys.htm). b) Continue with step “12” on page 37. 9. If you are using the AIX operating system, complete the following steps to configure the adapter: a) Press the F3 key to return to the Task Selection list. b) Select Log Repair Action. c) Select the resource that was replaced and select Commit. d) Press the F3 key to return to Task Selection List. 36 Power Systems: PCIe adapters for the 9040-MR9

f) Select the device that you replaced from the list, and then press Enter. The device is now configured. g) Press the F10 key to exit the diagnostic program. 10. If you are using the AIX operating system, complete the following steps to verify the PCIe adapter replacement: a) If the diagnostic menu is not displayed, type the diag command. b) Select Advance Diagnostic Routines > Problem Determination. c) Select the name of the resource that was replaced from the menu. If the resource replaced is not shown, choose the resource that is associated with it. d) Press Enter, and then press Commit (F7 or Esc+7). e) Did the Problem Determination identify any problems? • No: Continue to the next step. • Yes: A problem is identified. – If you are a customer, record the error information, and then contact your service provider. – If you are an authorized service provider, return to map 210-5. f) Press the F10 key to exit the diagnostic program. g) Continue with step “12” on page 37. 11. If you are using the Linux operating system, complete the following steps: a) In the Linux session on the console, press Enter after you install or replace the adapter to put the slot into the action state. b) Enter the slot information with the lsslot command, as shown in the following example. For example, if the slot in which you installed the PCIe adapter was U7879.001.DQD014E-P1-C3. Enter lsslot -c pci -s U7879.001.DQD014E-P1-C3 The following screen is an example of the information that is displayed by this command: # Slot Description Device(s) U7879.001.DQD014E-P1-C3 PCI-X capable, 64 bit, 133MHz slot 0001:40:01.0 12. Verify the installed part. • If you replaced the part because of a service action, verify the installed part. For instructions, see https://www.ibm.com/support/knowledgecenter/POWER9/p9ect/ pxect_verifyrepair.htm (www.ibm.com/support/knowledgecenter/POWER9/p9ect/ pxect_verifyrepair.htm). • If you installed the part for any other reason, verify the installed part. For instructions, see https:// www.ibm.com/support/knowledgecenter/POWER9/p9haj/pxhaj_hsmverify.htm (www.ibm.com/ support/knowledgecenter/POWER9/p9haj/pxhaj_hsmverify.htm). 13. Turn off the identify LED. For instructions, see Deactivating an identify LED (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/p9haj_turn_off_identify_led.htm). 14. If you replaced a Fibre Channel adapter, the customer might need to update the Fibre Channel SAS path to the external storage subsystems with the new WWPN number or numbers. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 37

Find information about permanently removing Peripheral Component Interconnect Express (PCIe) adapters from the IBM Power System E950 (9040-MR9) server. About this task Note: Removing or replacing this feature is a customer task. You can complete this task yourself, or contact a service provider to complete the task for you. You might be charged a fee by the service provider for this service. If your system is managed by the Hardware Management Console (HMC), use the HMC to repair a part in the system. For instructions, see Repairing a part by using the HMC (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/p9haj_hmc_repair.htm). If your system is not managed by an HMC, complete the steps in the following procedures to permanently remove a PCIe adapter from the system. Preparing the 9040-MR9 system to permanently remove a PCIe adapter To prepare the system to permanently remove a PCIe adapter from the system, complete the steps in this procedure. Procedure 1. Take appropriate precautions for avoiding electric shock and for handling static-sensitive devices. For more information, see “Avoiding electric shock” on page 49 and “Handling static-sensitive devices” on page 51. 2. Review the locations of the PCIe adapter slots and the service indicators for the PCIe adapter that you want to service. The PCIe adapters are installed in the rear of the system. Figure 26 on page 39 shows the locations of the PCIe adapters in a 9040-MR9 system and the locations of the light- emitting diodes (LEDs) for the PCIe adapter. The number of the PCIe adapter slots that are available in the system depends on the number of system processors in the system. For the PCIe adapter placement information for the available slots in this system, see PCIe adapter placement rules and slot priorities for the 9040-MR9. The PCIe adapter has two LEDs that indicate the status: • A power/activity LED (green) • An error and identify function LED (amber) The status of the LEDs are as follows: • (B) Indicates that the PCIe adapter is operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is off. • (C) Indicates that the PCIe adapter is not operating correctly. The green power LED (top LED) is lit solid and the amber fault LED (bottom LED) is lit solid. • (D) Indicates the faulty or failed PCIe adapter was selected by using the identify function. The green power LED (top LED) is lit and the amber fault LED (bottom LED) is flashing. Attention: If the PCIe adapter is operating correctly, then the Activity LED is ON (green) and the Error amber LED is OFF, as shown in (B). Do not attempt to remove a working PCIe adapter. 38 Power Systems: PCIe adapters for the 9040-MR9

3. Use the service indicator light-emitting diodes (LEDs) to help identify the part. For instructions, see Identifying a part (www.ibm.com/support/knowledgecenter/POWER9/p9haj/sal.htm). 4. If applicable, open the rack door at the rear of the system. 5. Physically verify that the slot you identified is where you want to remove the adapter. • Use the blue identify LED on the enclosure to locate the system. Ensure that the serial number of the system matches the serial number to be serviced. • Look for a flashing amber LED, which identifies the slot that was selected by using the identify function. 6. If you have not yet recorded the PCIe slot details, record the slot number and location of each adapter that is being removed. Note: Adapter slots are numbered on the rear of the system. 7. Stop the system. For instructions, see Stopping a system (www.ibm.com/support/knowledgecenter/ POWER9/p9haj/crustopsys.htm). 8. Label and disconnect the power cords from the system unit as shown in the following figure. Notes: • This system might be equipped with two or more power supplies. If the removing and replacing procedures require the system power to be turned off, ensure that all power sources to the system are disconnected. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 39

the system in a service position after you disconnect the power cords, ensure that you unstrap the fastener. Figure 27. Removing the power cords (L003) or 40 Power Systems: PCIe adapters for the 9040-MR9

or multiple DC power cables. To remove all hazardous voltages, disconnect all power cords and power cables. (L003) 9. At the rear of the system, put the cable management bracket in the service position. See the following figure. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 41

management bracket (A) to its raised position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. Figure 28. Placing the cable management bracket in the raised service position 10. Label and disconnect all of the cables or transceivers that are attached to the adapter you plan to remove. Use a hook-and-loop fastener to hold the cables in place. Important: If you are disconnecting the SAS AZ cable which controls the internal disk bays, use extra care when removing cables from the adapter connectors. Gently pull the blue tab straight back from the system to release the latches that secure the SAS cable to the PCIe adapter. To prevent possible tab breakage, Do not pull the blue tab to the side; instead, pull the blue tab straight back. While gently pulling the blue tab, unplug and remove the SAS cable from the PCIe adapter by pulling on the black SAS cable itself. Do not apply extra pulling force on the blue tab to remove cable. 11. Attach the electrostatic discharge (ESD) wrist strap. The ESD wrist strap must be connected to an unpainted metal surface until the service procedure is completed, and if applicable, until the service access cover is replaced. Attention: 42 Power Systems: PCIe adapters for the 9040-MR9

jack, or to an unpainted metal surface of your hardware to prevent the electrostatic discharge from damaging your hardware. • When you use an ESD wrist strap, follow all electrical safety procedures. An ESD wrist strap is used for static control. It does not increase or decrease your risk of receiving electric shock when using or working on electrical equipment. • If you do not have an ESD wrist strap, just prior to removing the product from ESD packaging and installing or replacing hardware, touch an unpainted metal surface of the system for a minimum of 5 seconds. If at any point in this service process you move away from the system, it is important to again discharge yourself by touching an unpainted metal surface for at least 5 seconds before you continue with the service process. Permanently removing a PCIe adapter from the 9040-MR9 system To permanently remove a PCIe adapter from the system, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Remove the PCIe cassette from the slot: a) Squeeze the latch lever (A) and press down on the latch (B) in the direction shown. This action pivots the cassette forward and releases the cassette from the slot. b) Pull out the cassette from its slot. Figure 29. Removing a PCIe adapter cassette from the 9040-MR9 system 3. Choose from the following options: Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 43

“4” on page 44. • To remove a PCIe adapter from a cassette that doesn't have adapter retainers and strips, continue with step “5” on page 44. 4. If your cassette has two adapter retainers and strips, complete the following steps to remove a PCIe adapter from a cassette. a) Open the tailstock clamp (A) by rotating the clamp in the direction that is shown. b) Remove the adapter retainers (B) by rotating them in the direction that is shown and moving them all the way off the strips. c) Open the cassette by pressing and sliding the release bar (C) in the direction that is shown. d) Remove the PCIe adapter. Figure 30. Removing a PCIe adapter from a cassette for the 9040-MR9 system 5. If your cassette doesn't have two adapter retainers and strips, complete the following steps to remove the PCIe adapter from the cassette. a) Open the tailstock clamp (A) by rotating the clamp in the direction that is shown. b) Open the cassette by pressing and sliding the release bar (B) in the direction that is shown. c) Remove the PCIe adapter. 44 Power Systems: PCIe adapters for the 9040-MR9

6. Place the removed adapter on an approved ESD surface. 7. After permanently removing a PCIe adapter, you must install a PCIe tailstock filler into the cassette to ensure proper airflow. a) Open the tailstock clamp (A) in the direction that is shown. b) Insert the tailstock filler firmly into the tailstock retaining channel and ensure that it is seated. c) Rotate the tailstock filler toward the front of the cassette and into place. d) Put the adapter retainers (B) into the locked position by rotating them in the direction that is shown. e) Close the tailstock clamp (A) by rotating the clamp in the direction that is shown. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 45

8. Slide the PCIe cassette into the slot by completing the following steps: Attention: Ensure proper alignment when you insert a PCI adapter cassette into the system. a) Align the cassette on the cassette rail in the cassette slot. b) Slide the cassette forward until the cassette is fully seated. c) To lock the cassette in its slot, squeeze the latch lever (A) toward the latch (B) and rotate the latch (B) in the direction that is shown. 46 Power Systems: PCIe adapters for the 9040-MR9

Preparing the 9040-MR9 system for operation after permanently removing a PCIe adapter To prepare the system for operation after permanently removing a PCIe adapter, complete the steps in this procedure. Procedure 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Choose from the following options: • If you completed the procedure with the system power turned off, continue with step “3” on page 47. • If you completed the procedure with the system power turned on, continue with step “4” on page 48. 3. Using your labels, reconnect the power cords (A) to the system unit as shown in the following figure. Fasten the power cords to the system by using the hook-and-loop fasteners (B) as shown in the following figure. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 47

4. Put the cable management bracket in the operating position. See the following figure. a) Pull out the quarter-turn fasteners (B) and turn them to disengage them while moving the cable management bracket (A) to its lower position. b) Turn the quarter-turn fasteners (B) to engage and lock the bracket into position. 48 Power Systems: PCIe adapters for the 9040-MR9

5. Close the rack door at the rear of the system. 6. Start the system. For instructions, see Starting a system (www.ibm.com/support/knowledgecenter/ POWER9/p9haj/crustartsys.htm). 7. Place the removed adapter in a safe place. Related procedures for installing and removing PCIe adapters Find procedures that are related to installing and removing PCIe adapters. Avoiding electric shock Find information about precautions you should take to avoid electric shock when working on our around a computer system. DANGER: When working on or around the system, observe the following precautions: Electrical voltage and current from power, telephone, and communication cables are hazardous. To avoid a shock hazard: • If IBM supplied the power cord(s), connect power to this unit only with the IBM provided power cord. Do not use the IBM provided power cord for any other product. • Do not open or service any power supply assembly. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 49

of this product during an electrical storm. • The product might be equipped with multiple power cords. To remove all hazardous voltages, disconnect all power cords. – For AC power, disconnect all power cords from their AC power source. – For racks with a DC power distribution panel (PDP), disconnect the customer’s DC power source to the PDP. • When connecting power to the product ensure all power cables are properly connected. – For racks with AC power, connect all power cords to a properly wired and grounded electrical outlet. Ensure that the outlet supplies proper voltage and phase rotation according to the system rating plate. – For racks with a DC power distribution panel (PDP), connect the customer’s DC power source to the PDP. Ensure that the proper polarity is used when attaching the DC power and DC power return wiring. • Connect any equipment that will be attached to this product to properly wired outlets. • When possible, use one hand only to connect or disconnect signal cables. • Never turn on any equipment when there is evidence of fire, water, or structural damage. • Do not attempt to switch on power to the machine until all possible unsafe conditions are corrected. • Assume that an electrical safety hazard is present. Perform all continuity, grounding, and power checks specified during the subsystem installation procedures to ensure that the machine meets safety requirements. • Do not continue with the inspection if any unsafe conditions are present. • Before you open the device covers, unless instructed otherwise in the installation and configuration procedures: Disconnect the attached AC power cords, turn off the applicable circuit breakers located in the rack power distribution panel (PDP), and disconnect any telecommunications systems, networks, and modems. DANGER: • Connect and disconnect cables as described in the following procedures when installing, moving, or opening covers on this product or attached devices. To Disconnect: 1. Turn off everything (unless instructed otherwise). 2. For AC power, remove the power cords from the outlets. 3. For racks with a DC power distribution panel (PDP), turn off the circuit breakers located in the PDP and remove the power from the Customer's DC power source. 4. Remove the signal cables from the connectors. 5. Remove all cables from the devices. To Connect: 1. Turn off everything (unless instructed otherwise). 2. Attach all cables to the devices. 3. Attach the signal cables to the connectors. 4. For AC power, attach the power cords to the outlets. 5. For racks with a DC power distribution panel (PDP), restore the power from the Customer's DC power source and turn on the circuit breakers located in the PDP. 6. Turn on the devices. 50 Power Systems: PCIe adapters for the 9040-MR9

handling equipment to avoid cuts, scrapes and pinching. (D005) Handling static-sensitive devices Find information about precautions you must take to prevent damage to electronic components from static electricity discharge. Electronic boards, adapters, media drives, and disk drives are sensitive to static electricity discharge. These devices are wrapped in antistatic bags to prevent this damage. Take the following precautions to prevent damage to these devices from static electricity discharge. • Attach a wrist strap to an unpainted metal surface of your hardware to prevent electrostatic discharge from damaging your hardware. • When you are using a wrist strap, follow all electrical safety procedures. A wrist strap is for static control. It does not increase or decrease your risk of receiving electric shock when you are using or working on electrical equipment. • If you do not have a wrist strap, before you remove the product from ESD packaging and installing or replacing hardware, touch an unpainted metal surface of the system for a minimum of 5 seconds. • Do not remove the device from the antistatic bag until you are ready to install the device in the system. • With the device still in its antistatic bag, touch it to the metal frame of the system. • Grasp cards and boards by the edges. Avoid touching the components and gold connectors on the adapter. • If you need to lay the device down while it is out of the antistatic bag, lay it on the antistatic bag. Before you pick it up again, touch the antistatic bag and the metal frame of the system at the same time. • Handle the devices carefully to prevent permanent damage. Updating the worldwide port name for a new 5735 IOA If you exchange a 5735 Fibre Channel input/output adapter (IOA), the IBM external storage subsystem must be updated to use the worldwide port name (WWPN) of the new 5735 IOA. Any SAN hardware that uses the WWPN zoning might also need updating. For instructions on how to update the external storage subsystem or SAN hardware configurations, see the documentation for those systems. The WWPN for the Fibre Channel IOA can be found by using the Hardware Service Manager in SST or DST to display details about the 5735 IOA Logical Hardware Resource information, and use the port worldwide name field. The 16-digit WWPN can also be determined by appending the digits 1000 to the beginning of the 12-digit IEEE address that is found on the tailstock label of the Fibre Channel IO adapter. Removing and replacing the tailstock on a PCIe adapter Find information about removing and replacing the tailstock on a PCIe adapter. You can replace the tailstock on an adapter if the adapter has a different tailstock. Before you begin You must have removed the PCIe adapter or the adapter filler from the system. About this task The tailstock for individual PCIe adapters can be either tall (high-profile) or short (low-profile). New systems ship with the correct tailstock on the cards already installed. Through unique feature codes, a miscellaneous equipment specification (MES) also ships with the correct sized tailstock. For repairs, however, replacement parts usually ship with high-profile tailstock. If the failed adapter was a low-profile adapter installed in the system, then you must exchange the tall tailstock that was shipped on the replacement card with the low-profile tailstock from the failed card. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 51

To remove and replace a PCIe adapter tailstock, choose from the following options: • Removing the tailstock from a failed adapter or adapter filler • Removing the tailstock from a new adapter • Replacing the tailstock in a new adapter Procedure Removing the tailstock from a failed adapter or adapter filler 1. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 2. Using a Phillips screwdriver, unscrew the two screws (A) that attach the tailstock (B) to the PCIe adapter, as shown in Figure 36 on page 52. Figure 36. Removing the low-profile tailstock from a PCIe adapter 3. Detach the tailstock from the PCIe adapter and place the tailstock and the screws on a flat, static- protective surface. 4. If you are removing the tailstock from a 4-Port 10/100/1000 Base-TX PCIe adapter, there is only one screw to remove. After removing the screw, disengage the tabs from the card hardware and remove the tailstock. 5. If you are removing the tailstock from an Ethernet or Fibre Channel adapter that has small form- factor pluggable (SFP) transceivers, you must first remove the SFP transceivers. To remove the SFP transceivers, complete the following steps: a) Place the card flat on a static-protective surface. 52 Power Systems: PCIe adapters for the 9040-MR9

c) Slide the transceiver (B) out of the adapter hardware. d) Repeat for the other transceiver. Figure 37. Removing SFP transceivers from adapters 6. If necessary, remove the adapter from the antistatic package. Attention: Avoid touching the components and gold connectors on the adapter. Removing the tailstock from a new adapter 7. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 8. Using a Phillips screwdriver, unscrew the two retaining screws (A) that attach the high-profile tailstock (B) to the new PCIe adapter, as shown in Figure 38 on page 54. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 53

9. If the new adapter is a 4-Port 10/100/1000 Base-TX PCIe adapter, see “4” on page 52 for removing the tailstock and then return here. 10. If the new adapter is an Ethernet or Fibre Channel adapter, see “5” on page 52 for removing the tailstock and then return here. 11. Detach the tailstock from the new PCIe adapter and store the tailstock and its screws in the antistatic package that the new adapter shipped in. Replacing the tailstock in a new adapter 12. Ensure that you have the electrostatic discharge (ESD) wrist strap on and that the ESD clip is plugged into a ground jack or connected to an unpainted metal surface. If not, do so now. 13. Place the adapter, component-side up, on a flat, antistatic surface. Attention: A pin on the tailstock of the adapter resembles a removable screw. Do not remove this pin. It is required for correct alignment and seating. 14. Using a Phillips screwdriver and the two retaining screws (B), attach the tailstock (A) from the failed PCIe adapter onto the new PCIe adapter, as shown in Figure 39 on page 55. 54 Power Systems: PCIe adapters for the 9040-MR9

15. If the new adapter is a 4-Port 10/100/1000 Base-TX PCI Express adapter, first engage the tailstock with the tabs on the card hardware. After the tailstock is attached, secure the tailstock with its screw. 16. If the new adapter is an Ethernet or Fibre Channel adapter, you must attach the SFP transceivers after you replace the tailstock. To attach the SFP transceivers, complete the following steps: a) Place the card flat on a static-protective surface. b) Slide the transceiver (A) into the adapter hardware. c) With the latch (B) in the open position, push the latch by rotating it up and toward the card as shown in Figure 40 on page 56. d) Repeat for the other transceiver. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 55

17. Put the failed PCIe adapter in the shipping material and return it. What to do next Return to the procedure that sent you here. PCIe hot-plug manager access for AIX Find information about how to service PCIe adapters with the system power on in AIX. The instructions for servicing PCIe adapters with the system power on in AIX refer you to these procedures when it is appropriate to perform them. Note: For an adapter to be serviced with the system power on, both the adapter and the system unit must support hot-plug procedures. To identify adapters that are hot-pluggable in the system you are servicing, refer to the following placement information: PCI adapter placement. Accessing hot-plug management functions Find information about how to use the PCI hot-plug manager to service PCIe adapters with the system power turned on in AIX. About this task Note: Procedures that are completed with the system power turned on also known as hot-plug procedures. The hot-plug procedure for PCIe adapter in an AIX system requires the system administrator to take the PCIe adapter offline before starting the procedure. Before taking an adapter offline, the devices attached to the adapter must be taken offline as well. This action prevents a service representative or user from causing an unexpected outage for system users. To access the hot-plug menus, complete the following steps: Procedure 1. Log in as root user. 56 Power Systems: PCIe adapters for the 9040-MR9

3. Select Devices. 4. Select PCI Hot Plug Manager and press Enter. The PCI Hot-Plug Manager menu displays. For details about the menu options in the PCI Hot-Plug Manager screen, see “PCI hot-plug manager menu” on page 57. What to do next Return to the procedure that directed you here. PCI hot-plug manager menu Find information about how to use the PCI Hot Plug Manager to service PCI adapters with the system power on in AIX. The following options are available from the PCI Hot Plug Manager menu. Note: For information about the PCI slot LED states, see “Component LEDs” on page 58. Table 1. PCI Hot Plug Manager menu options Menu option Description List PCI hot-plug slots Provides a descriptive list of all slots that support PCI hot-plug capability. If the listing for a slot indicates it holds an unknown device, select the Install/configure Devices added after IPL to configure the adapter in that slot. Add a PCI hot- plug adapter Allows the user to add a new PCI hot-plug-capable adapter to the slot with the system turned on. You will be asked to identify the PCI slot that you have selected prior to the actual operation. The selected PCI slot will go into the Action state and finally into the On state. Note: The system will indicate the slot holds an unknown device until you perform the Install/configure devices added after IPL option to configure the adapter. Replace/remove a PCI hot-plug adapter Allows the user to remove an existing adapter, or replace an existing adapter with an identical one. For this option to work, the adapter must be in the Defined state (see the "Unconfigure a Device" option). You will be asked to identify the PCI slot prior to the actual operation. The selected PCI slot will go into the Action state. Identify a PCI hot-plug slot Allows the user to identify a PCI slot. The selected PCI slot will go into the Identify state. See “Component LEDs” on page 58. Unconfigure a device Allows the user to put an existing PCI adapter into the Defined state if the device is no longer in use. This step must be completed successfully before starting any removal or replacement operation. If this step fails, the customer must take action to release the device. Configure a defined device Allows a new PCI adapter to be configured into the system if software support is already available for the adapter. The selected PCI slot will go into the On state. Install/configure devices added after IPL The system attempts to configure any new devices and tries to find and install any required software from a user-selected source. The add, remove, and replace functions return information to the user indicating whether the operation was successful. If additional instructions are provided on the screen, complete the recommended actions. If the instructions do not resolve the problem, do the following: • If the adapter is listed as Unknown, perform the Install/configure devices Added After IPL option to configure the adapter. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 57

administrator must install the specified packages before you can configure or diagnose the adapter. • If you receive a failure message indicating a hardware error, the problem might be either the adapter or the PCI slot. Isolate the problem by retrying the operation in a different PCI slot, or trying a different adapter in the slot. If you determine that you have failing hardware, call your service representative. • Do not use Install/configure devices added after IPL if your system is set up to run HACMP clustering. Consult with your system administrator or software support to determine the correct method to configure the replacement device. Component LEDs Individual LEDs are located on or near the failing components. Use the information in this section to interpret the LEDs. The LEDs are located either on the component itself or on the carrier of the component (for example, memory card, fan, memory module, or processor). LEDs are either green or amber. Green LEDs indicate either of the following: • Electrical power is present. • Activity is occurring on a link. (The system could be sending or receiving information.) Amber LEDs indicate a fault or identify condition. If your system or one of the components on your system has an amber LED turned on or flashing, identify the problem and take the appropriate action to restore the system to normal. Resetting the LEDs in AIX Individual LEDs are located on or near the failing components. You can use this procedure to reset the LEDs after you have completed a repair action. About this task After the repair action is complete, perform the following tasks: Procedure 1. Log in as root user. 2. At the command line, type diag. 3. Select Task Selection. 4. Select Log Repair Action. 5. Select the device that was repaired. 6. Press F10 to exit diagnostics. Results If the Attention LED remains on after you have completed the repair action and reset the LEDs, call for service support. Installing the AIX device driver software Find information about how to install the AIX device driver software for a PCIe adapter. Before you begin If you are installing the AIX operating system at this time, install the adapter before you install the operating system. When you install the AIX operating system, the adapter device driver is automatically installed and the following procedure does not apply to your situation. About this task If you are installing only the device driver for a PCIe adapter, perform these steps: 58 Power Systems: PCIe adapters for the 9040-MR9

2. Insert the media containing the device driver software (for example, the CD) into the media device. If your system does not have a CD-ROM drive, refer to your system documentation for performing a Network Installation Management (NIM) installation. 3. Type the following command at the System Management Interface Tool (SMIT) fast path: smit devinst. Press Enter. The Install Additional Device Software window appears with INPUT device / directory for software option highlighted. 4. Type the name of the input device that you are using, or press F4 to select the input device from a list. Press Enter. The Install Additional Device Software window is shown with the SOFTWARE to install option highlighted. 5. Press F4 to select List. 6. Type / to display the Find window. 7. Type the device package name, and press Enter. The device driver software is shown. 8. Press F7 to select the highlighted device driver software, and press Enter. The Install Additional Device Software window is displayed. The entry fields are automatically updated. Press Enter twice to confirm. The Command Status window is displayed. • The message RUNNING is highlighted to indicate that the installation and configuration command is in progress. • When RUNNING changes to OK, scroll to the bottom of the page and locate the installation summary. • After a successful installation, SUCCESS is displayed in the Result column of the installation summary at the bottom of the page. 9. Remove the installation media from the drive. 10. Press F10 to exit SMIT. 11. To verify that the device drives is installed, complete the following steps: a) If necessary, log in as root user. b) At the command line, type lslpp -l devices.xxxxxxxxxx where xxxxxxxxxx is the device package name. Press Enter. The following example shows when the adapter device driver is installed. File set Level State Description Path: /usr/lib/objrepos devices.xxxxxxxxxx 5.3.8.0 COMMITTED Adapter name software Verify that the file sets are installed at the AIX version level you are running. Level 5.3.8.0 is an example. If no data is displayed on your screen, the adapter device driver was not installed correctly. Try reinstalling the driver. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 59

Server Find information about how to install or replace a PCIe adapter in the Virtual I/O Server logical partition. Before you begin The Virtual I/O Server includes a PCIe Hot Plug Manager that is similar to the PCIe Hot Plug Manager in the AIX operating system. The PCIe Hot Plug Manager allows you to hot plug PCIe adapters into the system and then activate them for the logical partition without having to reboot the system. Use the PCIe Hot Plug Manager for adding, identifying, or replacing PCIe adapters in the system that are currently assigned to the Virtual I/O Server. Getting started Before you begin Prerequisites: • If you are installing a new adapter, an empty system slot must be assigned to the Virtual I/O Server logical partition. This task can be done through dynamic logical partitioning (DLPAR) operations. • If you are using a Hardware Management Console (HMC), you must also update the logical partition profile of the Virtual I/O Server so that the new adapter is configured to the Virtual I/O Server after you restart the system. • If you are installing a new adapter, ensure that you have the software required to support the new adapter and determine whether there are any existing PTF prerequisites to install. For information about software prerequisites, see the IBM Prerequisite website (http://www-912.ibm.com/e_dir/ eServerPrereq.nsf). About this task Follow these steps to access the PCIe Hot Plug Manager from the Virtual I/O Server: Procedure 1. Use the diagmenu command to open the Virtual I/O Server diagnostic menu. The menus are similar to the AIX diagnostic menus. 2. Select Task Selection, then press Enter. 3. At the Task Selection list, select PCIe Hot Plug Manager. Installing a PCIe adapter About this task To install a PCIe adapter with the system power on in Virtual I/O Server, complete the following steps: Procedure 1. From the PCIe Hot Plug Manager, select Add a PCIe Hot Plug Adapter, then press Enter. The Add a Hot-Plug Adapter window is displayed. 2. Select the appropriate empty PCIe slot from those listed, and press Enter. A fast-blinking amber LED located at the back of the server near the adapter indicates that the slot has been identified. 3. Follow the instructions on the screen to install the adapter until the LED for the specified PCIe slot is set to the Action state. a. Set the adapter LED to the action state so that the indicator light for the adapter slot flashes b. Physically install the adapter 60 Power Systems: PCIe adapters for the 9040-MR9

4. Enter cfgdev to configure the device for the Virtual I/O Server. Results If you are installing a PCIe, Fibre Channel adapter, it is now ready to be attached to a SAN and have LUNs assigned to the Virtual I/O Server for virtualization. Replacing a PCIe adapter Before you begin Prerequisite: Before you can remove or replace a storage adapter, you must unconfigure that adapter. See “Unconfiguring storage adapters” on page 61 for instructions. About this task To replace a PCIe adapter with the system power turned on in Virtual I/O Server, complete the following steps: Procedure 1. From the PCIe Hot Plug Manager, select Unconfigure a Device, then press Enter. 2. Press F4 (or Esc +4) to display the Device Names menu. 3. Select the adapter you are removing in the Device Names menu. 4. In the Keep Definition field, use the Tab key to answer Yes. In the Unconfigure Child Devices field, use the Tab key again to answer YES, then press Enter. 5. Press Enter to verify the information on the ARE YOU SURE screen. Successful unconfiguration is indicated by the OK message displayed next to the Command field at the top of the screen. 6. Press F4 (or Esc +4) twice to return to the Hot Plug Manager. 7. Select replace/remove PCIe Hot Plug adapter. 8. Select the slot that has the device to be removed from the system. 9. Select replace. A fast-blinking amber LED located at the back of the machine near the adapter indicates that the slot has been identified. 10. Press Enter which places the adapter in the action state, meaning it is ready to be removed from the system. Unconfiguring storage adapters About this task Before you can remove or replace a storage adapter, you must unconfigure that adapter. Storage adapters are generally parent devices to media devices, such as disk drives or tape drives. Removing the parent requires that all attached child devices either be removed or placed in the define state. Unconfiguring a storage adapter involves the following tasks: • Closing all applications that are using the adapter you are removing, replacing, or moving • Unmounting file systems • Ensuring that all devices connected to the adapter are identified and stopped • Listing all slots that are currently in use or a slot that is occupied by a specific adapter • Identifying the adapter's slot location • Making parent and child devices unavailable • Making the adapter unavailable Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 61

steps on the client logical partition before unconfiguring the storage adapter. For instructions, see “Preparing the client logical partitions” on page 62. For example, the adapter might be in use because the physical volume was used to create a virtual target device, or it might be part of a volume group used to create a virtual target device. To unconfigure SCSI, SSA, and Fibre Channel storage adapters, complete the following steps: Procedure 1. Connect to the Virtual I/O Server command-line interface. 2. Enter oem_setup_env to close all applications that are using the adapter you are unconfiguring. 3. Type lsslot-c PCI to list all the hot plug slots in the system unit and display their characteristics. 4. Type lsdev -C to list the current state of all the devices in the system unit. 5. Type unmount to unmount previously mounted file systems, directories, or files using this adapter. 6. Type rmdev -l adapter -R to make the adapter unavailable. Attention: Do not use the -d flag with the rmdev command for hot plug operations because this action removes your configuration. Preparing the client logical partitions About this task If the virtual target devices of the client logical partitions are not available, the client logical partitions can fail or they might be unable to perform I/O operations for a particular application. If you use the HMC to manage the system, you might have redundant Virtual I/O Server logical partitions, which allow for Virtual I/O Server maintenance and avoid downtime for client logical partitions. If you are replacing an adapter on the Virtual I/O Server and your client logical partition is dependent on one or more of the physical volumes accessed by that adapter, then You can take action on the client before you unconfigure the adapter. The virtual target devices must be in the define state before the Virtual I/O Server adapter can be replaced. Do not remove the virtual devices permanently. Procedure To prepare the client logical partitions so that you can unconfigure an adapter, complete the following steps depending on your situation. Table 2. Situations and steps for preparing the client logical partitions Situation Steps You have redundant hardware on the Virtual I/O Server for the adapter. No action is required on the client logical partition. HMC-managed systems only: You have redundant Virtual I/O Server logical partitions that, in conjunction with virtual client adapters, provide multiple paths to the physical volume on the client logical partition. No action is required on the client logical partition. However, path errors might be logged on the client logical partition. 62 Power Systems: PCIe adapters for the 9040-MR9

Situation Steps HMC-managed systems only: You have redundant Virtual I/O Server logical partitions that, in conjunction with virtual client adapters, provide multiple physical volumes that are used to mirror a volume group. See the procedures for your client operating system. For example, for AIX, see Replacing a disk on the Virtual I/O Server in the Advanced POWER Virtualization Best Practices Redpaper. The procedure for Linux is similar to this procedure for AIX.For example, for AIX, see Replacing a disk on the Virtual I/O Server in the Advanced POWER Virtualization Best Practices Redpaper. The procedure for Linux is similar to this procedure for AIX. You do not have redundant Virtual I/O Server logical partitions. Shut down the client logical partition. For systems that are managed by the HMC, see Stopping a system (www.ibm.com/support/ knowledgecenter/POWER9/p9haj/ stopsyshmc.htm). Verifying that the hot-plug PCI tools are installed for Linux Find information about using the hot-plug PCI tools for a PCI adapter installed in a system with a Linux operating system. Procedure 1. Enter the following command to verify that the hot-plug PCI tools are installed: rpm -aq | grep powerpc-utils. If the command does not list any powerpc-utils packages, the PCI Hot Plug tools are not installed. 2. Enter the following command to ensure that the rpaphp driver is loaded: ls -l /sys/bus/pci/ slots/ The directory should contain data. • If the directory is empty, the driver is not loaded or the system does not contain hot-plug PCI slots. The following is an example of the information displayed by this command: drwxr-xr-x 15 root root 0 Feb 16 23:31 . drwxr-xr-x 5 root root 0 Feb 16 23:31 .. drwxr-xr-x 2 root root 0 Feb 16 23:31 0000:00:02.0 drwxr-xr-x 2 root root 0 Feb 16 23:31 0000:00:02.2 drwxr-xr-x 2 root root 0 Feb 16 23:31 0000:00:02.4 drwxr-xr-x 2 root root 0 Feb 16 23:31 0001:00:02.0 drwxr-xr-x 2 root root 0 Feb 16 23:31 0001:00:02.2 drwxr-xr-x 2 root root 0 Feb 16 23:31 0001:00:02.4 drwxr-xr-x 2 root root 0 Feb 16 23:31 0001:00:02.6 drwxr-xr-x 2 root root 0 Feb 16 23:31 0002:00:02.0 drwxr-xr-x 2 root root 0 Feb 16 23:31 0002:00:02.2 drwxr-xr-x 2 root root 0 Feb 16 23:31 0002:00:02.4 drwxr-xr-x 2 root root 0 Feb 16 23:31 0002:00:02.6 • If the directory does not exist, run the following command to mount the filesystem: mount -t sysfs sysfs /sys 3. Ensure the following tools are available in the /usr/sbin directory. • lsslot • drmgr -c pci 4. Return to the procedure that sent you here. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 63

Learn how to use operating system commands to find the amount of remaining life in an NVMe device. Using the operating system commands The AIX, IBM i, and Linux operating systems have commands that you can use to determine the amount of life that is remaining in an NVMe device. Running the AIX command to check the amount of remaining life in NVMe devices Find information about using the AIX operating system to find the amount of remaining life in an NVMe device. About this task To find the amount of remaining life in an NVMe device by using the AIX operating system, complete the steps in this procedure. Procedure 1. If the system has logical partitions, complete this procedure from the logical partition that owns the NVMe device 2. To use the AIX operating system to find the remaining life on an NVMe device, complete the following steps: a) From the AIX command line, type diag and press Enter. b) From the Function Selection menu, select Task Selection > NVMe general health information. c) Select the NVMe device that you want to check remaining life for and press Enter. d) View the Percentage of NVM subsystem life used field. Is the value in the Percentage of NVM subsystem life used field 100%? • Yes: Continue with the next step. • No: Continue with step “4” on page 64. 3. The NVMe device is nearing its end of life and must be replaced. The NVMe device will soon reach the limit for the number of write operations that are supported. Write operations to the NVMe device become slower over time, and at some point the NVMe device becomes a read-only device. When the operating system writes data to a read-only device, the write operations are rejected, and the operating system considers the device as if a failure occurred. To support normal write operations, the NVMe device must be replaced. Note: Failure of IBM NVMe devices is covered in the standard warranty and during the maintenance period only for devices that have not reached the maximum number of write cycles. Devices that reach this limit might fail to operate according to specifications and must be replaced. This replacement cost is not covered under the standard warranty or during the maintenance period. This ends the procedure. 4. Does the Critical Warning field show Available spare space has fallen below threshold? • Yes: Continue with the next step. • No: A service action is not required. This ends the procedure. 5. The NVMe device is nearing its end of life and must be replaced. The NVMe device will soon reach the limit for the number of write operations that are supported. Write operations to the NVMe device become slower over time, and at some point the NVMe device becomes a read-only device. When the operating system writes data to a read-only device, the write operations are rejected, and the operating system considers the device as if a failure occurred. To support normal write operations, schedule maintenance to replace the NVMe device at your earliest convenience. Note: Failure of IBM NVMe devices is covered in the standard warranty and during the maintenance period only for devices that have not reached the maximum number of write cycles. Devices that reach 64 Power Systems: PCIe adapters for the 9040-MR9

is not covered under the standard warranty or during the maintenance period. This ends the procedure. Running the Linux smart-log command to check the amount of remaining life in NVMe devices Find information about using the Linux operating system smart-log command to find the amount of remaining life in an NVMe device. About this task To find the amount of remaining life in an NVMe device by using the Linux operating system smart-log command, complete the steps in this procedure. Procedure 1. If the system has logical partitions, complete this procedure from the logical partition that owns the NVMe device 2. To use the Linux operating system to find the remaining life on an NVMe device, complete the following steps: a) From the Linux command line, type the following command and press Enter. nvme smart-log /dev/nvmeX -H where nvmeX is the resource name of the NVMe device. b) View the Percentage used field. Is the value in the Percentage used field 100%? • Yes: Continue with the next step. • No: Continue with step “4” on page 65. 3. The NVMe device is nearing its end of life and must be replaced. The NVMe device will soon reach the limit for the number of write operations that are supported. Write operations to the NVMe device become slower over time, and at some point the NVMe device becomes a read-only device. When the operating system writes data to a read-only device, the write operations are rejected, and the operating system considers the device as if a failure occurred. To support normal write operations, the NVMe device must be replaced. Note: Failure of IBM NVMe devices is covered in the standard warranty and during the maintenance period only for devices that have not reached the maximum number of write cycles. Devices that reach this limit might fail to operate according to specifications and must be replaced. This replacement cost is not covered under the standard warranty or during the maintenance period. This ends the procedure. 4. In the critical_warning portion of the command output, is the Available Spare field displayed? • Yes: Continue with the next step. • No: Continue with step “6” on page 65. 5. Is the Available Spare field equal to 1? • Yes: The NVMe device has low spares. Continue with step “7” on page 66. • No: A service action is not required. This ends the procedure. 6. Complete the following steps to determine if the NVMe device has low spares: a) View the critical_warning field. b) The critical_warning field is a hexadecimal number. Convert the hexadecimal number to a binary number. c) Is the rightmost digit of the binary number equal to 1? • Yes: The NVMe device has low spares. Continue with the next step. Installing, removing, and replacing PCIe adapters in the 9040-MR9 system 65

7. The NVMe device is nearing its end of life and must be replaced. The NVMe device will soon reach the limit for the number of write operations that are supported. Write operations to the NVMe device become slower over time, and at some point the NVMe device becomes a read-only device. When the operating system writes data to a read-only device, the write operations are rejected, and the operating system considers the device as if a failure occurred. To support normal write operations, schedule maintenance to replace the NVMe device at your earliest convenience. Note: Failure of IBM NVMe devices is covered in the standard warranty and during the maintenance period only for devices that have not reached the maximum number of write cycles. Devices that reach this limit might fail to operate according to specifications and must be replaced. This replacement cost is not covered under the standard warranty or during the maintenance period. This ends the procedure. 66 Power Systems: PCIe adapters for the 9040-MR9

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service. IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not grant you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing IBM Corporation North Castle Drive, MD-NC119 Armonk, NY 10504-1785 US INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some jurisdictions do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you. This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice. Any references in this information to non-IBM websites are provided for convenience only and do not in any manner serve as an endorsement of those websites. The materials at those websites are not part of the materials for this IBM product and use of those websites is at your own risk. IBM may use or distribute any of the information you provide in any way it believes appropriate without incurring any obligation to you. The performance data and client examples cited are presented for illustrative purposes only. Actual performance results may vary depending on specific configurations and operating conditions. Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products. Statements regarding IBM's future direction or intent are subject to change or withdrawal without notice, and represent goals and objectives only. All IBM prices shown are IBM's suggested retail prices, are current and are subject to change without notice. Dealer prices may vary. This information is for planning purposes only. The information herein is subject to change before the products described become available. This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to actual people or business enterprises is entirely coincidental. © Copyright IBM Corp. 2018, 2019 67

The drawings and specifications contained herein shall not be reproduced in whole or in part without the written permission of IBM. IBM has prepared this information for use with the specific machines indicated. IBM makes no representations that it is suitable for any other purpose. IBM's computer systems contain mechanisms designed to reduce the possibility of undetected data corruption or loss. This risk, however, cannot be eliminated. Users who experience unplanned outages, system failures, power fluctuations or outages, or component failures must verify the accuracy of operations performed and data saved or transmitted by the system at or near the time of the outage or failure. In addition, users must establish procedures to ensure that there is independent data verification before relying on such data in sensitive or critical operations. Users should periodically check IBM's support websites for updated information and fixes applicable to the system and related software. Homologation statement This product may not be certified in your country for connection by any means whatsoever to interfaces of public telecommunications networks. Further certification may be required by law prior to making any such connection. Contact an IBM representative or reseller for any questions. Accessibility features for IBM Power Systems servers Accessibility features assist users who have a disability, such as restricted mobility or limited vision, to use information technology content successfully. Overview The IBM Power Systems servers include the following major accessibility features: • Keyboard-only operation • Operations that use a screen reader The IBM Power Systems servers use the latest W3C Standard, WAI-ARIA 1.0 (www.w3.org/TR/wai-aria/), to ensure compliance with US Section 508 (www.access-board.gov/guidelines-and-standards/ communications-and-it/about-the-section-508-standards/section-508-standards) and Web Content Accessibility Guidelines (WCAG) 2.0 (www.w3.org/TR/WCAG20/). To take advantage of accessibility features, use the latest release of your screen reader and the latest web browser that is supported by the IBM Power Systems servers. The IBM Power Systems servers online product documentation in IBM Knowledge Center is enabled for accessibility. The accessibility features of IBM Knowledge Center are described in the Accessibility section of the IBM Knowledge Center help (www.ibm.com/support/knowledgecenter/doc/ kc_help.html#accessibility). Keyboard navigation This product uses standard navigation keys. Interface information The IBM Power Systems servers user interfaces do not have content that flashes 2 - 55 times per second. The IBM Power Systems servers web user interface relies on cascading style sheets to render content properly and to provide a usable experience. The application provides an equivalent way for low-vision users to use system display settings, including high-contrast mode. You can control font size by using the device or web browser settings. The IBM Power Systems servers web user interface includes WAI-ARIA navigational landmarks that you can use to quickly navigate to functional areas in the application. 68 Power Systems: PCIe adapters for the 9040-MR9

license agreement. IBM makes no representation about the accessibility features of these products. Contact the vendor for accessibility information about its products. Related accessibility information In addition to standard IBM help desk and support websites, IBM has a TTY telephone service for use by deaf or hard of hearing customers to access sales and support services: TTY service 800-IBM-3383 (800-426-3383) (within North America) For more information about the commitment that IBM has to accessibility, see IBM Accessibility (www.ibm.com/able). Privacy policy considerations IBM Software products, including software as a service solutions, (“Software Offerings”) may use cookies or other technologies to collect product usage information, to help improve the end user experience, to tailor interactions with the end user, or for other purposes. In many cases no personally identifiable information is collected by the Software Offerings. Some of our Software Offerings can help enable you to collect personally identifiable information. If this Software Offering uses cookies to collect personally identifiable information, specific information about this offering’s use of cookies is set forth below. This Software Offering does not use cookies or other technologies to collect personally identifiable information. If the configurations deployed for this Software Offering provide you as the customer the ability to collect personally identifiable information from end users via cookies and other technologies, you should seek your own legal advice about any laws applicable to such data collection, including any requirements for notice and consent. For more information about the use of various technologies, including cookies, for these purposes, see IBM’s Privacy Policy at http://www.ibm.com/privacy and IBM’s Online Privacy Statement at http:// www.ibm.com/privacy/details the section entitled “Cookies, Web Beacons and Other Technologies” and the “IBM Software Products and Software-as-a-Service Privacy Statement” at http://www.ibm.com/ software/info/product-privacy. Trademarks IBM, the IBM logo, and ibm.com® are trademarks or registered trademarks of International Business Machines Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available on the web at Copyright and trademark information. The registered trademark Linux is used pursuant to a sublicense from the Linux Foundation, the exclusive licensee of Linus Torvalds, owner of the mark on a worldwide basis. Electronic emission notices Class A Notices The following Class A statements apply to the IBM servers that contain the POWER9 processor and its features unless designated as electromagnetic compatibility (EMC) Class B in the feature information. Notices 69

interference suppression devices supplied with the monitor. Canada Notice CAN ICES-3 (A)/NMB-3(A) European Community and Morocco Notice This product is in conformity with the protection requirements of Directive 2014/30/EU of the European Parliament and of the Council on the harmonization of the laws of the Member States relating to electromagnetic compatibility. IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a non-recommended modification of the product, including the fitting of non- IBM option cards. This product may cause interference if used in residential areas. Such use must be avoided unless the user takes special measures to reduce electromagnetic emissions to prevent interference to the reception of radio and television broadcasts. Warning: This equipment is compliant with Class A of CISPR 32. In a residential environment this equipment may cause radio interference. Germany Notice Deutschsprachiger EU Hinweis: Hinweis für Geräte der Klasse A EU-Richtlinie zur Elektromagnetischen Verträglichkeit Dieses Produkt entspricht den Schutzanforderungen der EU-Richtlinie 2014/30/EU zur Angleichung der Rechtsvorschriften über die elektromagnetische Verträglichkeit in den EU-Mitgliedsstaatenund hält die Grenzwerte der EN 55022 / EN 55032 Klasse A ein. Um dieses sicherzustellen, sind die Geräte wie in den Handbüchern beschrieben zu installieren und zu betreiben. Des Weiteren dürfen auch nur von der IBM empfohlene Kabel angeschlossen werden. IBM übernimmt keine Verantwortung für die Einhaltung der Schutzanforderungen, wenn das Produkt ohne Zustimmung von IBM verändert bzw. wenn Erweiterungskomponenten von Fremdherstellern ohne Empfehlung von IBM gesteckt/eingebaut werden. EN 55032 Klasse A Geräte müssen mit folgendem Warnhinweis versehen werden: "Warnung: Dieses ist eine Einrichtung der Klasse A. Diese Einrichtung kann im Wohnbereich Funk- Störungen verursachen; in diesem Fall kann vom Betreiber verlangt werden, angemessene Maßnahmen zu ergreifen und dafür aufzukommen." Deutschland: Einhaltung des Gesetzes über die elektromagnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem “Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) “. Dies ist die Umsetzung der EU-Richtlinie 2014/30/EU in der Bundesrepublik Deutschland. Zulassungsbescheinigung laut dem Deutschen Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) (bzw. der EMC Richtlinie 2014/30/EU) für Geräte der Klasse A Dieses Gerät ist berechtigt, in Übereinstimmung mit dem Deutschen EMVG das EG-Konformitätszeichen - CE - zu führen. Verantwortlich für die Einhaltung der EMV Vorschriften ist der Hersteller: International Business Machines Corp. New Orchard Road Armonk, New York 10504 Tel: 914-499-1900 Der verantwortliche Ansprechpartner des Herstellers in der EU ist: IBM Deutschland GmbH Technical Relations Europe, Abteilung M456 IBM-Allee 1, 71139 Ehningen, Germany 70 Power Systems: PCIe adapters for the 9040-MR9

email: [email protected] Generelle Informationen: Das Gerät erfüllt die Schutzanforderungen nach EN 55024 und EN 55022 / EN 55032 Klasse A. Japan Electronics and Information Technology Industries Association (JEITA) Notice This statement applies to products less than or equal to 20 A per phase. This statement applies to products greater than 20 A, single phase. This statement applies to products greater than 20 A per phase, three-phase. Japan Voluntary Control Council for Interference (VCCI) Notice Korea Notice Notices 71

Russia Notice Taiwan Notice IBM Taiwan Contact Information: United States Federal Communications Commission (FCC) Notice This equipment has been tested and found to comply with the limits for a Class A digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful interference when the equipment is operated in a commercial environment. This equipment generates, uses, and can radiate radio frequency energy and, if not installed and used in accordance with the instruction manual, may cause harmful interference to radio communications. Operation of this equipment in a residential area is likely to cause harmful interference, in which case the user will be required to correct the interference at his own expense. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. Proper cables and connectors are available from IBM-authorized dealers. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors or by unauthorized changes or modifications to this equipment. Unauthorized changes or modifications could void the user's authority to operate the equipment. This device complies with Part 15 of the FCC rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation. Responsible Party: International Business Machines Corporation New Orchard Road Armonk, NY 10504 Contact for FCC compliance information only: [email protected] 72 Power Systems: PCIe adapters for the 9040-MR9

The following Class B statements apply to features designated as electromagnetic compatibility (EMC) Class B in the feature installation information. When attaching a monitor to the equipment, you must use the designated monitor cable and any interference suppression devices supplied with the monitor. Canada Notice CAN ICES-3 (B)/NMB-3(B) European Community and Morocco Notice This product is in conformity with the protection requirements of Directive 2014/30/EU of the European Parliament and of the Council on the harmonization of the laws of the Member States relating to electromagnetic compatibility. IBM cannot accept responsibility for any failure to satisfy the protection requirements resulting from a non-recommended modification of the product, including the fitting of non- IBM option cards. German Notice Deutschsprachiger EU Hinweis: Hinweis für Geräte der Klasse B EU-Richtlinie zur Elektromagnetischen Verträglichkeit Dieses Produkt entspricht den Schutzanforderungen der EU-Richtlinie 2014/30/EU zur Angleichung der Rechtsvorschriften über die elektromagnetische Verträglichkeit in den EU-Mitgliedsstaatenund hält die Grenzwerte der EN 55022/ EN 55032 Klasse B ein. Um dieses sicherzustellen, sind die Geräte wie in den Handbüchern beschrieben zu installieren und zu betreiben. Des Weiteren dürfen auch nur von der IBM empfohlene Kabel angeschlossen werden. IBM übernimmt keine Verantwortung für die Einhaltung der Schutzanforderungen, wenn das Produkt ohne Zustimmung von IBM verändert bzw. wenn Erweiterungskomponenten von Fremdherstellern ohne Empfehlung von IBM gesteckt/eingebaut werden. Deutschland: Einhaltung des Gesetzes über die elektromagnetische Verträglichkeit von Geräten Dieses Produkt entspricht dem “Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) “. Dies ist die Umsetzung der EU-Richtlinie 2014/30/EU in der Bundesrepublik Deutschland. Zulassungsbescheinigung laut dem Deutschen Gesetz über die elektromagnetische Verträglichkeit von Geräten (EMVG) (bzw. der EMC Richtlinie 2014/30/EU) für Geräte der Klasse B Dieses Gerät ist berechtigt, in Übereinstimmung mit dem Deutschen EMVG das EG-Konformitätszeichen - CE - zu führen. Verantwortlich für die Einhaltung der EMV Vorschriften ist der Hersteller: International Business Machines Corp. New Orchard Road Armonk, New York 10504 Tel: 914-499-1900 Der verantwortliche Ansprechpartner des Herstellers in der EU ist: IBM Deutschland GmbH Technical Relations Europe, Abteilung M456 IBM-Allee 1, 71139 Ehningen, Germany Tel: +49 (0) 800 225 5426 email: [email protected] Generelle Informationen: Das Gerät erfüllt die Schutzanforderungen nach EN 55024 und EN 55032 Klasse B Notices 73

This statement applies to products less than or equal to 20 A per phase. This statement applies to products greater than 20 A, single phase. This statement applies to products greater than 20 A per phase, three-phase. Japan Voluntary Control Council for Interference (VCCI) Notice Taiwan Notice United States Federal Communications Commission (FCC) Notice This equipment has been tested and found to comply with the limits for a Class B digital device, pursuant to Part 15 of the FCC Rules. These limits are designed to provide reasonable protection against harmful 74 Power Systems: PCIe adapters for the 9040-MR9

energy and, if not installed and used in accordance with the instructions, may cause harmful interference to radio communications. However, there is no guarantee that interference will not occur in a particular installation. If this equipment does cause harmful interference to radio or television reception, which can be determined by turning the equipment off and on, the user is encouraged to try to correct the interference by one or more of the following measures: • Reorient or relocate the receiving antenna. • Increase the separation between the equipment and receiver. • Connect the equipment into an outlet on a circuit different from that to which the receiver is connected. • Consult an IBM-authorized dealer or service representative for help. Properly shielded and grounded cables and connectors must be used in order to meet FCC emission limits. Proper cables and connectors are available from IBM-authorized dealers. IBM is not responsible for any radio or television interference caused by using other than recommended cables and connectors or by unauthorized changes or modifications to this equipment. Unauthorized changes or modifications could void the user's authority to operate the equipment. This device complies with Part 15 of the FCC rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation. Responsible Party: International Business Machines Corporation New Orchard Road Armonk, New York 10504 Contact for FCC compliance information only: [email protected] Terms and conditions Permissions for the use of these publications are granted subject to the following terms and conditions. Applicability: These terms and conditions are in addition to any terms of use for the IBM website. Personal Use: You may reproduce these publications for your personal, noncommercial use provided that all proprietary notices are preserved. You may not distribute, display or make derivative works of these publications, or any portion thereof, without the express consent of IBM. Commercial Use: You may reproduce, distribute and display these publications solely within your enterprise provided that all proprietary notices are preserved. You may not make derivative works of these publications, or reproduce, distribute or display these publications or any portion thereof outside your enterprise, without the express consent of IBM. Rights: Except as expressly granted in this permission, no other permissions, licenses or rights are granted, either express or implied, to the publications or any information, data, software or other intellectual property contained therein. IBM reserves the right to withdraw the permissions granted herein whenever, in its discretion, the use of the publications is detrimental to its interest or, as determined by IBM, the above instructions are not being properly followed. You may not download, export or re-export this information except in full compliance with all applicable laws and regulations, including all United States export laws and regulations. IBM MAKES NO GUARANTEE ABOUT THE CONTENT OF THESE PUBLICATIONS. THE PUBLICATIONS ARE PROVIDED "AS-IS" AND WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO IMPLIED WARRANTIES OF MERCHANTABILITY, NON- INFRINGEMENT, AND FITNESS FOR A PARTICULAR PURPOSE. Notices 75