Avaya Pod Fx Troubleshooting Manual

All Rights Reserved. Notice While reasonable efforts have been made to ensure that the information in this document is complete and accurate at the time of printing, Avaya assumes no liability for any errors. Avaya reserves the right to make changes and corrections to the information in this document without the obligation to notify any person or organization of such changes. Documentation disclaimer “Documentation” means information published in varying mediums which may include product information, operating instructions and performance specifications that are generally made available to users of products. Documentation does not include marketing materials. Avaya shall not be responsible for any modifications, additions, or deletions to the original published version of Documentation unless such modifications, additions, or deletions were performed by or on the express behalf of Avaya. End User agrees to indemnify and hold harmless Avaya, Avaya's agents, servants and employees against all claims, lawsuits, demands and judgments arising out of, or in connection with, subsequent modifications, additions or deletions to this documentation, to the extent made by End User. Link disclaimer Avaya is not responsible for the contents or reliability of any linked websites referenced within this site or Documentation provided by Avaya. Avaya is not responsible for the accuracy of any information, statement or content provided on these sites and does not necessarily endorse the products, services, or information described or offered within them. Avaya does not guarantee that these links will work all the time and has no control over the availability of the linked pages. Warranty Avaya provides a limited warranty on Avaya hardware and software. Refer to your sales agreement to establish the terms of the limited warranty. In addition, Avaya’s standard warranty language, as well as information regarding support for this product while under warranty is available to Avaya customers and other parties through the Avaya Support website: https://support.avaya.com/helpcenter/ getGenericDetails?detailId=C20091120112456651010 under the link “Warranty & Product Lifecycle” or such successor site as designated by Avaya. Please note that if You acquired the product(s) from an authorized Avaya Channel Partner outside of the United States and Canada, the warranty is provided to You by said Avaya Channel Partner and not by Avaya. “Hosted Service” means an Avaya hosted service subscription that You acquire from either Avaya or an authorized Avaya Channel Partner (as applicable) and which is described further in Hosted SAS or other service description documentation regarding the applicable hosted service. If You purchase a Hosted Service subscription, the foregoing limited warranty may not apply but You may be entitled to support services in connection with the Hosted Service as described further in your service description documents for the applicable Hosted Service. Contact Avaya or Avaya Channel Partner (as applicable) for more information. Hosted Service THE FOLLOWING APPLIES ONLY IF YOU PURCHASE AN AVAYA HOSTED SERVICE SUBSCRIPTION FROM AVAYA OR AN AVAYA CHANNEL PARTNER (AS APPLICABLE), THE TERMS OF USE FOR HOSTED SERVICES ARE AVAILABLE ON THE AVAYA WEBSITE, HTTPS://SUPPORT.AVAYA.COM/LICENSEINFO UNDER THE LINK “Avaya Terms of Use for Hosted Services” OR SUCH SUCCESSOR SITE AS DESIGNATED BY AVAYA, AND ARE APPLICABLE TO ANYONE WHO ACCESSES OR USES THE HOSTED SERVICE. BY ACCESSING OR USING THE HOSTED SERVICE, OR AUTHORIZING OTHERS TO DO SO, YOU, ON BEHALF OF YOURSELF AND THE ENTITY FOR WHOM YOU ARE DOING SO (HEREINAFTER REFERRED TO INTERCHANGEABLY AS “YOU” AND “END USER”), AGREE TO THE TERMS OF USE. IF YOU ARE ACCEPTING THE TERMS OF USE ON BEHALF A COMPANY OR OTHER LEGAL ENTITY, YOU REPRESENT THAT YOU HAVE THE AUTHORITY TO BIND SUCH ENTITY TO THESE TERMS OF USE. IF YOU DO NOT HAVE SUCH AUTHORITY, OR IF YOU DO NOT WISH TO ACCEPT THESE TERMS OF USE, YOU MUST NOT ACCESS OR USE THE HOSTED SERVICE OR AUTHORIZE ANYONE TO ACCESS OR USE THE HOSTED SERVICE. Licenses THE SOFTWARE LICENSE TERMS AVAILABLE ON THE AVAYA WEBSITE, HTTPS://SUPPORT.AVAYA.COM/LICENSEINFO, UNDER THE LINK “AVAYA SOFTWARE LICENSE TERMS (Avaya Products)” OR SUCH SUCCESSOR SITE AS DESIGNATED BY AVAYA, ARE APPLICABLE TO ANYONE WHO DOWNLOADS, USES AND/OR INSTALLS AVAYA SOFTWARE, PURCHASED FROM AVAYA INC., ANY AVAYA AFFILIATE, OR AN AVAYA CHANNEL PARTNER (AS APPLICABLE) UNDER A COMMERCIAL AGREEMENT WITH AVAYA OR AN AVAYA CHANNEL PARTNER. UNLESS OTHERWISE AGREED TO BY AVAYA IN WRITING, AVAYA DOES NOT EXTEND THIS LICENSE IF THE SOFTWARE WAS OBTAINED FROM ANYONE OTHER THAN AVAYA, AN AVAYA AFFILIATE OR AN AVAYA CHANNEL PARTNER; AVAYA RESERVES THE RIGHT TO TAKE LEGAL ACTION AGAINST YOU AND ANYONE ELSE USING OR SELLING THE SOFTWARE WITHOUT A LICENSE. BY INSTALLING, DOWNLOADING OR USING THE SOFTWARE, OR AUTHORIZING OTHERS TO DO SO, YOU, ON BEHALF OF YOURSELF AND THE ENTITY FOR WHOM YOU ARE INSTALLING, DOWNLOADING OR USING THE SOFTWARE (HEREINAFTER REFERRED TO INTERCHANGEABLY AS “YOU” AND “END USER”), AGREE TO THESE TERMS AND CONDITIONS AND CREATE A BINDING CONTRACT BETWEEN YOU AND AVAYA INC. OR THE APPLICABLE AVAYA AFFILIATE (“AVAYA”). Avaya grants You a license within the scope of the license types described below, with the exception of Heritage Nortel Software, for which the scope of the license is detailed below. Where the order documentation does not expressly identify a license type, the applicable license will be a Designated System License. The applicable number of licenses and units of capacity for which the license is granted will be one (1), unless a different number of licenses or units of capacity is specified in the documentation or other materials available to You. “Software” means computer programs in object code, provided by Avaya or an Avaya Channel Partner, whether as stand-alone products, pre-installed on hardware products, and any upgrades, updates, patches, bug fixes, or modified versions thereto. “Designated Processor” means a single stand-alone computing device. “Server” means a Designated Processor that hosts a software application to be accessed by multiple users. “Instance” means a single copy of the Software executing at a particular time: (i) on one physical machine; or (ii) on one deployed software virtual machine (“VM”) or similar deployment. Licence types Designated System(s) License (DS). End User may install and use each copy or an Instance of the Software only on a number of Designated Processors up to the number indicated in the order. Avaya may require the Designated Processor(s) to be identified in the order by type, serial number, feature key, Instance, location or other specific designation, or to be provided by End User to Avaya through electronic means established by Avaya specifically for this purpose. Concurrent User License (CU). End User may install and use the Software on multiple Designated Processors or one or more Servers, so long as only the licensed number of Units are accessing and using the Software at any given time. A “Unit” means the unit on which Avaya, at its sole discretion, bases the pricing of its licenses and can be, without limitation, an agent, port or user, an e-mail or voice mail account in the name of a person or corporate function (e.g., webmaster or helpdesk), or a directory entry in the administrative database utilized by the Software that permits one user to interface with the Software. Units may be linked to a specific, identified Server or an Instance of the Software. Database License (DL). End User may install and use each copy or an Instance of the Software on one Server or on multiple Servers provided that each of the Servers on which the Software is installed communicates with no more than one Instance of the same database. CPU License (CP). End User may install and use each copy or Instance of the Software on a number of Servers up to the number

Server(s) does not exceed the performance capacity specified for the Software. End User may not re-install or operate the Software on Server(s) with a larger performance capacity without Avaya’s prior consent and payment of an upgrade fee. Named User License (NU). You may: (i) install and use each copy or Instance of the Software on a single Designated Processor or Server per authorized Named User (defined below); or (ii) install and use each copy or Instance of the Software on a Server so long as only authorized Named Users access and use the Software. “Named User”, means a user or device that has been expressly authorized by Avaya to access and use the Software. At Avaya’s sole discretion, a “Named User” may be, without limitation, designated by name, corporate function (e.g., webmaster or helpdesk), an e-mail or voice mail account in the name of a person or corporate function, or a directory entry in the administrative database utilized by the Software that permits one user to interface with the Software. Shrinkwrap License (SR). You may install and use the Software in accordance with the terms and conditions of the applicable license agreements, such as “shrinkwrap” or “clickthrough” license accompanying or applicable to the Software (“Shrinkwrap License”). Heritage Nortel Software “Heritage Nortel Software” means the software that was acquired by Avaya as part of its purchase of the Nortel Enterprise Solutions Business in December 2009. The Heritage Nortel Software is the software contained within the list of Heritage Nortel Products located at https://support.avaya.com/LicenseInfo under the link “Heritage Nortel Products” or such successor site as designated by Avaya. For Heritage Nortel Software, Avaya grants Customer a license to use Heritage Nortel Software provided hereunder solely to the extent of the authorized activation or authorized usage level, solely for the purpose specified in the Documentation, and solely as embedded in, for execution on, or for communication with Avaya equipment. Charges for Heritage Nortel Software may be based on extent of activation or use authorized as specified in an order or invoice. Copyright Except where expressly stated otherwise, no use should be made of materials on this site, the Documentation, Software, Hosted Service, or hardware provided by Avaya. All content on this site, the documentation, Hosted Service, and the product provided by Avaya including the selection, arrangement and design of the content is owned either by Avaya or its licensors and is protected by copyright and other intellectual property laws including the sui generis rights relating to the protection of databases. You may not modify, copy, reproduce, republish, upload, post, transmit or distribute in any way any content, in whole or in part, including any code and software unless expressly authorized by Avaya. Unauthorized reproduction, transmission, dissemination, storage, and or use without the express written consent of Avaya can be a criminal, as well as a civil offense under the applicable law. Virtualization The following applies if the product is deployed on a virtual machine. Each product has its own ordering code and license types. Note that each Instance of a product must be separately licensed and ordered. For example, if the end user customer or Avaya Channel Partner would like to install two Instances of the same type of products, then two products of that type must be ordered. Third Party Components “Third Party Components” mean certain software programs or portions thereof included in the Software or Hosted Service may contain software (including open source software) distributed under third party agreements (“Third Party Components”), which contain terms regarding the rights to use certain portions of the Software (“Third Party Terms”). As required, information regarding distributed Linux OS source code (for those products that have distributed Linux OS source code) and identifying the copyright holders of the Third Party Components and the Third Party Terms that apply is available in the products, Documentation or on Avaya’s website at: https:// support.avaya.com/Copyright or such successor site as designated by Avaya. The open source software license terms provided as Third Party Terms are consistent with the license rights granted in these Software License Terms, and may contain additional rights benefiting You, such as modification and distribution of the open source software. The Third Party Terms shall take precedence over these Software License Terms, solely with respect to the applicable Third Party Components to the extent that these Software License Terms impose greater restrictions on You than the applicable Third Party Terms. The following applies only if the H.264 (AVC) codec is distributed with the product. THIS PRODUCT IS LICENSED UNDER THE AVC PATENT PORTFOLIO LICENSE FOR THE PERSONAL USE OF A CONSUMER OR OTHER USES IN WHICH IT DOES NOT RECEIVE REMUNERATION TO (i) ENCODE VIDEO IN COMPLIANCE WITH THE AVC STANDARD (“AVC VIDEO”) AND/OR (ii) DECODE AVC VIDEO THAT WAS ENCODED BY A CONSUMER ENGAGED IN A PERSONAL ACTIVITY AND/OR WAS OBTAINED FROM A VIDEO PROVIDER LICENSED TO PROVIDE AVC VIDEO. NO LICENSE IS GRANTED OR SHALL BE IMPLIED FOR ANY OTHER USE. ADDITIONAL INFORMATION MAY BE OBTAINED FROM MPEG LA, L.L.C. SEE HTTP://WWW.MPEGLA.COM. Service Provider THE FOLLOWING APPLIES TO AVAYA CHANNEL PARTNER’S HOSTING OF AVAYA PRODUCTS OR SERVICES. THE PRODUCT OR HOSTED SERVICE MAY USE THIRD PARTY COMPONENTS SUBJECT TO THIRD PARTY TERMS AND REQUIRE A SERVICE PROVIDER TO BE INDEPENDENTLY LICENSED DIRECTLY FROM THE THIRD PARTY SUPPLIER. AN AVAYA CHANNEL PARTNER’S HOSTING OF AVAYA PRODUCTS MUST BE AUTHORIZED IN WRITING BY AVAYA AND IF THOSE HOSTED PRODUCTS USE OR EMBED CERTAIN THIRD PARTY SOFTWARE, INCLUDING BUT NOT LIMITED TO MICROSOFT SOFTWARE OR CODECS, THE AVAYA CHANNEL PARTNER IS REQUIRED TO INDEPENDENTLY OBTAIN ANY APPLICABLE LICENSE AGREEMENTS, AT THE AVAYA CHANNEL PARTNER’S EXPENSE, DIRECTLY FROM THE APPLICABLE THIRD PARTY SUPPLIER. WITH RESPECT TO CODECS, IF THE AVAYA CHANNEL PARTNER IS HOSTING ANY PRODUCTS THAT USE OR EMBED THE G.729 CODEC, H.264 CODEC, OR H.265 CODEC, THE AVAYA CHANNEL PARTNER ACKNOWLEDGES AND AGREES THE AVAYA CHANNEL PARTNER IS RESPONSIBLE FOR ANY AND ALL RELATED FEES AND/OR ROYALTIES. THE G.729 CODEC IS LICENSED BY SIPRO LAB TELECOM INC. SEE WWW.SIPRO.COM/CONTACT.HTML. THE H.264 (AVC) CODEC IS LICENSED UNDER THE AVC PATENT PORTFOLIO LICENSE FOR THE PERSONAL USE OF A CONSUMER OR OTHER USES IN WHICH IT DOES NOT RECEIVE REMUNERATION TO: (I) ENCODE VIDEO IN COMPLIANCE WITH THE AVC STANDARD (“AVC VIDEO”) AND/OR (II) DECODE AVC VIDEO THAT WAS ENCODED BY A CONSUMER ENGAGED IN A PERSONAL ACTIVITY AND/OR WAS OBTAINED FROM A VIDEO PROVIDER LICENSED TO PROVIDE AVC VIDEO. NO LICENSE IS GRANTED OR SHALL BE IMPLIED FOR ANY OTHER USE. ADDITIONAL INFORMATION FOR H.264 (AVC) AND H.265 (HEVC) CODECS MAY BE OBTAINED FROM MPEG LA, L.L.C. SEE HTTP:// WWW.MPEGLA.COM. Compliance with Laws You acknowledge and agree that it is Your responsibility for complying with any applicable laws and regulations, including, but not limited to laws and regulations related to call recording, data privacy, intellectual property, trade secret, fraud, and music performance rights, in the country or territory where the Avaya product is used. Preventing Toll Fraud “Toll Fraud” is the unauthorized use of your telecommunications system by an unauthorized party (for example, a person who is not a corporate employee, agent, subcontractor, or is not working on your company's behalf). Be aware that there can be a risk of Toll Fraud associated with your system and that, if Toll Fraud occurs, it can result in substantial additional charges for your telecommunications services. Avaya Toll Fraud intervention If You suspect that You are being victimized by Toll Fraud and You need technical assistance or support, call Technical Service Center Toll Fraud Intervention Hotline at +1-800-643-2353 for the United States and Canada. For additional support telephone numbers, see the Avaya Support website: https://support.avaya.com or such successor site as designated by Avaya.

the Security Policies and Support section of https:// support.avaya.com/security. Suspected Avaya product security vulnerabilities are handled per the Avaya Product Security Support Flow (https:// support.avaya.com/css/P8/documents/100161515). Downloading Documentation For the most current versions of Documentation, see the Avaya Support website: https://support.avaya.com, or such successor site as designated by Avaya. Contact Avaya Support See the Avaya Support website: https://support.avaya.com for product or Hosted Service notices and articles, or to report a problem with your Avaya product or Hosted Service. For a list of support telephone numbers and contact addresses, go to the Avaya Support website: https://support.avaya.com (or such successor site as designated by Avaya), scroll to the bottom of the page, and select Contact Avaya Support. Trademarks Avaya Aura® is a registered trademark of Avaya Inc. The trademarks, logos and service marks (“Marks”) displayed in this site, the Documentation, Hosted Service(s), and product(s) provided by Avaya are the registered or unregistered Marks of Avaya, its affiliates, its licensors, its suppliers, or other third parties. Users are not permitted to use such Marks without prior written consent from Avaya or such third party which may own the Mark. Nothing contained in this site, the Documentation, Hosted Service(s) and product(s) should be construed as granting, by implication, estoppel, or otherwise, any license or right in and to the Marks without the express written permission of Avaya or the applicable third party. Avaya is a registered trademark of Avaya Inc. All non-Avaya trademarks are the property of their respective owners. Linux® is the registered trademark of Linus Torvalds in the U.S. and other countries.

Purpose.................................................................................................................................. 7 Product registration................................................................................................................. 7 Warranty................................................................................................................................. 7 Chapter 2: New in this document............................................................................................ 9 VMware vSphere 6.0 Update 3B............................................................................................... 9 Avaya Pod Orchestration Suite Release 3.1.............................................................................. 9 Avaya Pod Fx upgrades to Release 3.1................................................................................... 10 HPE Nimble CS1000 Array..................................................................................................... 11 Chapter 3: Configuring monitoring and alerts..................................................................... 12 Configuring Nimble CS1000 monitoring................................................................................... 12 Nimble CS1000 and VPFM alarms.......................................................................................... 13 Configuring EMC VNXe3200 monitoring and EMC Secure Remote Support............................... 14 Configuring EMC VNX5300 monitoring and EMC Secure Remote Support................................ 15 Configuring alarm notifications................................................................................................ 17 Configuring VPFM to monitor Avaya Pod Fx elements........................................................ 17 Creating an email action.................................................................................................. 18 Creating a response configuration.................................................................................... 19 Configuring SNMP traps for the SAL Gateway................................................................... 21 Chapter 4: Troubleshooting general issues......................................................................... 24 General troubleshooting......................................................................................................... 24 Quick Fixes........................................................................................................................... 28 Troubleshooting with PVM...................................................................................................... 31 Chapter 5: Troubleshooting software issues....................................................................... 34 Troubleshooting Avaya Aura ® Communication Manager in an Avaya Aura ® Virtualized Environment......................................................................................................................... 34 ESXi host alarm after VMware 6.0 Update 3A upgrade............................................................. 36 vCenter Server inaccessible after server failure or unexpected server reboot............................. 37 Reconfiguring a cluster.......................................................................................................... 37 Timeout error configuring High Availability in vCenter .............................................................. 39 Backing up and restoring vCenter configuration....................................................................... 41 Recovering from a failed VCSA 5.5 upgrade............................................................................ 42 Avaya Aura ® VMs do not recover............................................................................................ 43 Chapter 6: Troubleshooting hardware issues...................................................................... 45 Lenovo compute server power supply alarm troubleshooting.................................................... 45 Upgrading HP ProLiant Gen8 server BIOS for POST errors...................................................... 50 Upgrading Lenovo server BIOS for fan alarm........................................................................... 52 Checklist for replacing a Power Distribution Unit...................................................................... 53 Configuring switched PDUs over Ethernet......................................................................... 55 October 2017 Troubleshooting the Avaya Pod Fx™ 5 Comments on this document? [email protected]

Chapter 7: Resources............................................................................................................. 60 Support................................................................................................................................ 60 Documentation reference....................................................................................................... 62 Training................................................................................................................................ 63 Avaya Mentor videos............................................................................................................. 63 Searching a documentation collection..................................................................................... 64 Subscribing to e-notifications.................................................................................................. 65 Contents October 2017 Troubleshooting the Avaya Pod Fx™ 6 Comments on this document? [email protected]

Purpose This document contains advice, solutions, and procedures for dealing with common issues encountered while using the Avaya Pod Fx. This document is intended for the following audience: • end users • Avaya certified technicians • Avaya partners • Avaya Sales Engineer (SE) • Avaya System Integrator (SI) Product registration To prevent service interruption, you must register your Avaya Pod Fx products. Following are the available methods for product registration: • Implementation as a service. If Avaya Professional Services provided implementation services on site, Avaya Professional Services also performs product registration on your behalf. • Avaya Partner and Customer implementation. For information about the step-by-step registration process, see the Avaya Classic Global Registration Process Help Document on the Avaya Product Support Registration page. The document ID is 100162279. Product registration is a required element for effective Avaya customer support. You must follow the Partner and Customer Guidance in the Avaya Global Registration Process to help ensure the seamless support you have come to expect from Avaya. Warranty Avaya provides a limited warranty on its Hardware and Software (“Product(s)”). Refer to your sales agreement to establish the terms of the limited warranty. October 2017 Troubleshooting the Avaya Pod Fx™ 7 Comments on this document? [email protected]

while under warranty, is available to Avaya customers and other parties through the Avaya Support website: http://support.avaya.com. Note: If you acquired the Product(s) from an authorized Avaya reseller outside of the United States and Canada, the warranty is provided to you by said Avaya reseller and not by Avaya. Example 1. Go to the Avaya Support website at http://support.avaya.com. 2. In the Global search field, type warranty, to search the Avaya Knowledge Base for warranty topics. The system displays a list of all warranty topics. 3. Click the relevant warranty topic. Introduction October 2017 Troubleshooting the Avaya Pod Fx™ 8 Comments on this document? [email protected]

The following sections outline what is new in Troubleshooting the Avaya Pod Fx (NN47204–700). VMware vSphere 6.0 Update 3B Avaya Pod Fx Release 3.1 uses VMware vSphere 6.0 vCenter Server Appliance Update 3B for all Avaya Pod Fx platforms. Compute servers are the physical servers, operating as VMware ESXi hosts. Avaya Pod Fx component applications are virtualized and run on a VMware High Availability (HA) cluster of ESXi hosts. VMware vCenter Server Appliance (VCSA) provides management for the HA clusters, vMotion, and Dynamic Resource Schedule (DRS) rules for the ESXi hosts in the Avaya Pod Fx. Avaya Pod Orchestration Suite Release 3.1 Avaya Pod Orchestration Suite (POS) is a bundle of visualization and management applications required to configure, orchestrate, manage, and monitor your Avaya Pod Fx. POS Release 3.1 applications include the following enhancements: • Support for Avaya Aura® 7.1 applications. • Updates to overall platform security. New Avaya Pod Fx Release 3.1 system builds provide the following POS application versions pre- deployed. Existing systems require an upgrade to Avaya Pod Fx Release 3.1, and the following POS versions: • Pod Visualization Manager (PVM) Release 3.1. • Visualization Performance and Fault Monitoring (VPFM) Release 3.1. • Management Server Console (MSC) Release 3.1 Important: The following applications have been removed from the POS software starting in Release 3.1: • Configuration and Orchestration Manager (COM) October 2017 Troubleshooting the Avaya Pod Fx™ 9 Comments on this document? [email protected]

• IP Flow Manager (IPFM) • Pod Utility Manager (PUM) These applications are end of sale and end of support in Release 3.1. Avaya Pod Fx upgrades to Release 3.1 Preexisting Avaya Pod Fx platforms can upgrade to Release 3.1 software. The following software upgrade scenarios are supported: • One step upgrade path for all Avaya Pod Fx platforms from Release 2.0.1, 2.1.1, 3.0, 3.0.1, or 3.0.2 to Release 3.1. Important: The Avaya Pod Utility Module has been removed from the Management Server Console software in Release 3.1. All upgrades are now conducted using the procedures documented in Using the Management Server Console for Avaya Pod Fx (NN47204–303). Avaya Pod Fx upgrades must be performed by Avaya Professional Services, or an Avaya Pod Fx Select certified Business Partner. They must plan and prepare to perform the upgrades, such as downloading and transferring all the upgrade files required before starting any component upgrades. To perform an Avaya Pod Fx upgrade, update the following components to the Release 3.1 software baseline: • Avaya Aura® System Manager • Avaya Pod Orchestration Suite (POS) software • VMware vCenter and ESXi software • Avaya network switches software and firmware • EMC storage array software and firmware • HPE Nimble storage array software and firmware • HPE or Lenovo compute servers BIOS and firmware • ServerTech Power Distribution Unit (PDU) firmware • Remove Unisphere Remote and deploy Unisphere Central if applicable Warning: Avaya Pod Fx Release 3.1 requires an Avaya Aura 7.1 System Manager. Before performing any upgrades, verify if your existing solution applications are supported on Avaya Aura 7.1 using the product compatibility matrix available at https://secureservices.avaya.com/compatibility-matrix/ menus/product.xhtml. You can upgrade ACM, and ANAV to the xCaaS 2.0 supported versions before performing an upgrade of the Avaya Pod Fx infrastructure to Release 3.1. New in this document October 2017 Troubleshooting the Avaya Pod Fx™ 10 Comments on this document? [email protected]

Management Server Console for Avaya Pod Fx (NN47204–303) (Available for Avaya authorized professionals and Avaya Pod Fx Select certified Business Partners only). It is recommended that you perform the Avaya Pod Fx Configurator Design Review process in the event of an upgrade. This is recommended to validate the updated software line up for the applications does not require additional resources in the Avaya Pod Fx such as additional storage or servers. To schedule a design review, email [email protected]. HPE Nimble CS1000 Array Release 3.1 introduces the HPE Nimble CS1000 hybrid storage array as the storage solution for all new Avaya Pod Fx platforms. The HPE Nimble CS1000 storage array contains redundant disk controllers, redundant power, networking ports, and expansion shelves for the storage system. The CS1000 Array can accommodate up to 2 ES2–H expansion shelves in Release 3.1. HPE Nimble CS1000 Array October 2017 Troubleshooting the Avaya Pod Fx™ 11 Comments on this document? [email protected]

alerts This section contains information about the configuration of monitoring and alerts for the elements of the Avaya Pod Fx. Related links Configuring Nimble CS1000 monitoring on page 12 Nimble CS1000 and VPFM alarms on page 13 Configuring EMC VNXe3200 monitoring and EMC Secure Remote Support on page 14 Configuring EMC VNX5300 monitoring and EMC Secure Remote Support on page 15 Configuring alarm notifications on page 17 Configuring Nimble CS1000 monitoring About this task Use the following procedure to configure diagnostic monitoring on the Nimble CS1000. Procedure 1. Open a new browser window or tab. 2. Enter the management IP address of the storage array. 3. Log in to the management interface with the Administrator credentials. 4. Select Administration > Alerts and Monitoring > Diagnostics from the menu. 5. Select Allow Nimble Storage Support to collect analytics data automatically from the array. 6. Select Enable Secure Tunnel. 7. Click Test Diagnostics for Nimble Analytics Settings. A test is performed on the diagnostic settings. A checkmark should appear beside every item in the test window. If the test fails, ensure that your firewall is allowing traffic from the Management IP and Controller Support IP addresses. Run the test once again. Troubleshoot and run the test until all items in the test pass. 8. Click Send Diagnostics for Nimble Analytics. October 2017 Troubleshooting the Avaya Pod Fx™ 12 Comments on this document? [email protected]

configuration settings if an email is not received at the anticipated address. 9. Click Save. Related links Configuring monitoring and alerts on page 12 Nimble CS1000 and VPFM alarms About this task The Nimble CS1000 does not currently send alarms or notifications to VPFM. Configure email alerts to receive alarms and notifications from the device. Use the following procedure to configure email alerts. Procedure 1. Open a new browser window or tab. 2. Enter the management IP address of the storage array. 3. Log in to the management interface with the Administrator credentials. 4. Select Administration > Alerts and Monitoring > Email from the menu. 5. Enter a reporting email address in the Send From Address section. Note: This address is displayed in the From field of an email when an alarm or notification is sent from the array. This does not have to be a valid email address. However, the address should provide enough information to be clearly identifiable to the recipient. For example, a corporation has 2 Pods named ExampleCorpCallCenter and ExampleCorpSAN each with one storage array. The administrator could enter [email protected] and [email protected] when configuring each device to identify where an alarm or notification originates. 6. Click Add in the Send To Addresses section. 7. Enter an email address to send alarms and notifications. 8. Repeat the previous two steps for each address to receive alarms and notifications. 9. Select Send event data to Nimble Storage Support if you wish to also send this information directly to Nimble Support. 10. Enter the information for your corporate SMTP server SMTP Server section. 11. Set the level at which alarms and notifications are sent in the Alert Options section. 12. Click Test to test the settings. Nimble CS1000 and VPFM alarms October 2017 Troubleshooting the Avaya Pod Fx™ 13 Comments on this document? [email protected]

Related links Configuring monitoring and alerts on page 12 Configuring EMC VNXe3200 monitoring and EMC Secure Remote Support About this task Use the following procedure to configure EMC monitoring and EMC Secure Remote Support (ESRS) on the EMC VNXe3200. Before you begin • Work closely with the customer and local EMC support team to implement ConnectEMC and ESRS and to discuss the implementation and Internet security issues. The customer system must be registered in the EMC database by the EMC support team. • Work with the customer to ensure the port requirements described here are met: https:// support.emc.com/ DOCU55301_EMC_SECURE_REMOTE_SERVICES_RELEASE_3.08_PORT_REQUIREMEN TS.PDF?LANGUAGE=EN_US&LANGUAGE=EN_US. • Valid EMC support credentials. • Customer’s SMTP IP address (alias). Important: You must schedule EMC support to complete these tasks. Procedure 1. Access EMC Unisphere by opening the IP address of the device in a browser tab or from the PVM storage component view. 2. Enter the username and password in the provided fields. 3. Click Login. 4. On the dashboard, navigate to Settings > Alert Settings. 5. Select the checkboxes entitled Enable ConnectEMC and Include Customer Contact Information. 6. In the ConnectEMC SMTP Server field, enter the customer’s SMTP IP address. 7. In the Email Alerts section, add the email addresses of the individuals that will receive email notifications. 8. In the Severity level of alerts to send field, set the severity level that will trigger emails to be sent. Configuring monitoring and alerts October 2017 Troubleshooting the Avaya Pod Fx™ 14 Comments on this document? [email protected]

10. (Optional) On the dashboard, navigate to Settings > EMC Secure Remote Support. 11. (Optional) Click Add Credentials. 12. (Optional) Enter the EMC support credentials. 13. (Optional) Click Apply. 14. (Optional) Select the Enable ESRS check box. 15. (Optional) Configure a proxy server or policy manager, as required. Related links Configuring monitoring and alerts on page 12 Configuring EMC VNX5300 monitoring and EMC Secure Remote Support About this task Use the following procedure to configure EMC monitoring and EMC Secure Remote Support (ESRS) on the EMC VNX5300. Before you begin • Work closely with the customer and local EMC support team to implement ConnectEMC and ESRS and to discuss the implementation and Internet security issues. The customer system must be registered in the EMC database by the EMC support team. • Work with the customer to ensure the port requirements described here are met: HTTPS:// SUPPORT.EMC.COM/ DOCU55301_EMC_SECURE_REMOTE_SERVICES_RELEASE_3.08_PORT_REQUIREMEN TS.PDF?LANGUAGE=EN_US&LANGUAGE=EN_US. • Valid EMC support credentials. • Customer’s SMTP IP address (alias). • Work with EMC and the customer to decide which version of ESRS should be used: - ESRS in Unisphere for a single system. - ESRS VE for multiple EMC VNX systems. An HA option for ESRS VE is also available. Important: You must schedule EMC support to complete these tasks. Procedure 1. Access EMC Unisphere by opening the IP address of the device in a browser tab or from the PVM storage component view. 2. Log in with the required username and password credentials. Configuring EMC VNX5300 monitoring and EMC Secure Remote Support October 2017 Troubleshooting the Avaya Pod Fx™ 15 Comments on this document? [email protected]

4. Select the system serial number to open the dashboard. 5. On the dashboard, navigate to System > Service Tasks. 6. Click Manage Customer Contact to set the customer contact information. 7. Update the contact information as required. Mandatory fields are noted with an asterisk. 8. Click Update. 9. Click Manage ConnectEMC. 10. In the Email section, complete the fields as follows: a. Set the Email Priority field to Primary. b. Set the Email Server (SMTP) field to the customer SMTP server IP address. c. Set the Subject field to ConnectEMC. d. Set the Recipient Address(es) field to [email protected]. e. Set the Send Email Address field to @. f. Click OK. 11. Click Manage ESRS. 12. In the Support Credentials section, enter the customer EMC credentials. 13. Click OK. 14. Ensure the Credentials Type is set to EMC Online Support. 15. Click OK. 16. Click Yes when prompted to continue with the operation. 17. On the dashboard, navigate to System > Notifications. 18. Click the Notification Templates tab. 19. Press CTRL+SHIFT+F12 on the keyboard. 20. Enter the password messner when prompted. 21. Click OK. 22. Select the latest version of the Call_Home_Template from the provided list. 23. Select Explicit Events from the Events section. 24. Select the ConnectEMC tab. 25. Select Connect Home. 26. Click OK. 27. Select the Distributed Monitors tab. Configuring monitoring and alerts October 2017 Troubleshooting the Avaya Pod Fx™ 16 Comments on this document? [email protected]

29. Click Use Template. 30. Select the template you previously edited. 31. Click Send Test Event. 32. Verify with EMC support personnel the test event was received successfully. Related links Configuring monitoring and alerts on page 12 Configuring alarm notifications The following section contains procedures required to allow Avaya Pod Fx element events to trigger a response in VPFM. In this example, the response will be to send an email message to the Avaya Pod Fx operator(s) when IP availability is lost to an Avaya Pod Fx element. The Avaya Pod Fx operator can specify which element events will trigger an alert, and configure the trigger requirements. Related links Configuring monitoring and alerts on page 12 Configuring VPFM to monitor Avaya Pod Fx elements on page 17 Creating an email action on page 18 Configuring SNMP traps for the SAL Gateway on page 21 Configuring VPFM to monitor Avaya Pod Fx elements About this task Use this procedure to configure VPFM to monitor network elements on the domain. Before you begin Ensure that Avaya Pod Fx elements have been discovered by VPFM and that the VPFM Domain holding those elements is enabled for Monitoring. If new elements are being added to the Avaya Pod Fx solution, you must rediscover your network so they will be monitored. Procedure 1. Click Configurations > Monitoring. 2. Click Add a new monitoring configuration . 3. Enter the name Avaya CPod Managed Devices and click OK. 4. In These elements click Everything. Configuring alarm notifications October 2017 Troubleshooting the Avaya Pod Fx™ 17 Comments on this document? [email protected]

5. Click Show Private Scopes . 6. Type Avaya Pod Managed Devices in the search box, select the scope then click OK. 7. In Monitor for these information types select Monitored Information. All boxes should now display a green arrow. Note: By default the monitoring data will be retained for 30 days, and the polling interval will be once every ten minutes. You can adjust this as needed and the smallest polling interval is 1 minute. However, Avaya recommends leaving the polling interval at the default value. 8. Click Apply your changes . Related links Configuring alarm notifications on page 17 Creating an email action About this task Use this procedure to create an email action that will inform the Avaya Pod Fx operator of any alarms that are triggered during VPFM monitoring of the Avaya Pod Fx network elements. Before you begin Configure VPFM to monitor network elements. Procedure 1. Navigate to Configurations > Actions.. Highlight Sample Email Action and click Clone selected action. Enter the name Pod Fx IP Availability Email Action then click OK. 2. Enter an email alias address in the To: field. You can customize the alias in the From: field or leave it as default. You can enter multiple email addresses, each separated by a semi-colon. 3. Click the Primary SMTP tab, enter the SMTP hostname or IP address for your environment. If necessary for your environment, enter the optional information for SMTP username, SMTP password, and SMTP Port, if not default port 25. 4. Select Use SSL if SSL is used in your environment. Configuring monitoring and alerts October 2017 Troubleshooting the Avaya Pod Fx™ 18 Comments on this document? [email protected]

changes when finished. Next steps To test the email action, click Test action (green right-facing arrow). If the connection to your SMTP server is successful, the system displays a pop-up window stating Action executed successfully. The alias email account(s) will also receive a generic email message from the test. Related links Configuring alarm notifications on page 17 Creating a response configuration About this task Use this procedure to create a response configuration for all Avaya Pod Fx managed devices. Procedure 1. Navigate to Actions > By Event Response. Click Add a new response. Enter the name Pod Fx IP Availability Response. Click OK. 2. In These Elements click Everything. The system displays the Choose a Scope screen. 3. Click Show Private Scopes 4. Enter Avaya Pod Managed Devices in the search box, select the scope then click OK. 5. On the Domain Elements & Event Types tab, select Response applies to Event Types. 6. Expand Availability Event. Expand Availability Problem. Expand Availability Failure. Expand IP Availability Failure. Select IP Availability Failure. Clear Phone IP Disconnect. The tree should look like the following diagram: Configuring alarm notifications October 2017 Troubleshooting the Avaya Pod Fx™ 19 Comments on this document? [email protected]

event is posted. 8. In Execute the following actions, select Pod Fx IP Availability Email Action. 9. Click Apply your changes button. Example After you have completed all of the previous configuration steps, if an Avaya Pod Fx managed element is polled and does not respond, the operator(s) will receive an email similar to the following example: From: [email protected] [mailto:[email protected]] Sent: Thursday, April 21, 2016 2:16 PM To: Pod Fx Operators Subject: IPAvailabilityFailure on vnx-avstor-1 Domain: tpod_4200 Network Element: vnx-avstor-1 Address: 10.10.8.8 Event Type: IPAvailabilityFailure Time: Thu Apr 21 14:09:00 PDT 2016 Configuring monitoring and alerts October 2017 Troubleshooting the Avaya Pod Fx™ 20 Comments on this document? [email protected]

Error Description: unknown IP availability failure on vnx-avstor-1. Configuring SNMP traps for the SAL Gateway About this task Use this task to configure elements of the Avaya Pod Fx to send SNMP traps to the SAL Gateway. Procedure 1. Configure Avaya Virtual Services Platform 4000 Series switches to send SNMP traps. a. Start an SSH session with the switch. b. Log in as the admin user. c. Use the enable command to enter Privileged mode. d. Use the config t command to enter Configuration mode. e. Use the snmp-server host v2c readview command to add the SAL Gateway as a trap receiver. f. Use the save config command to save the configuration. g. End the SSH session. h. Logout. 2. Configure Avaya Virtual Services Platform 7200 Series switches to send SNMP traps. a. Start an SSH session with the switch. b. Log in as the admin user. c. Use the enable command to enter Privileged mode. d. Use the config t command to enter Configuration mode. e. Use the snmp-server host v2c readview command to add the SAL Gateway as a trap receiver. f. Use the save config command to save the configuration. g. End the SSH session. h. Logout. 3. Configure Avaya Virtual Services Platform 7000 Series switches to send SNMP traps. a. Start an SSH session with the switch. b. Log in as the admin user. c. Use the config t command to enter Configuration mode. d. Use the snmp-server host avaya123 command to add the SAL Gateway as a trap receiver. Configuring alarm notifications October 2017 Troubleshooting the Avaya Pod Fx™ 21 Comments on this document? [email protected]

f. End the SSH session. g. Logout. 4. Configure PDUs to send SNMP traps. a. Log in as the admin user to the HTTPS web interface. b. Select Configuration > SNMP/Thresholds from the menu. c. Locate the field Trap Destination 2. d. Enter the SAL Gateway IP address in the field. e. Click Apply. f. Logout. 5. Configure the vCenter Server Appliance to send SNMP traps. a. Log in to the VCSA web client located at: https://:9443/vsphere- client/. b. Select the applicable vCenter instance. c. Select Manage > Settings > General from the menu. d. Click Edit. e. Select SNMP receivers. f. Enter the SAL Gateway IP address in the next available receiver field. g. Verify that it is enabled. h. Click OK. i. Logout. 6. Configure the EMC VNXe3200 to send SNMP traps. a. Log in as the admin user to the HTTPS web interface. The web interface is located: https://EMC_MGMT_IP_ADDRESS. b. Select Settings > More configuration > Alert Settings from the menu. c. Click Add under SNMP alerts. d. Enter the IP address of the SAL Gateway. e. Enter the user name avaya123. f. Select None for the Authentication Protocol. g. Click OK. h. Logout. 7. Configure HP iLO to send SNMP traps. a. Log in to the web interface of the HP iLO. The web interface is located at: https:// HP_iLO_IP_ADDRESS. Configuring monitoring and alerts October 2017 Troubleshooting the Avaya Pod Fx™ 22 Comments on this document? [email protected]

c. Enter the SAL GW IP address as a new SNMP Alert Destination. d. Click Apply. e. Logout. 8. Configure Lenovo TMM to send SNMP traps. a. Log in to the HTTPS web interface. b. Select Server Information > Event Management > Platform Events from the menu. c. Select the Global Alerting Enable checkbox. d. Select Generate PET for each of the filter names. e. Click Apply Changes. f. Select Server Information > Event Management > Trap Settings from the menu. g. Enter and enable the next available destination. Note: A test trap can be sent to verify it is enabled. h. Click Apply Changes. i. Logout. Related links Configuring alarm notifications on page 17 Configuring alarm notifications October 2017 Troubleshooting the Avaya Pod Fx™ 23 Comments on this document? [email protected]

This section contains general information and tips for troubleshooting issues. Related links General troubleshooting on page 24 Quick Fixes on page 28 Troubleshooting with PVM on page 31 General troubleshooting VSP 4000 unit failure In the event of a VSP 4000 Series switch unit failure, do not perform a hot swap. For assistance, contact Avaya Support. Note: There is no emergency shutdown procedure for VSP 4000 Series switches. SMLT Basic Troubleshooting Checks Begin any SMLT troubleshooting with the following checks: • Check that the IST is up and running. • Check that SMLT is up and running. • Check the autotopology table for possible loops. EMC storage array EMC recommends that you enable Connect EMC support service. This feature automatically sends an email message to EMC support when problems occur that affect service or require user action. You can enable Connect EMC functionality in the Unisphere Configuration wizard Product Support Option by selecting the Yes option and specifying the network address of an SMTP email server. You can also enable the service from the Alert Settings page. Tip: Booting up and shutting down the EMC VNXe 3200 can take approximately 30 minutes. Please be patient. October 2017 Troubleshooting the Avaya Pod Fx™ 24 Comments on this document? [email protected]

Ensure your FQDN does not exceed a maximum of 35 characters to avoid a system error message during logon. If a blank page appears during login ensure that you are using one of the following supported browsers: • Internet Explorer 8.x, 9.x, or 10.x • Firefox 19.0, 20.0, or 21.0 Note: This problem is seen only when upgrading POS applications. Virtualization Performance Fault Manager If VPFM is installed properly, the network was discovered and the trap viewer shows that traps were received all events appear on the events browser. If no events appear, ensure that monitoring is turned on. MLT configuration problem The VPFM event browser displays a MLT configuration problem message after a discovery is run on a VPFM domain. The message is a low priority (level 4) and warns the NOC operator of potential MLT misconfigurations. To understand the reason for the message click the Attribute tab. The Attribute tab displays the details. MLT configuration problem messages can mean that a port is configured as an MLT but the link is not physically there or VPFM could not find a device on the other end. If this occurs, ensure that the MLT links for the devices is configured and connected correctly, and then acknowledge the event. Managed device warning As a general warning that an IP address was found in the SONMP, LLDP, or CDP table of a discovered device, but the IP object did not respond, VPFM displays the following message: potential managed device was not explored during General troubleshooting October 2017 Troubleshooting the Avaya Pod Fx™ 25 Comments on this document? [email protected]

to query the device. VPFM monitoring When the domain state of an agent is set to monitoring the agents enter a running state and receive VPFM monitoring requests. If the domain monitoring state and the agent run state are green the agent is monitoring both. If a user stops an agent, VPFM monitoring details will be stopped on the agent and can only be restarted by stopping and restarting monitoring for the whole domain. Note: It is not recommended to start and stop individual agents. Pod Utility Module All upgrade files must be provided in order for the VMs to be upgraded. Note: Upgrades are supported for the POS vApps, for all other vApps, manual upgrade is recommended. See Using the Management Server Console for Avaya Pod Fx for more information on prerequisites. Pod Visualization Manager Pod Visualization Manager (PVM) discovers all physical components and builds a view of the Avaya Pod Fx. If you change a component or physically move a component to another rack, ensure that you update the PVM profile to reflect those changes and then use the Rediscover function. This updates the profile with the current component information. Tip: Always have a current profile Inventory backup. A backup can be used at a later date to recover from accidental data loss or administrator error. Note: A red indicator on the PVM dashboard indicates the value of the component has surpassed the threshold value set for this component in the Threshold Table on the PVM Preferences page. High Availability warnings in vSphere client Right-click each Host in the cluster and select Reconfigure for HA if High Availability warnings are encountered in vSphere client. Unexpected Virtual Machine behavior Check Datastore access for the Virtual Machine host in vSphere client if the Virtual Machine behaves in an unexpected manner. If the Application Datastore cannot be reached or the information displayed is incorrect, select Rescan All. You should also check for snapshots and remove any that are present. Troubleshooting general issues October 2017 Troubleshooting the Avaya Pod Fx™ 26 Comments on this document? [email protected]

may be necessary to take the additional step of resetting the Virtual Machine if the Rescan All option does not fix the issue. Unregistered network connections Disruptions in network connectivity to the iSCSI ports on the ESXi hosts can cause ports to lose their registration to the target on the storage system. Select Host > Configuration > Storage adapters > iSCSI software adapter > Rescan All from the vCenter menu to rescan the datastore. Rescanning the datastore should reestablish the port connections. Recovering the MSC Administrator password If the Administrator password is changed from the default and subsequently lost, the only way to recover access to the MSC is to reinstall the MSC OVA as documented in Using the Management Server Console. vCenter does not boot with error See the following troubleshooting procedures if the vCenter Server Appliance does not boot with a fsck failed error: https://kb.vmware.com/selfservice/microsites/search.do? language=en_US&cmd=displayKC&externalId=2081464#EnableBASH. vCenter inventory is empty Check the size of the ldapmessages file if the vCenter inventory is empty. The ldapmessages file does not have any limits on the file size and can grow until it is the size of the partition. Delete the file and restart all services if this file has consumed all resources on the partition. The vCenter inventory will return to normal after the restart. Monitor the size of this file going forward and delete as necessary to prevent the issue from happening in the future. Performance and latency issues The following are common causes of performance and latency issues on the Avaya Pod Fx. Perform these troubleshooting tasks before contacting support. • Resource reservations Search for and remove all resource reservations. • Snapshots Search for and remove all snapshots. • Deployment details do not match initial sizing Performance and latency can be effected if the applications currently deployed on the Avaya Pod Fx do not match the approved design. Obtain the Lifecycle Customer Workbook and approved design documentation. This represents the agreed design of the Avaya Pod Fx and is the documentation used to build it. If the applications currently running on the Avaya Pod Fx outstrip the originally designed capabilities of it, performance will be effected. HP DL360 10GB vmnic in half-duplex mode There are instances when a HP DL360 server starts up with a 10GB vmnic in half-duplex mode when it is connected to a VSP 7254XSQ. General troubleshooting October 2017 Troubleshooting the Avaya Pod Fx™ 27 Comments on this document? [email protected]

esxcli network nic down -n vmnic esxcli network nic up -n vmnic Where is the number of the vmnic to be fixed. Related links Troubleshooting general issues on page 24 Quick Fixes Application Problem Description Solution or Workaround Pod Visualization Manager The application does not open when launched or produces a “SA not accessible” warning. 1. The jboss application server may be down. Check for status and restart jboss using the following commands as root user: • On the PVM VM check the jboss status using the command service jboss status. • Restart jboss using the command service jboss restart. • 2. The Serviceability Agent may be down. Check for status and restart spirit agent using the following commands as root user: • On the PVM VM check the spirit agent status using the command service spiritAgent status. • Restart spirit agent using the following commands: service spiritAgent stop service snmpd restart service spiritAgent start Table continues… Troubleshooting general issues October 2017 Troubleshooting the Avaya Pod Fx™ 28 Comments on this document? [email protected]

Pod Visualization Manager CPU/Temperature/Memory on the PVM Dashboard shows “—” for some IP addresses. The SNMP credentials or ssh username or password combination is incorrect. 1. Ensure that PVM is able to reach the devices and SNMP credentials are configured. 2. Check that the ssh username and password are entered correctly in the Credentials Database. 3. Ensure that the polling interval setting for the device is set to “No Polling” and “N” seconds in the preferences screen. Pod Visualization Manager PVM discovery is not operating as expected. You can stop or restart the Serviceability Agent (SA) using the Administrative toolbar Diagnostics > Stop SA and Diagnostics > Restart SA functions. If required, you can manually start or restart a service using the following procedure: 1. Open a console to the PVM VM. 2. Login to the guest OS. 3. From the PVM VM desktop, open a terminal window. 4. In the Terminal window: a. Restart jboss: • # service jbossd {start|stop|status| restart} b. Restart Serviceability Agent: • service spiritAgent stop service snmpd restart service spiritAgent start Table continues… Quick Fixes October 2017 Troubleshooting the Avaya Pod Fx™ 29 Comments on this document? [email protected]

5. Inspect the logs to confirm the services started correctly. a. /opt/spirit/ logging/spirit.log 6. Verify SA inventory: a. /opt/spirit/ xmlData/ Inventory.xml b. /opt/spirit/ xmlData/Backup Pod Visualization Manager Unable to add a device using Profile Editor. The device being added may not be reachable from the PVM virtual machine. 1. Ping the device from the PVM Virtual Machine. If the device responds it is reachable. 2. Check if the SNMP ports are reachable from the PVM Virtual Machine. 3. Check if the SNMP credentials are configured properly in the Credentials Editor. ESXi hosts EVC mode are disabled on the ESXi host clusters by default. Avaya Pod Fx that have been expanded with additional compute servers of different make and type can experience issues migrating VMs between hosts in the cluster with EVC mode disabled. Mitigation of this requires creation of a new host cluster with the EVC mode enabled, moving hosts to the EVC mode cluster, manual migration of the vCenter to new EVC cluster, powering down VMs and moving them to the EVC cluster, and cleaning up the rest of the solution. The following must be done if this situation is encountered: 1. Create a new host cluster with EVC mode enabled. 2. Move hosts to the new EVC mode enabled cluster. 3. Manual migration of vCenter to the new cluster. Table continues… Troubleshooting general issues October 2017 Troubleshooting the Avaya Pod Fx™ 30 Comments on this document? [email protected]

4. Power down VMs and move them to the new cluster. 5. Take any other steps necessary to use this new cluster. See https://kb.vmware.com/ selfservice/microsites/search.do? language=en_US&cmd=displayKC &externalId=1013111&src=vmw_s o_vex_pbrar_70 for additional information and procedures on this topic. vSphere client The vSphere client prompts you to accept an SSL certificate the first time you access a Nimble device on the Nimble management tab. Accept the SSL certificate. Related links Troubleshooting general issues on page 24 Troubleshooting with PVM Pod Visualization Manager (PVM) provides access to the visualization and management applications required to configure, orchestrate, manage, and monitor your Avaya Pod Fx. The following table lists some common troubleshooting information displayed in PVM. Symptom Troubleshooting Issue or Action A component displays Unreachable and a green LED is not displayed. Check the physical connectivity of the component. HP Server: an amber LED is displayed. Faulty disk. HP Server: no LED displayed on power supply. The power supply is missing or faulty. HP Server: no LED on the iLO port. The iLO port is not configured. Lenovo Server: an amber LED is displayed on the power supply. Faulty power supply. Nimble CS1000 Storage Server Disk LED: an orange LED Faulty disk. Table continues… Troubleshooting with PVM October 2017 Troubleshooting the Avaya Pod Fx™ 31 Comments on this document? [email protected]

Note: Hovering the mouse over an LED presents more information. Nimble CS1000 Storage Server Disk LED: EMPTY Disk. The disk is missing. Note: Hovering the mouse over an LED presents more information. Nimble CS1000 Storage Power LED: an orange LED. An orange LED can mean any of the following: • Power is off and AC power connected. • The power supply is not properly seated. • The array is powered off. • Power supply internal fault. Note: Hovering the mouse over an LED presents more information. Nimble CS1000 Storage Ethernet Links LED: on orange LED. Ethernet link is down. Note: Hovering the mouse over an LED presents more information. Nimble CS1000 Storage Fiber Links LED: on orange LED. Fiber link is down. Note: Hovering the mouse over an LED presents more information. EMC VNXe 3200 Storage Server Disk LED: an orange LED next to a blinking LED. Faulty disk. Note: Hovering the mouse over an LED presents more information. EMC VNXe 3200 Storage Server Disk LED: no LED shown. The disk is missing. Note: Hovering the mouse over an LED presents more information. EMC VNX 5300 Storage Server Disk LED: an orange LED Faulty disk. Note: Hovering the mouse over an LED presents more information. Power Consumption Chart: unusual under or over consumption of power is shown. Power consumption should be monitored on a regular basis using the tools provided by PVM. Unusual power fluctuations are a sign of a malfunctioning device or too much power is drawn by the Avaya Pod Fx. Either circumstance requires further investigation. Red thresholds are reached in CPU usage, memory usage, or The Pod Dashboard displays information on CPU and memory usage and temperature for all Avaya Pod Fx components. A red threshold Table continues… Troubleshooting general issues October 2017 Troubleshooting the Avaya Pod Fx™ 32 Comments on this document? [email protected]

temperature on the Pod Dashboard. indicator for any of these metrics indicates a situation that requires immediate attention. Note: Threshold values are initially set to default values. They are configurable through the PVM to suit individual situations and preferences. PVM is a consolidated launch pad for all elements in the Avaya Pod Fx solution. Related links Troubleshooting general issues on page 24 Troubleshooting with PVM October 2017 Troubleshooting the Avaya Pod Fx™ 33 Comments on this document? [email protected]

This section contains information and procedures for troubleshooting software issues. Related links Troubleshooting Avaya Aura Communication Manager in an Avaya Aura Virtualized Environment on page 34 ESXi host alarm after VMware 6.0 Update 3A upgrade on page 36 vCenter Server inaccessible after server failure or unexpected server reboot on page 37 Reconfiguring a cluster on page 37 Timeout error configuring High Availability in vCenter on page 39 Backing up and restoring vCenter configuration on page 41 Recovering from a failed VCSA 5.5 upgrade on page 42 Avaya Aura VMs do not recover on page 43 Troubleshooting Avaya Aura® Communication Manager in an Avaya Aura® Virtualized Environment The issues in the following section are encountered when running Avaya Aura® Communication Manager in an Avaya Aura® Virtualized Environment such as the Avaya Pod Fx. Servers are temporarily in an active / active state Condition Avaya Aura® Communication Manager servers temporarily run in an active / active state in a duplicated server configuration. The active server becomes unresponsive, Avaya Aura® Communication Manager interchanges to the standby server, and the unresponsive server returns to an active state. Avaya Aura® Communication Manager then runs in an active / active state until the arbiter resolves the dual state and returns one server to standby status. Cause Avaya Aura® Communication Manager is working as designed. There is an underlying cause in the overall solution configuration that needs to be addressed. Solution October 2017 Troubleshooting the Avaya Pod Fx™ 34 Comments on this document? [email protected]

• Datastore architecture should be designed to tolerate drive failures, controller failures, and connectivity issues without disruption. The active and standby servers should be located on separate datastores. • Avaya Aura® Communication Manager CPU reservations should be used if ESXi hosts are oversubscribed. See Deploying Avaya Aura® Communication Manager on VMware® in Virtualized Environment (6.x VE only) and Deploying Avaya Aura® Communication Manager in Virtualized Environment (7.x VE and AVP) for reservation guidelines. • VMware recommends only allocating to a virtual machine as many virtual resources as it requires. Allocating more resources than necessary can, in some cases, reduce VM performance as well as the performance of other VMs on the same host. See the section ESXi CPU Considerations in Performance Best Practices for VMware vSphere 6.0 in the VMware 6.0 documentation for more information. Link bandwidth or connectivity issues cause non-deterministic behavior Condition Active / active states, filesync failures, server interchanges, server resets, or other non-deterministic behavior. Cause Duplication link bandwidth or connectivity issues might cause active / active states, filesync failures, server interchanges, server resets, or other non-deterministic behavior. Solution The duplication link must have a total capacity of 1Gbps that is dedicated to CM duplication. The duplication link interface should be a separate and dedicated interface. See PSN003556u on the Avaya Support website for additional information. See Duplicated Avaya Aura® Communication Manager on VMware for best practices on deploying duplicated CM servers in a VE configuration. It should be referenced for any duplicated CM VE deployment. Snapshots can adversely impact service Condition Use of snapshots on Avaya Aura® Communication Manager can cause interchanges or adversely impact service in other ways. Extreme caution must be exercised when using virtual machine snapshots with Avaya Aura® Communication Manager deployed on VMware. Solution The following must be understood if you are considering the use of snapshots: • Snapshot operations should never be performed on the active VM. The application must be stopped and placed out of service before performing snapshot operations. • The application should be started and brought back into service after the snapshot operation is complete. • If the duplex Open Virtual Application (OVA) is in use, snapshot operations should be performed on the standby CM VM after ensuring the standby is refreshed. If a snapshot is taken on the active VM a service impacting interchange is likely to occur. Troubleshooting Avaya Aura® Communication Manager in an Avaya Aura® Virtualized Environment October 2017 Troubleshooting the Avaya Pod Fx™ 35 Comments on this document? [email protected]

deleted or reverted. Additional information on snapshot removal can be found in KB1002836 on the VMware Knowledge Base website. • Third party based snapshots are not supported. • Best practices for snapshots are documented in the Deploying Avaya Aura® Communication Manager on VMware® in Virtualized Environment document. Important: Snapshots should not be used as a backup and recovery mechanism. They are only intended to test an action that may result in an unpredictable outcome. Using vMotion with real time applications can impact service Condition Use of vMotion with real time applications such as Avaya Aura® Communication Manager can be service impacting and is not recommended. Solution If vMotion is used, it is recommended that manual migrations with vMotion be done only during a maintenance window since the migration could be service impacting. If vMotion DRS automation is used, it is recommended that the DRS automation level be set to the most conservative level possible, such as level 2 or level 1. See PSN20151u on the Avaya Support website for more information. Related links Troubleshooting software issues on page 34 ESXi host alarm after VMware 6.0 Update 3A upgrade Condition The alarm message System logs on host are stored on non-persistent storage is displayed after upgrading an ESXi host to VMware 6.0 Update 3A. Cause The upgrade changes the System logs location is changed to a location that would not save them if the server was rebooted. This is a default action of the upgrade process and cannot be changed before the upgrade begins. Solution Use the following procedure to fix this issue: https://kb.vmware.com/selfservice/microsites/ search.do?language=en_US&cmd=displayKC&externalId=2032823. Related links Troubleshooting software issues on page 34 Troubleshooting software issues October 2017 Troubleshooting the Avaya Pod Fx™ 36 Comments on this document? [email protected]

unexpected server reboot The VMware vCenter Server status shows Stopped and fails to Start. In normal operation, status shows Running. Occurrence of issue: very rare Overall completion: approximately 1 hour Cause A host server failure or unexpected reboot of the host server with the vCenter Appliance and the vCenter database virtual machines, which causes the vCenter appliance and the vCenter database server to be migrated to a new host. This results in a temporary interruption in access to vCenter. If vCenter cannot be reached or accessed after 15 minutes, the vCenter appliance can have issues with the services starting up. The message states vpxd failed to start. Contact Avaya Support for advanced troubleshooting and repair. Solution 1. Log in to the vCenter web interface located at: https://. 2. Click Summary. 3. In the vCenter Server section, click Start to restart the Server process. 4. Verify that the status changes to Running. vCenter is back online. Proceed to Cluster Reconfiguration on page 37 for the procedure to configure the cluster. Related links Troubleshooting software issues on page 34 Reconfiguring a cluster About this task This procedure is used to reconfigure a cluster after an unexpected shutdown of the Avaya Pod Fx. Procedure 1. On the Management Server Console, launch the vCenter vSphere Client, and log in. Important: Administrator must have advanced knowledge of VMware and vCenter. The vCenter screens do not show any administration. 2. Use the wizard to create a Data Center and Cluster. 3. Add the hosts to the Cluster. vCenter Server inaccessible after server failure or unexpected server reboot October 2017 Troubleshooting the Avaya Pod Fx™ 37 Comments on this document? [email protected]

4. In vCenter licensing, add missing licenses for vCenter and hosts. 5. Assign the licenses to the vCenter and hosts, as applicable. 6. In vCenter, click the address for the Avaya Pod Fx settings you want to administer, select the Turn On vSphere HA and Turn On vSphere DRS check boxes, and click OK. 7. In the Advanced Options (vSphere HA) dialog box, use the data in the following table to restore the VMware HA advanced settings, and click OK. Option Value Description das.isolationaddress0 to das.isolationaddress9 Enter the Default Gateway IP addresses. Changes the default isolation IP address. The default address is the Default Gateway address for the ESXi host because it is a highly available IP address. The das.isolationaddress is queried to determine if an isolation response has occurred on the host. If you change this parameter, you must use a reliable, highly available IP address with as few failure points as possible between it and the ESXi host. das.usedefaultisolationaddress Enter false. Disables the use of the Default Gateway as an isolation address. Change the value to false. By default, the value is true. This parameter is generally used in conjunction with the das.isolationaddress0 to das.isolationaddress9 parameters. The following example shows the default Gateway IP addresses for das.isolationaddress1 and das.isolationaddress2. The das.usedefaultisolationaddress is set to false. Troubleshooting software issues October 2017 Troubleshooting the Avaya Pod Fx™ 38 Comments on this document? [email protected]

9. Move the threshold slider to the right and apply priority 1 and 2 recommendations, and click OK. Next steps Important: After vCenter is back to normal operation, remove any snapshots that you took to preserve data. Snapshots can adversely affect VM performance if left in place. You must then configure HA/DRS. Related links Troubleshooting software issues on page 34 Timeout error configuring High Availability in vCenter Condition A timeout might occur during the configuration of High Availability in VMware vCenter. An error similar to the following appears in the /var/log/fdm.log file. [ClusterManagerImpl::ConnectToMaster] Couldn't connect to master N7Vmacore16TimeoutExceptionE(Operation timed out) [ClusterManagerImpl::AddBadIP] IP 10.x.x.x marked bad for reason Unreachable IP [ClusterElection::ChangeState] SlaveConnecting => Startup : SlaveConnectingStateFunc [ClusterManagerImpl::IsBadIP] 10.x.x.x is bad ip The following situations are observed as a result of the timeout: • High Availability configuration fails. • High Availability cannot be configured on a cluster. • High Availability configuration completes on the master but fails on all slaves. Cause This issue occurs if Jumbo Frames are enabled on the host Management Network (the VMkernel port used for host management) and the network configuration is not configured for jumbo frames. This issue is resolved by ensuring that the vSwitch and Management Network MTU values are configured to 1500. Solution 1. Right-click the cluster host in vSphere Client. 2. Click the Configuration tab. 3. Click Networking. 4. Click Properties for the Management Network on vSwitch0 on Release 3.0 Pods and vSwitch1 on pre-Release 3.0 Pods. Timeout error configuring High Availability in vCenter October 2017 Troubleshooting the Avaya Pod Fx™ 39 Comments on this document? [email protected]

6. Click vSwitch, 7. Click Edit. 8. Change the MTU value to 1500 if it is not already configured to that value. 9. Click OK. 10. Click the Ports tab. 11. Click Management Network. 12. Click Edit. 13. Change the MTU value to 1500 if it is not already configured to that value. 14. Click OK. 15. Repeat steps 1–14 for all hosts in the cluster. 16. Right-click the cluster and attempt to configure High Availability. Verify the MTU settings of the Avaya Virtual Services Platform 4850GTS switches if you continue to encounter issues with High Availability configuration. Both devices must be configured to have an MTU setting of 1950 to ensure proper operation. 1. Connect to the switch. 2. Enter Global Configuration mode. enable configure terminal 3. Display the current switch MTU setting. show sys mtu 4. Change the MTU setting to 1950 if it is not already that value. sys mtu 1950 5. Confirm the new MTU setting. show sys mtu 6. Save the new configuration. save config 7. Repeat steps 1–6 for all applicable Avaya Virtual Services Platform 4850GTS switches. Related links Troubleshooting software issues on page 34 Troubleshooting software issues October 2017 Troubleshooting the Avaya Pod Fx™ 40 Comments on this document? [email protected]

About this task This procedure provides steps for backing up an existing vCenter server and restoring the configuration to a new server. See https://kb.vmware.com/selfservice/microsites/search.do? language=en_US&cmd=displayKC&externalId=2034505 for additional information about this procedure. Procedure 1. Deploy the new vCenter server. 2. Navigate to the new server using a web browser and the URL https:// :5480. 3. Accept the certificate when prompted. 4. Log in to the new server. 5. Log in to the old server using PuTTY, 6. Stop the VMware vCenter Server service running on the old server using the command vmware-vpxd stop. 7. Navigate to the vPostgres utility directory on the old server using the command cd /opt/ vmware/vpostgres/current/bin. 8. Display the vPostgres database configuration file using the command cat /etc/vmware- vpx/embedded_db.cfg. 9. Note the value of EMB_DB_INSTANCE and EMB_DB_USER. 10. Create the vCenter backup using the command ./pg_dump -U -Fp -c > /tmp/VCDBBackupFile where and are the values noted in the previous step. 11. Copy the VCDBBackupFile to the local system using SCP. 12. Create an SCP session with the new vCenter Server. 13. Copy the VCDBBackupFile to /tmp on new vCenter Server. 14. Create a SSH session to the new vCenter Server using PuTTY. 15. Display the new vPostgres database configuration file on new vCenter Server using the command cat /etc/vmware-vpx/embedded_db.cfg. 16. Note the value of EMB_DB_PASSWORD and EMB_DB_INSTANCE. 17. Navigate to the vPostgres utility directory using the command cd /opt/vmware/ vpostgres/current/bin. 18. Stop the VMware vCenter Server service using the command service vmware-vpxd stop. Backing up and restoring vCenter configuration October 2017 Troubleshooting the Avaya Pod Fx™ 41 Comments on this document? [email protected]

PGPASSWORD= ./psql -d -Upostgres -f VCDBBackupFile where and are the values noted in the previous step. 20. Restart the vCenter Server using the command service vmware-vpxd start. Related links Troubleshooting software issues on page 34 Recovering from a failed VCSA 5.5 upgrade About this task The following procedure provides steps for recovering from a failed upgrade to vCenter Server Appliance (VCSA) Release 5.1 to 5.5. This procedure presents a high level overview of the process. Refer to the appropriate documentation for procedures such as: • Upgrading VCSA deployments. • vPostgres database file backup and restoration. • VCSA deployment. • Host connection. • Reverting to VCSA snapshots. Procedure 1. Backup and restore the failed upgrade to a new 5.5 instance. a. Backup the vPostgres database from the failed 5.5 upgrade. b. Note the IP address, DNS settings, and Hostname of the failed upgrade. c. Shutdown the failed 5.5 upgrade. d. Deploy a new 5.5 instance with the same IP address, DNS settings, and Hostname as the failed upgrade. e. Restore the vPostgres database backup file to the new 5.5 instance. f. Log in to the new vCenter instance. g. Connect any hosts that display as disconnected using root credentials. h. Proceed to the next step if the 5.5 upgrade does not complete successfully. 2. Perform a fresh upgrade from 5.1 to 5.5. a. Revert back to the 5.1 snapshot. b. Perform a fresh upgrade from 5.1 to 5.5. c. Proceed to the next step if the 5.5 upgrade does not complete successfully. Troubleshooting software issues October 2017 Troubleshooting the Avaya Pod Fx™ 42 Comments on this document? [email protected]

instance. a. Revert back to the 5.1 snapshot. b. Backup the vPostgres database from the 5.1 instance. c. Note the IP address, DNS settings, and Hostname of the 5.1 instance. d. Shutdown the 5.1 instance. e. Deploy a new 5.1 instance with the same IP address, DNS settings, and Hostname as the 5.1 instance just shut down. f. Restore the vPostgres database file to the new 5.1 instance. g. Upgrade the new 5.1 instance to 5.5. Related links Troubleshooting software issues on page 34 Avaya Aura® VMs do not recover Condition Avaya Aura® VMs may not recover successfully after a sudden and complete power outage when systems are not able to gracefully shut down. Cause vCenter attempts to migrate the VMs to a different host when the power outage starts but cannot complete the action when all hosts lose power. When the hosts are powered on again, vCenter attempts to restart all hosts but cannot successfully reconnect the VMs to the network. This issue impacts SMGR, AMS, CMS, and Session Manager instances. The following symptoms are found on the Avaya Pod Fx when a power outage has occurred: • vCenter is not able to complete the process of powering on the VM after the outage. • An event is displayed indicating the Ethernet interfaces associated with the Avaya Aura® VM will start disconnected. • An error message is displayed if an attempt is made to manually reconnect the network adapter. Solution 1. Remove network adapters of affected VMs. 2. Add back network adapters of affected VMs. 3. Assign the MAC address originally used with the VM so the correct HWADDR value is assigned to the Eth0 interface. 4. Install any necessary authentication files associated with the affected VMs. Avaya Aura® VMs do not recover October 2017 Troubleshooting the Avaya Pod Fx™ 43 Comments on this document? [email protected]

Troubleshooting software issues October 2017 Troubleshooting the Avaya Pod Fx™ 44 Comments on this document? [email protected]

issues This section contains information and procedures for troubleshooting hardware issues. It also contains procedures commonly used in hardware troubleshooting such as firmware upgrades. Related links Lenovo compute server power supply alarm troubleshooting on page 45 Upgrading HP ProLiant Gen8 server BIOS for POST errors on page 50 Upgrading Lenovo server BIOS for fan alarm on page 52 Checklist for replacing a Power Distribution Unit on page 53 Lenovo compute server power supply alarm troubleshooting Condition The system error LED and Power Supply Unit (PSU) alert on the front panel might stay lit even if the power supplies are operating normally and there are no errors in the System Event logs. In this scenario, the issue is cosmetic and does not affect power supply or system functionality. Cause False PSU alert errors are a known issue with Lenovo BMC firmware prior to version 20.00. Solution 1. Check the LED on the rear of the PSU when a system error LED and PSU alert is shown on the front panel. • If the LED on the PSU is green or blinking green, the errors and alerts are false and the PSU is indicating a good state. • If the LED on the PSU is amber, the errors and alerts are valid and the PSU might be failing and require a replacement. 2. Go to the Lenovo support website and search for keywords BMC update Web GUI. 3. Select the BMC Update utility for Web GUI that is applicable to your ThinkServer model. 4. Verify that the update version shown is version 20.00 or higher. Download both of the update files (the .tgz update utility and the readme). October 2017 Troubleshooting the Avaya Pod Fx™ 45 Comments on this document? [email protected]

host that requires the BMC update into maintenance mode. Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 46 Comments on this document? [email protected]

VMware maintenance mode. 6. Open the readme file that you downloaded from the Lenovo website. Perform the steps in the INSTALLATION INSTRUCTIONS section to extract the bmc_xxx.ast file, upload the file to the server, and apply the update. Note: The default username and password for the ThinkServer Management Module (TMM) is typically: • Username: lenovo • Password: len0vO (lenv) Lenovo compute server power supply alarm troubleshooting October 2017 Troubleshooting the Avaya Pod Fx™ 47 Comments on this document? [email protected]

proceeds. You can check the status by watching the status column. Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 48 Comments on this document? [email protected]

Confirm the firmware update by viewing the Properties page. 9. Using the Management Server Console on the Avaya Pod Fx, open vCenter and exit the host from maintenance mode. Lenovo compute server power supply alarm troubleshooting October 2017 Troubleshooting the Avaya Pod Fx™ 49 Comments on this document? [email protected]

fix false PSU alerts. Related links Troubleshooting hardware issues on page 45 Upgrading HP ProLiant Gen8 server BIOS for POST errors Condition HP ProLiant Gen8 servers may experience unexpected errors during a reboot. This involves the server displaying POST errors 236 or 239. The server may also reboot unexpectedly and display POST error 207. Cause These POST error conditions are a known issue with some HP ProLiant Gen8 servers. Use the following procedure to upgrade the server BIOS to the latest version to correct this error. Important: The host server must be in maintenance mode to perform a BIOS upgrade. Ensure that the server is in maintenance mode before beginning this procedure. Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 50 Comments on this document? [email protected]

out, and log in as the root user. Solution 1. Go to the HP support website (HTTP://SUPPORT.HP.COM/). 2. Select Software and Driver downloads. 3. Enter HP ProLiant DL360p Gen8 Server. 4. Select Find my Product. 5. Select Servers. 6. Select HP ProLiant DL360p Gen8 Server. 7. Select the applicable version of VMWare OS from the drop down list. 8. Select BIOS (Entitlement Required)- System ROM. 9. Select HP BIOS version 2014.11.01 (CP025384.scexe) or higher for VMware ESXI and download it to a temporary directory such as /tmp. Note: A valid HP account with a valid contract and warranties is required to access the necessary downloads. 10. Perform the following steps to enable Tech Support Mode on the server: a. Access the server console using the Integrated Lights-Out (ILO) interface or by directly attaching a monitor and keyboard to the server b. Log in using root credentials. c. Navigate to Troubleshooting Options using the keyboard. d. Enable ESXi Shell and SSH Support. 11. Copy the downloaded file to a temporary directory on the server to be upgraded. Note: HP recommends verifying the results of your download with the MD5 checksum value found on the Installation Instructions tab to ensure the integrity of your download. 12. Locate the temporary directory that contains the file you copied in step 9 and execute the file. Note: Ensure that the versions listed are accurate before proceeding. 13. Upgrade the software to the newer version by typing y when prompted. 14. A message is displayed when the upgrade is complete. 15. Reboot from vCenter or from the server. 16. Use vCenter client to log in directly to the server when the update completes. 17. Confirm the BIOS was upgraded, the server is fully functional, and there are no major hardware alarms. Upgrading HP ProLiant Gen8 server BIOS for POST errors October 2017 Troubleshooting the Avaya Pod Fx™ 51 Comments on this document? [email protected]

Related links Troubleshooting hardware issues on page 45 Upgrading Lenovo server BIOS for fan alarm Condition Lenovo servers may experience the fan alarm Fan sensor, warning event was asserted, reading value: 0RPM (Threshold : 2496RPM) when it is not applicable. Cause This is a known issue with Lenovo servers. Use the following procedure to upgrade the server BIOS to the latest version to correct this error. Important: The host server must be in maintenance mode to perform a BIOS upgrade. Ensure that the server is in maintenance mode before beginning this procedure. Solution 1. Go to the Lenovo support website (HTTP://SUPPORT.LENOVO.COM/). 2. Select Servers > ThinkServer rack servers. 3. Select the applicable server from the drop down list. 4. Select BIOS in the Component list. 5. Select BIOS Update Utility for BMC Web GUI-ThinkServer RDXXX where RDXXX represents the specific server model. 6. Verify that the update version shown is version A1TS8BA or higher. 7. Download the TGZ update utility and readme files. 8. Decompress the TGZ update utility and confirm it contains a file with an RBU extension. 9. Confirm the BIOS version is A1TS8BA or higher. 10. Save the decompressed files to a location that is reachable by the server to be upgraded. 11. Place the host in maintenance mode using vCenter on the Management Server Console. 12. Connect to the IP address of the BMC port on the server using a web browser. 13. Log into the ThinkServer Management Module (TMM). Note: The default username and password for the TMM is typically: • Username: lenovo • Password: len0VO (lenv) or lenovo Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 52 Comments on this document? [email protected]

15. Select BIOS from the Firmware Type drop down list. 16. Select Browse. 17. Browse to the location of the RBU file from step 8. 18. Select Upload. 19. Verify that the correct file is present when the upload completes. 20. Select Update. 21. Reboot from vCenter or from the server. 22. Use vCenter client to log in directly to the server when the update completes. 23. Confirm the BIOS was upgraded, the server is fully functional, and there are no major hardware alarms. 24. Remove the server from Maintenance Mode. Related links Troubleshooting hardware issues on page 45 Checklist for replacing a Power Distribution Unit Use the following general checklist for the field replacement of a Power Distribution Unit. Refer to the following manufacturer documentation for specific instructions on installation and use: • ServerTech — https://www.servertech.com/support Important: Ensure there is no strain when dressing the RJ45 Ethernet cables connecting the PDU to the switch. These cables have to make a right turn through the chassis to get to their connected pieces of equipment. Always allow adequate strain relief at the PDU end of the cables. No. Task Description Notes 1 PDU arrives on site. PDU arrives at the site of Avaya Pod Fx with faulty PDU. 2 Mark defective PDU. Defective PDU is marked in the support ticket. 3 Note shipped PDU serial number. The technician or installer must write down the serial number of the shipped PDU. Table continues… Checklist for replacing a Power Distribution Unit October 2017 Troubleshooting the Avaya Pod Fx™ 53 Comments on this document? [email protected]

has redundant power by ensuring that both power supply LEDs are green on all devices. 5 Troubleshoot non- redundant equipment power. If not all equipment has redundant power, confirm that the Power Supply with the unlit LED is connected to the faulty PDU. See the Customer Lifecycle Workbook for information on which equipment is connected to which PDU. 6 Remove PDU power. Remove power from the faulty PDU by removing the cord that connects it to the customer power supply. 7 Remove all cables. Remove all cables from the faulty PDU. 8 Remove PDU from enclosure. Remove the PDU from the Avaya Pod Fx enclosure, ensuring not to bump or disconnect any cords from active PDUs. 9 Locate PDU serial number. Locate the faulty PDU serial number in the Customer Lifecycle Workbook. Replace it with the serial number of the replacement PDU. 10 Install PDU in enclosure. Install the replacement PDU in the Avaya Pod Fx enclosure. 11 Power replacement PDU. Plug the replacement PDU into the customer power source. 12 Configure PDU. See Configuring switched PDUs over Ethernet on page 55 or Configuring switched PDUs over Console Connection on Table continues… Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 54 Comments on this document? [email protected]

procedures. 13 Attach all cables. Attach equipment cables to the PDU in the same location they were attached in the faulty PDU. See the Customer Lifecycle Workbook for cable locations. 14 Confirm PDU operation. Inspect all equipment to ensure it is now displaying two green power supply LEDs for each piece of equipment. 15 Confirm equipment operation. Confirm that all equipment is operational, including the replacement PDU. The PDU LED display should be showing the same current level as the corresponding paired PDU. 16 Return faulty PDU. Installer or technician must return the faulty PDU to Avaya using standard equipment return procedures. Related links Troubleshooting hardware issues on page 45 Configuring switched PDUs over Ethernet on page 55 Configuring switched PDUs over a Console connection on page 58 Configuring switched PDUs over Ethernet About this task This procedure provides the information necessary to configure a switched PDU using the internal Avaya Pod Fx network. PDU configuration is required after installing a new PDU. See Configuring switched PDUs over Console Connection on page 58 for the procedure to configure a switched PDU using a DB9 connection. Checklist for replacing a Power Distribution Unit October 2017 Troubleshooting the Avaya Pod Fx™ 55 Comments on this document? [email protected]

You will require the following items to complete this procedure: • A laptop with an Ethernet port. • RJ45 to RJ45 cable. Use a crossover cable if the laptop in use does not support autosensing. Procedure 1. Configure the IP settings on the service laptop to the following: • IP Address — 192.168.1.253 • Subnet Mask — 255.255.255.0 • Default Gateway — None 2. Connect the service laptop directly to the PDU Ethernet port label NET using an appropriate RJ45 UTP cable. Note: The PDU Ethernet port labeled SER cannot be used for this task. 3. Open a terminal session and ping the PDU IP address of 192.168.1.254. If there is a ping response, the network connection can be used to configure the PDU. If there is no response from the ping, verify the laptop network settings and try again. If there is still no response, see Configuring switched PDUs over Console Connection on page 58 for a procedure to use a Console connection to configure the PDU. 4. Open a browser window and connect to the PDU management software at http:// 192.168.1.254. 5. Log in to the management software with the username and password of admn. 6. Select Configuration > Users from the menu. 7. Select Edit for the ADMN user. 8. Consult the Customer Lifecycle Workbook for the PDU password. 9. Change the PDU password to the password contained in the workbook. Use the password CPod2013!! if there is no password in the workbook. 10. Select Configuration > Telnet/SSH from the menu. 11. Select Disabled from the Server dropdown list. 12. Click Apply. 13. Perform the following steps if the PDU is vertically mounted with the exit line and cord in the UP position: a. Select Configuration > System from the menu. b. Select Inverted from the Display Orientation dropdown list. c. Click Apply. Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 56 Comments on this document? [email protected]

15. Deselect DHCP. 16. Consult the Customer Lifecycle Workbook for the PDU IPv4 address information. 17. Use the information from the workbook to configure the IPv4 address options of the PDU. 18. Click Apply. 19. Select Configuation > SNMP/Thresholds from the menu. 20. Change the GET Community (Read-Only) field to avaya123. 21. Change the SysName field to Sentry3_SMART_PDU_#_type where # is the PDU number and type is either MAIN or EXT. MAIN denotes a PDU in a main Avaya Pod Fx and EXT denotes a PDU in an extension Avaya Pod Fx. Examples PDU 2 on a main Avaya Pod Fx would be labelled Sentry3_SMART_PDU_2_MAIN. PDU 1 on an extention Avaya Pod Fx would be labelled Sentry3_SMART_PDU_1_EXT. 22. Change the SysLocation field to Main Pod or Extension Pod depending on the installation type. 23. Click Apply. 24. Select Tools > Restart from the menu. 25. Select Restart from the dropdown list. 26. Click Apply. 27. Click OK. The PDU will now restart. 28. Disconnect the Ethernet cable used to connect the laptop to the PDU. 29. Connect the PDU to the correct port on the switch. See the Customer Lifecycle Workbook for port information. 30. Use the browser provided with the MSC to connect to the PDU management software. 31. Log into the management software to confirm connectivity. 32. Repeat these steps for the next PDU if more than one requires configuration. Related links Checklist for replacing a Power Distribution Unit on page 53 Checklist for replacing a Power Distribution Unit October 2017 Troubleshooting the Avaya Pod Fx™ 57 Comments on this document? [email protected]

Before you begin You will require the following items to complete this procedure: • A laptop with a serial port or a laptop with no serial port and a USB to DB9 adapter. • RJ45 to DB9 RS-232 (DTE) cable. PDUs are equipped with an RJ45 DTE RS-232c serial port. RJ45 to DB9 RS-232 (DTE) cable must have the pinout configuration outlined in the following table. Pin DTE Signal Name Input/Output 1 Request to Send RTS Output 2 Data Terminal Ready DTR Output 3 Transmit Data TD Output 4 Signal Ground 5 Signal Ground 6 Receive Data RD Input 7 Data Set ready DSR Input 8 Clear to Send CTS Input About this task This procedure provides the information necessary to configure a switched PDU using a Console connection. PDU configuration is required after installing a new PDU. See Configuring switched PDUs over Ethernet on page 55 for the procedure to configure a switched PDU using an Ethernet connection. Procedure 1. Connect the Console cable to the laptop and the PDU service port. 2. Connect to the PDU with a terminal program such as PuTTY. 3. Log in with the username and password of admn. 4. Use the command set ipv4 address to enter the new IPv4 address. 5. Use the command set ipv4 subnet to enter the new IPv4 subnet mask. 6. Use the command set ipv4 gateway to enter the new IPv4 default gateway. 7. Use the command restart to restart the administrative interface. Note: This action only restarts the administrative interface. It does not reboot the power outlets or remove power from the outlets. 8. Confirm by pressing Y when prompted. Troubleshooting hardware issues October 2017 Troubleshooting the Avaya Pod Fx™ 58 Comments on this document? [email protected]

10. Use the command show network to confirm the new network information was applied. 11. If the network information was applied properly, close the Console connection. 12. Disconnect the Console cable used to connect the laptop to the PDU. 13. Connect the PDU to the correct port on the switch. See the Customer Lifecycle Workbook for port information. 14. Use the browser provided with the MSC to connect to the PDU management software. 15. Log into the management software to confirm connectivity. 16. Repeat these steps for the next PDU if more than one requires configuration. Related links Checklist for replacing a Power Distribution Unit on page 53 Checklist for replacing a Power Distribution Unit October 2017 Troubleshooting the Avaya Pod Fx™ 59 Comments on this document? [email protected]

This section contains information on additional available resources. Related links Support on page 60 Documentation reference on page 62 Training on page 63 Avaya Mentor videos on page 63 Searching a documentation collection on page 64 Subscribing to e-notifications on page 65 Support The Avaya Pod Fx is an integrated solution with integrated support. Integrated support means that Avaya support personnel can isolate and resolve any issues with hardware and software applications packaged within the Avaya Pod Fx. Note: Avaya is responsible only for support of the Avaya and third-party hardware and software within the Avaya Pod Fx that has been purchased through Avaya as part of the Avaya Pod Fx, or that has been explicitly approved in writing by the Avaya Pod Fx product management team. Documentation for specific Avaya products can be found on the Avaya website at http:// support.avaya.com. Important: If you deviate from an Avaya Pod Fx release software baseline, change physical equipment or virtual machines, modify network configurations, or add custom applications to Avaya Pod Fx in any way, without express consent from Avaya, you will void warranty and service contracts for Avaya Pod Fx. The approved application baseline deployed in the Avaya Pod Fx is part of the design review process. It is documented in the LCM workbook. You must stay within that defined baseline to remain in a supported configuration. If application types or counts must change, the new requirements must be reviewed by the Avaya Pod Fx Go To Market team in a design review through your account team or sales authorized business partner. October 2017 Troubleshooting the Avaya Pod Fx™ 60 Comments on this document? [email protected]

or indirect impact of same, if installed by the customer. To ensure the integrity of an Avaya Pod Fx solution and product warranties, any changes must be approved in writing by the Avaya Pod Fx product management team through the Avaya Pod Fx Configurator Design Review process. To request a design change review, send an email to [email protected]. Important: The initial software baseline can require software patches and updates after initial installation. You are not permitted to update or upgrade Avaya Pod Fx Infrastructure applications or Avaya Aura® System Manager. This can only be done as a whole, with upgrade from one Avaya Pod Fx baseline release to another, unless approved by a PSN or PCN issued by the Avaya Pod Fx product team. This is as per the defined Lifecycle Policy for Avaya Pod Fx release management. Applications outside of the Avaya Pod Fx Infrastructure baseline or Avaya Aura® System Manager can be upgraded at the discretion of the system owner by following the Avaya Product Compatibility Matrix at https://support.avaya.com/CompatibilityMatrix/Index.aspx. You must ensure that automatic software updates are disabled for applications. Avaya tests updates to determine compatibility and Avaya will advise you about updates as they are verified and approved for use. Note: Avaya recommends that you install important Windows updates on the Windows Virtual Machines running on an Avaya Pod Fxas a best practice. See and follow your corporation security policies regarding Windows updates. Contacting Avaya Pod Fx Support The following support mailing lists are available for specific product queries. Use the mailing list appropriate to your query type. • Avaya Pod Fx sales and sales-related queries — [email protected]. • Avaya Pod Fx support — [email protected]. • Avaya Pod Fx partner support — [email protected]. Avaya associates can access the LCM repository at https://products.share.avaya.com/sites/podlcm/ default.aspx to obtain the latest copy of the appropriate workbook. Business Partners and customers should send a request for the latest copy of the workbook to [email protected]. About this task Avaya Support includes: • Products • Downloads and documents • Service requests • Parts replacement Support October 2017 Troubleshooting the Avaya Pod Fx™ 61 Comments on this document? [email protected]

• Training • Help and policies • Alerts and reports Procedure 1. Go to the Avaya Support home page, www.avaya.com/support Note: Site registration is required to access Self Service Tools and Downloads. Registration instructions are available on the Support home page. 2. From the Support home page: • Enter a keyword in the Avaya Support search dialog box. • Click any of the topic-based links on the page. Related links Resources on page 60 Documentation reference The following provides a high-level list of links to the Avaya support website and third-party websites. Go to the following websites to obtain documentation for the components supported on Avaya Pod Fx. • Avaya: support.avaya.com–Contains documentation and downloads for networking, management, and applications for Avaya Pod Fx. • VMware: www.vmware.com • Compute servers: - HPE ProLiant DL360 Generation 9 servers: www.hp.com - HPE ProLiant DL360p and DL360e Generation 8 servers: www.hp.com - Lenovo ThinkServer RD540 and RD340: www.lenovo.com • EMC² VNX storage array: www.emc.com • HPE Nimble storage array: www.nimblestorage.com • ServerTech Power Distribution Units: www.servertech.com Resources October 2017 Troubleshooting the Avaya Pod Fx™ 62 Comments on this document? [email protected]

document: • Avaya Pod Fx Documentation Reference (NN47204–113) Related links Resources on page 60 Training Product training is available on the Avaya Learning website. For more information or to register, see http://avaya-learning.com. Related links Resources on page 60 Avaya Mentor videos Avaya Mentor videos are available on an Avaya-run Internet channel dedicated to technical content. Playlist categories include: • Unified Communications (tested on Internet Explorer and Firefox). • Contact Centers. • Networking. • Small and Midsize Business. • Uploaded videos. A composite of all available Avaya Mentor videos. Before you begin You must have a valid Internet browser installed and working on your device. About this task The Avaya Mentor videos include the following content categories: • How to install Avaya products. • How to configure Avaya products. • How to troubleshoot Avaya products. Procedure To go to Avaya Mentor videos, click the following link: http://www.youtube.com/avayamentor Training October 2017 Troubleshooting the Avaya Pod Fx™ 63 Comments on this document? [email protected]

• Enter a key word or words in the Search channel dialog box to search for a specific product or topic. • Click the name of a playlist to scroll through the available videos. Related links Resources on page 60 Searching a documentation collection On the Avaya Support website, you can download the documentation library for a specific product and software release to perform searches across an entire document collection. For example, you can perform a single, simultaneous search across the collection to quickly find all occurrences of a particular feature. Use this procedure to perform an index search of your documentation collection. Before you begin • Download the documentation collection zip file to your local computer. • You must have Adobe Acrobat or Adobe Reader installed on your computer. Procedure 1. Extract the document collection zip file into a folder. 2. Navigate to the folder that contains the extracted files and open the file named .pdx. 3. In the Search dialog box, select the option In the index named .pdx. 4. Enter a search word or phrase. 5. Select any of the following to narrow your search: • Whole Words Only • Case-Sensitive • Include Bookmarks • Include Comments 6. Click Search. The search results show the number of documents and instances found. You can sort the search results by Relevance Ranking, Date Modified, Filename, or Location. The default is Relevance Ranking. Related links Resources on page 60 Resources October 2017 Troubleshooting the Avaya Pod Fx™ 64 Comments on this document? [email protected]

Subscribe to e-notifications to receive an email notification when documents are added to or changed on the Avaya Support website. About this task You can subscribe to different types of general notifications, for example, Product Correction Notices (PCN), which apply to any product or a specific product. You can also subscribe to specific types of documentation for a specific product, for example, Application & Technical Notes for Virtual Services Platform 7000. Procedure 1. In an Internet browser, go to https://support.avaya.com. 2. Type your username and password, and then click Login. 3. Under My Information, select SSO login Profile. 4. Click E-NOTIFICATIONS. 5. In the GENERAL NOTIFICATIONS area, select the required documentation types, and then click UPDATE. 6. Click OK. 7. In the PRODUCT NOTIFICATIONS area, click Add More Products. Subscribing to e-notifications October 2017 Troubleshooting the Avaya Pod Fx™ 65 Comments on this document? [email protected]

9. Select a release version. 10. Select the check box next to the required documentation types. 11. Click Submit. Resources October 2017 Troubleshooting the Avaya Pod Fx™ 66 Comments on this document? [email protected]