hit counter script
HP NC570C User Manual
  1. Manuals
  2. Brands
  3. HP Manuals
  4. PCI Card
  5. NC570C
  6. User manual

HP NC570C User Manual

Dual-port 4x fabric adapter
Hide thumbs Also See for NC570C:
Table of Contents

Advertisement

Quick Links

Download this manual
HP Dual-port 4x Fabric Adapter
User Guide
November 2004 (Second Edition)
Part Number 377704-002

Advertisement

Table of Contents
loading

Related Manuals for HP NC570C

Summary of Contents for HP NC570C

  • Page 1 HP Dual-port 4x Fabric Adapter User Guide November 2004 (Second Edition) Part Number 377704-002...
  • Page 2 The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

  • Page 3: Table Of Contents

    Grounding Methods To Prevent Electrostatic Damage .............. viii Contact Information ..........................viii 1: About the Host Channel Adapter (HCA) ....1 HP Dual-port 4x Fabric Adapters ........................1 Supported Protocols ........................1 HCA Package Contents ........................2 About the HCA Drivers ..........................2 IPoIB ...............................
  • Page 4 Installing Two HCAs in One Host with PCI-X Connectors ............10 Installing HCA(s) in a 1U Host with PCI-Express Connectors............ 11 Installing HCA(s) in a 2U+ Host with PCI-Express Connectors..........12 Connecting the InfiniBand Cables ......................12 3: Installing the HCA Drivers........15 About the Installation..........................15 Installing HCA Host Drivers ........................15 Verify the HCA and Driver Installation.....................18...
  • Page 5 Overview............................38 Sample Topology .......................... 38 Configure the Application Server ....................39 Configure the Database Server ..................... 39 Set Up Non-IB Connections ......................40 Troubleshoot the Configuration....................40 7: Configuring SRP Drivers.......... 41 Auto-Mount SRP Devices .........................41 Verify Configurations from the Host......................41 Verify the SCSI Devices from the Host..................
  • Page 6 About IPoIB ..........................69 Configuring IPoIB ........................69 IPoIB Performance vs Ethernet Using netperf...................70 Perform a Throughput Test......................70 Perform a Latency Test......................... 71 SDP Performance vs IPoIB Using netperf....................71 About SDP ............................ 71 Configuring SDP........................... 71 Perform a Throughput Test......................72 Perform a Latency Test.........................

  • Page 7: Regulatory Model Number

    Regulatory Notices Regulatory Model Number For the purpose of regulatory compliance certifications and identification, this product has been assigned a unique regulatory model number. The regulatory model number can be found on the product nameplate label, along with all required approval markings and information. When requesting compliance information for this product, always refer to this regulatory model number.

  • Page 8: Declaration Of Conformity For Products Marked With The Fcc Logo, United States Only

    Declaration of Conformity for Products marked with the FCC Logo, United States Only This device complies with Part 15 of the FCC Rules. Operation is subject to the following two conditions: (1) this device may not cause harmful interference, and (2) this device must accept any interference received, including interference that may cause undesired operation.

  • Page 9: Japanese Notice

    •EMC Directive 89/336/EEC Compliance with these directives implies conformity to applicable harmonized Euro- pean standards (European Norms) which are listed on the EU Declaration of Confor- mity issued by Hewlett-Packard for this product or product family. This compliance is indicated by the following conformity marking placed on the product: xxxx This marking is valid for non-Telecom products...

  • Page 10: Electrostatic Discharge

    For more information on static electricity, or assistance with product installation, contact your authorized reseller. Contact Information Table 2-1: Customer Contact Information For the name of your nearest authorized In the United States, call 1-800-345-1518. HP reseller: In Canada, call 1-800-263-5868.
  • Page 11 Table 2-1: Customer Contact Information For HP technical support: In the United States and Canada, call 1-800-HP-INVENT (1-800-474-6836). This service is available 24 hours a day, 7 days a week. For continuous quality improvement, calls may be recorded or monitored.

  • Page 13: 1: About The Host Channel Adapter (Hca)

    10Gbps connections per port in each direction. Each HCA and associated protocol drivers are designed to run in conjunction with an HP Dual-port 4x Fabric Adapter. The HP Dual-port 4x Fabric Adapters feature a full suite of upper-layer protocols and APIs.

  • Page 14: Hca Package Contents

    • Limited Warranty and Material Limitations Documentation About the HCA Drivers The HP Dual-port 4x Fabric Adapters provide a full suite of upper-layer protocols, including IPoIB, SDP, SRP, MPI and uDAPL. IPoIB IPoIB is a required protocol; it allows the IP network to utilize the InfiniBand fabric. It is used by SDP and uDAPL to resolve IP addresses.

  • Page 15: Linux Kernels

    • Bug fixes have been provided for the purpose of improving stability Linux Kernels Check the HP Support website at: http://support.hp.com/ website for the latest list of supported kernels and system architectures. About Boot Over InfiniBand Functionality The Host Channel Adapter has the capability of running bootable firmware, which allows you to use Boot Over InfiniBand functionality.

  • Page 17: 2: Installing The Host Channel Adapter (Hca)

    Installing the Host Channel Adapter (HCA) This chapter provides the following information: • “Requirements” on page 5 • “Installation Overview” on page 5 • “Selecting the Host Connector” on page 6 • “Selecting the Type of Installation” on page 8 Requirements •...

  • Page 18: Selecting The Host Connector

    • “Installing HCA Host Drivers” on page 15 Selecting the Host Connector The following types of connectors are supported: • “Selecting PCI-X Connector(s)” on page 6 • “Selecting PCI-Express Connector(s)” on page 7 Selecting PCI-X Connector(s) The HCA requires that specific PCI-X slots be used. When determining which PCI-X slot to use, inspect the server chassis and keep the following in mind: Consider the Speed of the Slot Locate the 133MHz PCI-X (64-bit, 3.3V) or 100MHz PCI-X (64-bit, 3.3V) slots.

  • Page 19: Selecting Pci-Express Connector(S)

    Consider Dual HCA Installation Requirements • For dual HCA installation in a single host, it is required to have two completely isolated PCI-X buses to avoid any performance degradation. • If the host has only one PCI-X 100 or 133MHz bus (regardless of the number of connectors), then this mother board should not be used for a dual HCA installation.

  • Page 20: Selecting The Type Of Installation

    • Installation of this HCA should be performed by individuals who are both qualified in the servicing of computer equipment, and trained in the hazards associated with products capable of producing hazardous energy levels. • To reduce the risk of personal injury from hot surfaces, allow the internal system components to cool before touching.

  • Page 21: Installing A Low Profile Hca In A Pci-X Connector

    (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step Slip the IB ports into the back of the open slot. Slide the edge-connector of the HCA into the PCI-X slot.

  • Page 22: Installing Two Hcas In One Host With Pci-X Connectors

    (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step Slip the IB ports into the back of the open slot. Slide the edge-connector of the HCA into the PCI-X slot.

  • Page 23: Installing Hca(S) In A 1U Host With Pci-Express Connectors

    (Optional) If it is not necessary for you to remove the riser from the server, slide the HCA edge-connector into the PCI-X slot now. If you need to remove the riser, refer to Step Slip the IB ports into the back of the open slot. Slide the edge-connector of the HCA into the PCI-X slot.

  • Page 24: Installing Hca(S) In A 2U+ Host With Pci-Express Connectors

    12. Snap the fastener on the host closed. 13. Make sure that the HCA installation is secure before connecting any InfiniBand cables. 14. Reinstall the host system cover. Installing HCA(s) in a 2U+ Host with PCI-Express Connectors The HCA comes preconfigured. You do not have to set any jumpers or connectors. To install the HCA: Note the Global Unique ID (GUID) numbers from the hardware.
  • Page 25 Plug InfiniBand cables from the host to the InfiniBand switch. To plug in an InfiniBand cable, push the connector into the interface until you hear/feel a click. Figure 2-2: Fully Installed IB Cable with Pinch Connector...
  • Page 26 NOTE: If your host does not provide an ample amount of free space around a given IB port, double-check that your IB cable connector engages fully. Wiggle your connector back and forth to be sure that both sides of the connector have locked firmly into place. To remove a cable with a pinch connector, pinch both sides of the back of the connector and pull the connector away from the port.

  • Page 27: 3: Installing The Hca Drivers

    “Configuring SRP Drivers” on page 41 • “Configuring uDAPL Drivers” on page 59 Installing HCA Host Drivers To install HCA software: Go to http://support.hp.com/ Select “Software & Driver downloads.” On the Software & Driver Downloads page, enter your product name, then click the double arrow.
  • Page 28 Install the software. Unzip the tar file containing the software using gunzip. Extract the software into a local directory using tar. Change to the local directory. In a terminal window, execute the command ./tsinstall to install the host drivers. This script automatically detects the available kernel and installs the appropriate RPM packages.
  • Page 29 You must reboot the host before using InfiniBand if either of the following scenarios occurred: • the firmware was upgraded • you uninstalled, then re-installed the firmware (Optional) Verify the installation. Example [root@elrond]# rpm -qa | grep topspin topspin-ib-rh9-1.1.3-687 topspin-ib-mod-rh9-2.4.20-8smp-1.1.3-687 [root@elrond]# Refer to the HP website http://support.hp.com/ for driver updates.

  • Page 30: Verify The Hca And Driver Installation

    The status should be PORT_ACTIVE. If the status is PORT_INITIALIZE, wait a few seconds and check again. Note the hw_ver (i.e., hardware version) and fw_ver (i.e., firmware version) fields. Check with HP Customer Support to determine the appropriate hardware and firmware versions for your HCA. [root@gandalf]# /usr/local/topspin/bin/vstat...

  • Page 31: Verify The Hca And Server Communication

    Example B: The following example shows one HCA port is connected to the InfiniBand fabric: [root@gandalf]# /usr/local/topspin/bin/vstat 1 HCA found: hca_id=InfiniHost0 vendor_id=0x02C9 part_id=0x5A44 hw_ver=0xA1 fw_ver=0x200000000 num_phys_ports=2 port=1 port_state=PORT_ACTIVE sm_lid=0x0001 port_lid=0x02b9 port_lmc=0x00 max_mtu=2048 gid_tbl_len=32 GID[ 0]= fe:80:00:00:00:00:00:00:00:05:ad:00:00:00:16:70 port=2 port_state=PORT_DOWN sm_lid=0x0000 port_lid=0x02ba port_lmc=0x00 max_mtu=2048 gid_tbl_len=32...

  • Page 32: Verify The Hca Initialization

    3. Check the modules running on the HCA server by using the lsmod command. Look for modules like ts_udapl, ts_sdp, ts_ipoib, ts_ib_sa_client, etc. Example [root@enclus2 root]# lsmod Module Size Used by Tainted: P ts_srp_host 70936 ts_ib_dm_client 22780 [ts_srp_host] ts_ib_useraccess 13252 (autoclean) (unused) ts_sdp 152376...

  • Page 33: Determine The Card Type

    Note: If you have a Boot Over IB license agreement, any HCA can be upgraded to become a bootable HCA. Determine the Card Type Determine card hardware version by entering: [root@test root]#/usr/local/topspin/sbin/tvflash –i • The card type will be Jaguar (older), Cougar, Cougar Cub. •...

  • Page 35: 4: Configuring Ipoib Drivers

    Configuring IPoIB Drivers IPoIB must be installed before it can be configured. Refer to “Installing the HCA Drivers” on page • “Assign Interfaces to HCAs” on page 23 • “Create Interface Partitions” on page 25 • “Run an IPoIB Performance Test” on page 27 Assign Interfaces to HCAs About Assigning Interface for Single HCAs When you are installing a single HCA in a server, the possible interfaces for the HCA will be ib0 and...

  • Page 36: View All The Interfaces

    • mask is the netmask for the IP address. Example of Single HCA [root@test root]# /usr/local/topspin/sbin/ [root@test root]# ifconfig ib0 192.168.0.0 netmask 255.255.255.0 [root@test root]# ifconfig ib1 192.168.0.1 netmask 255.255.255.0 Example of Two HCAs [root@test root]# /usr/local/topspin/sbin/ [root@test root]# ifconfig ib0 192.168.0.0 netmask 255.255.255.0 [root@test root]# ifconfig ib1 192.168.0.1 netmask 255.255.255.0 [root@test root]# ifconfig ib2 192.168.0.2 netmask 255.255.255.0 [root@test root]# ifconfig ib3 192.168.0.3 netmask 255.255.255.0...

  • Page 37: Create Interface Partitions

    [root@enclus2 root]# /usr/local/topspin/sbin/ [root@enclus2 root]# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:30:48:29:B9:FA inet addr:10.3.0.11 Bcast:10.3.255.255 Mask:255.255.0.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:1200029 errors:0 dropped:0 overruns:0 frame:0 TX packets:12095 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:99263236 (94.6 Mb) TX bytes:1293346 (1.2 Mb) Interrupt:54 Base address:0x3000 Memory:e8200000-e8220000 eth1...

  • Page 38: Configuring A Subinterface

    HCA • the ports for the InfiniBand server switch Refer to the HP 24-Port 4x Fabric Copper Switch User Guide for information regarding partitions on the IB switch. Locate the ipoibcfg utility through the following path: /usr/local/topspin/sbin 2.

  • Page 39: Deleting An Interface Partition

    Deleting an Interface Partition To delete a subinterface: 1. Enter the ipoibcfg del command, the parent interface from which you want to delete the subinterface, and the partition value that has been created on the InfiniBand switch: ipoibcfg del <parent interface> <p_key value> Example [root@test root]# /usr/local/topspin/sbin/ -bash: /usr/local/topspin/sbin/: is a directory...

  • Page 41: 5: Configuring Mpi Drivers

    Configuring MPI Drivers MPI must be installed before it can be configured. Refer to “Installing the HCA Drivers” on page • “Configure MPI” on page 29 • “Configure SSH” on page 30 • “Edit PATH Variable” on page 31 • “Perform Bandwidth Test”...

  • Page 42: Configure Ssh

    Configure SSH To configure SSH between two hosts so that a connection does not require a password, perform the following steps: Log in to the host that you want to configure as the local host (hereafter, “host 1”). (The second host serves as the remote host.) Example login: username...

  • Page 43: Edit Path Variable

    Move to the .ssh directory that you created. Example qa-bc1-blade4:~ # cd .ssh qa-bc1-blade4:~/.ssh # Copy the public key to a file. Example qa-bc1-blade4:~/.ssh # cp id_rsa.pub authorized_keys4 Log in to the host that you want to configure as the remote host (hereafter “host 2”). Example login: username Password: password...

  • Page 44: Perform Bandwidth Test

    NOTE: Optionally, you can add the paths for all users by adding export PATH=$PATH:/usr/local/topspin/mpi/mpich/bin:/usr/local/topspin/bin to your /etc/profile.d script. Verify that your compiler and MPI script match. Compilers reside in the /usr/local/topspin/mpi/mpich/bin directory. GNU compilers use mpicc and mpif77 scripts. Intel compilers use mpicc.i and mpif90.i scripts.
  • Page 45 • the -np keyword to specify the number of processes • the number of processes (integer) • the host name of the local host • the host name of the remote host • the mpi_latency command • the number of times to transfer the data (integer) •...

  • Page 47: 6: Configuring Sdp Drivers

    Configuring SDP Drivers SDP must be installed before it can be configured. Refer to “Installing the HCA Drivers” on page • “Configure IPoIB Interfaces” on page • “Specify Connection Overrides” on page 35 • “Convert Sockets-Based Applications” on page 35 •...

  • Page 48: Converting Sockets-Based Applications To Use Sdp

    Converting Sockets-Based Applications to Use There are three ways to convert your sockets-based applications to use SDP instead of TCP, which are described in the table below: Table 6-1: SDP Conversion Information Conversion Method Required Action Type Explicit/ Converts sockets to use 1.

  • Page 49: Run A Performance Test On Sdp

    Table 6-1: SDP Conversion Information Conversion Method Required Action Type Automatic Converts socket streams 1. Load the installed library in one of the following libsdp.so based upon destination ways: port, listening port, or • Edit the environment variable. Setting this to LD_PRELOAD program name.

  • Page 50: Sample Configuration - Oraclenet™ Over Sdp For Oracle 9I

    Sample Configuration - OracleNet™ Over SDP for Oracle 9i • “Sample Topology” on page 38 • “Configure the Application Server” on page 39 • “Configure the Database Server” on page 39 • “Set Up Non-IB Connections” on page 40 • “Troubleshoot the Configuration”...

  • Page 51: Configure The Application Server

    Configure the Application Server Set up a Preload Script 1. Set up a preload script in order to load the SDP library for all programs. # echo '/lib/libsdp.so' >> /etc/ld.so.preload Add Configuration Lines to the SDP Initialization Script 2. Add the appropriate configuration lines to the SDP initialization script. This sets the host to listen on port 1521 for SDP connections, and to use SDP for any outbound connections targeted to port 1521 on a remote host.

  • Page 52: Set Up Non-Ib Connections

    Examine Configuration Files 4. View SDP connection information by examining the following two files: /proc/topspin/sdp/conn_data /proc/topspin/sdp/conn_main These files should show you the connections coming over SDP. If there are no SDP connections, these special files will only show header information. If SDP is enabled properly on the server, you should see at least one connection in wait state on the server.

  • Page 53: 7: Configuring Srp Drivers

    Configuring SRP Drivers SRP must be installed before it can be configured. Refer to “Installing the HCA Drivers” on page For more information about SRP, refer to “Socket Direct Protocol (SDP)” on page • “Auto-Mount SRP Devices” on page 41. •...
  • Page 54 Verify SRP Functionality To show the SCSI devices that are currently visible from the SRP host: Example of CX200 # cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST336706LC Rev: 010A Type: Direct-Access ANSI SCSI revision: 03 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: SEAGATE...
  • Page 55 Reload the SRP host driver: Example /etc/init.d/ts_srp restart Rescan the SRP targets: Example /usr/local/topspin/sbin/rescan-scsi-bus.sh Host adapter 0 (aic79xx) found. Host adapter 1 (aic79xx) found. Host adapter 2 (srp) found. Scanning for device 0 0 0 0 ... OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST336607LC Rev: 0006...

  • Page 56: Special Considerations

    Example [root@enclus2 root]# iostat Linux 2.4.21-9.ELsmp avg-cpu: %user %nice %sys %idle 0.02 0.00 0.02 99.96 Device: Blk_read/s Blk_wrtn/s Blk_read Blk_wrtn dev8-0 0.48 0.44 4.49 492912 5028152 dev8-1 0.00 0.00 0.00 dev8-2 0.00 0.00 0.00 [root@enclus2 root]# Observe the results. Kill all dds when verification is complete. Example [root@enclus2 root]# pkill dd [root@enclus2 root]#...

  • Page 57: Srp Sample Configuration

    Follow Step 3 Step 6 to correct the LUN discovery. Example /usr/local/topspin/sbin/rescan-scsi-bus.sh Host adapter 0 (aic79xx) found. Host adapter 1 (aic79xx) found. Host adapter 2 (srp) found. Scanning for device 0 0 0 0 ... OLD: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST336607LC Rev: 0006...

  • Page 58: Viewing The Srp Host

    The service processor B is being used to access one RAID-5 group. RAID-5 group is exposed as LUN Note the following information: The WWN that ends with 12:33:0F:D8:11 The LUN 2 Figure 7-1: Viewing the Storage Configuration with Navisphere Viewing the SRP Host The InfiniBand driver on the SRP host system should be installed by using the procedure provided in “Installing HCA Host Drivers”...

  • Page 59: View The Topology

    Example # /usr/local/topspin/bin/vstat 1 HCA found: hca_id=InfiniHost0 vendor_id=0x02C9 part_id=0x5A44 hw_ver=0xA1 fw_ver=0x300000002 num_phys_ports=2 port=1 port_state=PORT_ACTIVE sm_lid=0x0007 port_lid=0x000f port_lmc=0x00 max_mtu=2048 gid_tbl_len=32 00:05:ad:00:00:01:29:81 GID[ 0]= fe:80:00:00:00:00:00:00: port=2 port_state=PORT_DOWN sm_lid=0x0000 port_lid=0x0002 port_lmc=0x00 max_mtu=2048 gid_tbl_len=32 GID[ 0]= fe:80:00:00:00:00:00:00:00:05:ad:00:00:01:29:82 View the Topology In this example, the SRP host is connected to port 3 on the InfiniBand switch card of the Topspin 360. Use the Element Manager’s Topology view to display the physical topology.

  • Page 60: Configure The Fibre Channel Gateway

    The InfiniBand Topology appears. Figure 7-2: Element Manager InfiniBand -> Topology View Configure the Fibre Channel Gateway The Fibre Channel gateway used in this example is the one in slot 11, although this particular Topspin 360 has several gateways installed. The example shown here is for reference purposes and to help make sense of the SRP host configuration procedure.
  • Page 61 View the InfiniBand chassis with Element Manager. Slot 11 Figure 7-3: Element Manager View of Topspin 360 Double-click the Fibre Channel gateway. The Fibre Channel Port Properties window appears. The figure below shows the fibre channel port properties. The fibre channel port is directly linked to the CX200 storage, with no intermediate fibre channel switches.
  • Page 62 View general information about the SRP host swclus6 with Element Manager. Select Fibre Channel -> Storage Manager. Click open the SRP Hosts folder from the left navigation bar. Click on the swclus6 host. View the General tab. Figure 7-5: Element Manager SRP Host Information View information for the SRP targets.

  • Page 63: Verify Configurations From The Host

    The following SRP targets are visible through port of the fibre channel gateway in slot 11. Figure 7-6: Element Manager - SRP Targets View Verify Configurations from the Host Once you have configured your storage and the Fibre Channel Gateway, verify the gateway and the storage configuration from the host.

  • Page 64: Verify Srp Functionality

    Verify SRP Functionality To show the SCSI devices that are currently visible from the SRP host: Example of CX200 # cat /proc/scsi/scsi Attached devices: Host: scsi0 Channel: 00 Id: 00 Lun: 00 Vendor: SEAGATE Model: ST336706LC Rev: 010A Type: Direct-Access ANSI SCSI revision: 03 Host: scsi0 Channel: 00 Id: 01 Lun: 00 Vendor: SEAGATE...
  • Page 65 Verify the SCSI HCA Driver Information The following examples show verification of an EMC CX200 configuration from the SRP host. To verify the SCSI HCA driver instance information that is associated with the SRP driver: Example of CX200 # cat /proc/scsi/srp/2 Topspin SRP Driver Index Service...

  • Page 66: Configure The Srp Target

    Example # iostat Observe the results. Kill all dds when verification is complete. Example # pkill dd Configure the SRP Target The following example shows a Logical Volume Manager (LVM) configuration of the SRP target Wipe out the current partition table and re-read. Example root@swclus6 root]# dd if=/dev/zero of=/dev/sde bs=1k count=1 1+0 records in...
  • Page 67 Create the volume group Example [root@swclus6 root]# vgcreate cx200_vg_000 /dev/sde vgcreate -- INFO: using default physical extent size 4 MB vgcreate -- INFO: maximum logical volume size is 255.99 Gigabyte vgcreate -- doing automatic backup of volume group “cx200_vg_000” vgcreate -- volume group “cx200_vg_000” successfully created and activated [root@swclus6 root]# vgdisplay --- Volume group --- VG Name...
  • Page 68 View performance results taken during the mkfs Example IO pattern during the mkfs: 09:27:49 PM rtps wtps bread/s bwrtn/s 09:27:50 PM 0.00 0.00 0.00 0.00 0.00 09:27:51 PM 0.00 0.00 0.00 0.00 0.00 09:27:52 PM 5.05 5.05 0.00 40.40 0.00 09:27:53 PM 160.61 13.13...
  • Page 69 Mount the file system Example [root@swclus6 root]# mount /dev/cx200_vg_000/swbld_lv /swbld Verify that the configuration is still working. Example Full fsck: [root@swclus6 /]# umount /swbld/ [root@swclus6 /]# fsck -f /dev/cx200_vg_000/swbld_lv fsck 1.32 (09-Nov-2002) e2fsck 1.32 (09-Nov-2002) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts...
  • Page 70 View the actual SRP host configuration Click into the swclus6 host in the left navigation bar. Click the LUN Access tab. Figure 7-7: Element Manager - Storage Manager View Click onto one of the LUNs in the Accessible LUNs window. Click the Edit ITL Properties button.

  • Page 71: 8: Configuring Udapl Drivers

    Configuring uDAPL Drivers The uDAPL drivers must be installed before they can be configured. Refer to “Installing the HCA Drivers” on page About the uDAPL Configuration The User Direct Access Programming Library (uDAPL) protocol is transparently installed and requires no further configuration. However, your application may require configuration for uDAPL. In addition, you may want to run the Performance and Latency tests that are provided with the RPMs.

  • Page 72: Run A Udapl Performance Test

    Run a uDAPL Performance Test The utility to test uDAPL performance is included with the RPMs after the host-side drivers are installed. The uDAPL test utility is located in the following directory: /usr/local/topspin/bin/ The uDAPL test must be run on a server and a client host. Run a uDAPL Throughput Test The Throughput test measures RDMA WRITE throughput using uDAPL.

  • Page 73: Run A Udapl Latency Test

    Run a uDAPL Latency Test The uDAPL Latency test measures the half of round-trip latency for uDAPL sends. Start the Latency test on the server host. Syntax for Server: /usr/local/topspin/bin/lat_server.x <device_name> <RDMA size> <iterations> <batch size> Example [root@cdrom]# /usr/local/topspin/bin/lat_server.x ib0 150000 1 1 •...

  • Page 75: 9: Troubleshooting The Hca Installation

    Troubleshooting the HCA Installation The following are a list of things you can check if the HCA does not operate appropriately. • “Interpret HCA LEDs” on page 63 • “Check the InfiniBand Cable” on page 64 • “Check the InfiniBand Network Interfaces” on page 64 •...

  • Page 76: Check The Infiniband Cable

    Make sure an InfiniBand cable is connected to a port on the HCA and a port on the InfiniBand switch card. HP recommends that you tug slightly on the cable to verify that is tightly connected as poorly connected InfiniBand cables can cause errors that are difficult to detect.

  • Page 77: Run The Hca Self-Test

    • addr is the IP address to assign the network interface. • netmask is a mandatory keyword. • mask is the netmask for the IP address. Run the HCA Self-Test The HCA Self-test verifies the state of the HCA component, the state of each port on the HCA, as well as the connectivity to the fabric.
  • Page 78 Example [root@1750]# /usr/local/topspin/sbin/hca_self_test ---- Performing InfiniBand HCA Self Test ---- Number of HCAs Detected ....1 PCI Device Check ....... PASS Host Driver Version ....rhel3-2.4.21-4.ELsmp-2.0.0-530 Host Driver RPM Check ....PASS HCA Type of HCA #0 ..... Cougar HCA Firmware on HCA #0 ....v3.01.0000 HCA Firmware Check on HCA #0 ...

  • Page 79: 10: Sample Test Plan

    Sample Test Plan The following evaluation test plan will walk you through basic setup of the InfiniBand-based switching fabric, introduce you to some of the ULPs (Upper Layer Protocols) supported on the fabric, and perform some basic tests that showcase the fabric's performance. Overview •...

  • Page 80: Network Topology

    low latency aspects of the fabric, a minimum of dual Xeon servers (in the neighborhood of 2.0 GHz) with 133Mhz PCI-X expansion busses are required. • An Ethernet switch should be used to network the two servers together. This switch can be of any speed, but a gigabit version will provide the best platform for comparing high performance communication over Ethernet and InfiniBand.

  • Page 81: Ipoib Setup

    IPoIB Setup About IPoIB IPoIB (IP over InfiniBand) is simply that: IP packets running over the InfiniBand fabric. This protocol is useful for testing connectivity into the fabric between two hosts, and also for taking advantage of the high speed fabric for “legacy” applications that are written to communicate over IP. The drawback to using IPoIB compared to standard Gigabit Ethernet is that the operating system must do packet checksumming, whereas modern NIC cards usually offload this function to hardware.

  • Page 82: Ipoib Performance Vs Ethernet Using Netperf

    IPoIB Performance vs Ethernet Using netperf To test the performance characteristics of IPoIB, use a tool called netperf. This utility runs on both machines with one machine listening on a TCP socket and the other connecting and sending test data. The listening program is called netserver while the test client is called netperf.

  • Page 83: Perform A Latency Test

    Perform a Latency Test To demonstrate the latency advantage of InfiniBand compared to Ethernet, use a netperf test called TCP request/response. This test will send a 1 byte request to the remote machine and the remote machine will issue a 1 byte response. Develop a base case for comparison on Server 1.

  • Page 84: Perform A Throughput Test

    10. Make sure processes include the SDP library when they load. The /etc/ld.so.preload file tells the system's dynamic linker to load the SDP library when processes are started. Create the /etc/ld.so.preload file if the file does not exist. Add the following line to /etc/ld.so.preload on both systems: /lib/libsdp_sys.so 11.
  • Page 85 Symbols file system .............55 firmware /etc/modules.conf ..........45 upgrade ............20 Numerics 133 MHz PCI-X GID ..............18, required speed ...........5 grounding methods to prevent electrostatic damage . viii grub ...............45 GUID ..............18, AF_INET_SDP .............36 alloc ...............55 hardware version ...........21 HCA initialization ..........20 Boot Over IB ............21 HCA self-test...
  • Page 86 Infiniband ............63 remove IB cables ..........14 LILO ..............45 required speed ............5 list of supported kernels ..........3 requirements list of supported protocols ........1 dual HCA install ..........7 Logical Volume Manager ........54 rescan SRP targets ..........43 low profile installation ........9, 11, restart ..............43 lsmod ..............20...
  • Page 87 IPoIB ...............70 ..............72 thru_server.x ............60 tsinstall ..............16 tvflash ..............21 uDAPL about ............2, application configuation .........59 sample make files ..........59 performance test ..........67 upgrading firmware ..........20 upper layer protocols performance test ..........67 verify modules lsmod ...............20 vgscan ..............54 vstat ..............18,...

This manual is also suitable for:

Nc571c

Table of Contents