ThunderboltDevice Driver Programming Guide by Apple

Thunderbolt Device
Driver Programming
Guide
Contents
About the Thunderbolt Technology 4

At a Glance 4
Prerequisites 4
See Also 5
Reference Material 5
Thunderbolt Technology Overview 6

The Beginning: The Thunderbolt Controller Chip 6
Using Thunderbolt Communication over Ports 8
Working with Thunderbolt Technology 10

Discovering Thunderbolt Devices 10
Using PCI Resources Wisely 10
Tolerating PCI Latency 11
Using a Registered Vendor ID 11
Enabling PCI Device Drivers for Thunderbolt Operation 11
Verifying Thunderbolt Connections in PCI Devices 13
Handling and Routing Interrupts 14

Building Only MSI-Capable Thunderbolt Devices 14
Handling Hardware Exceptions for MSI Support 15
Enabling MSI 16
Using Hot Plug Operation with PCI Devices 17
Supporting PCIe Pause 19
Returning I/O Operations as Errors 23
Debugging Thunderbolt Drivers 24

Debugging VT-d I/O MMU Virtualization 24
Enabling VT-d Panics 25
Disabling VT-d 26
Debugging PCIe Pause 26
Avoiding Memory Leaks 26
Document Revision History 28
2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.
Figures and Listings
Thunderbolt Technology Overview 6

Figure 1-1
Figure 1-2
Figure 1-3
Figure 1-4
Figure 1-5
Intel 82524EF controller chip (host mode) 6

Intel 82524EF controller chip (endpoint mode) 7
Expansion chassis utilizing PCI paths 8
PCI fanout (two disk controllers) 9
PCI direct routing (two disk controllers) 9
Working with Thunderbolt Technology 10

Listing 2-1
Listing 2-2
Listing 2-3
Listing 2-4
PCI Driver Key (Thunderbolt Operation) 12

PCI Driver Personality (Thunderbolt Key Set) 12
Verifying Thunderbolt Connections 13
PCI Driver: External Storage Key Set 13
Handling and Routing Interrupts 14

Listing 3-1
Listing 3-2
Enabling MSIs 16
Single Bottleneck Routine (MMIO Read/Write) 18
Debugging Thunderbolt Drivers 24

Listing 4-1
Searching for Memory Leaks 27
About the Thunderbolt Technology
The Thunderbolt interface is a new I/O technology that supports high-resolution displays and high-performance
data devices through a single expansion interface.
At a Glance
The Thunderbolt interface is made up of an Intel controller chip designed to tunnel DisplayPort (DP) version
1.1 and PCI Express (PCIe) information. Its dual-link cables are designed to carry 10 gigabits per second (Gbps)
of data bidirectionally on each link, for a total of 40 Gbps per cable. Thunderbolt ports may be hot-plugged,
daisy chained with up to six devices in depth, and can be connected host-to-host.
Thunderbolt ports use a Mini DisplayPort (mDP) connector and provide 10 W of power to downstream devices.
Thunderbolt cables may be connected by either end (both ends are exactly the same). Thunderbolt technology
is a host-centered and host-managed system (like USB), but also allows for host-to-host connections.
A Thunderbolt port or cable provides two 10 Gbps bidirectional links, but these two links cannot be bonded
into a single channel. Host software must assign specific paths (for example, DP, PCIe, native) to each link to
balance the load. This functionality is included in EFI and in OS X. For example, if there are two Thunderbolt
enabled displays attached, it can choose to route the DisplayPort (DP) over each link and use the remaining
bandwidth for PCI. If there is only one Thunderbolt display, software can route the DisplayPort traffic over the
optimal link. The Thunderbolt interface has no requirement for fixed routing.
It is possible to boot directly from Thunderbolt devices. Apple provides a Unified Target Disk Mode (UTDM),
which supports FireWire and Thunderbolt connections. Users may boot from UTDM over FireWire or Thunderbolt
connections as well.
Prerequisites
Drivers must be built as Universal and be able to support systems with greater than 2 GB of RAM. Refer to the
section entitled Kernel Extensions and Drivers in the 64-Bit Transition Guide for more details. In particular,
all drivers must use IODMACommand for scatter-gather list support.
About the Thunderbolt Technology

See Also
See Also
Apple provides the following reference material. For more information please refer to these documents.
Reference Material
I/O Kit Fundamentals

This document provides a broad, conceptual description of the I/O Kit and device-driver development on
OS X. This document is useful for both the developer who is creating a device driver that is resident in the
kernel and the developer who is using an I/O Kit device interface to communicate with the hardware.
64-Bit Transition Guide

This document assists you in the transition to Snow Leopard and Lion where the kernel is using a 64-bit
environment on some hardware.
Thunderbolt Technology Overview
This chapter describes the basic components and functions of the Thunderbolt technology and how these are
used with DisplayPort (DP) and PCI Express (PCIe) devices.
The Beginning: The Thunderbolt Controller Chip

The Thunderbolt interface is a revolutionary I/O technology that supports high-resolution displays and
high-performance data devices through a single, compact port. It sets new standards for speed, flexibility, and
simplicity. At the core of this technology is the Intel Thunderbolt controller chip.
For example, the first generation of controller chip is the 82524EF controller as shown in Figure 1-1.
Figure 1-1
Intel 82524EF controller chip (host mode)
The Intel controller chip can be used in both host mode and endpoint mode. In host mode, the controller has
a Gen2 x4 uplink to the system PCI Express Root Complex and one or more DisplayPort (DP) inputs (depending
on the graphics capabilities of the system). Additionally, there is a PCI switch and a collection of DMA engines,
referred to as the Native Host Interface (NHI). The PCI switch enables PCI uplink for downstream devices and
the NHI is used for software protocols and device discovery. The controller chip includes four Thunderbolt
ports as outputs.

The Beginning: The Thunderbolt Controller Chip
Note: Each open circle in the diagram indicates an adapter, that translates the native transport (NHI,
PCI, DP) for transmission over Thunderbolt ports and cables.
In endpoint mode, the controller chip provides a Gen2 x4 downlink (supports x1, x2, x4 at Gen2 speeds) or
Gen1 4x1 downlinks to support multiple devices. Figure 1-2 provides an example of the controller chip as an
endpoint with both a x4 Gen2 PCI and a DisplayPort (DP) output option.
Figure 1-2
Intel 82524EF controller chip (endpoint mode)

Using Thunderbolt Communication over Ports
Note: The PCI switch gets its PCI input from the upstream Thunderbolt connection through the
Thunderbolt switch. Similarly, the DisplayPort (DP) out must be connected through the Thunderbolt
topology to a DP in adapter.

Communication over Thunderbolt ports is achieved by establishing paths between two adapters. A practical
example of a PCI expansion chassis as shown in Figure 1-3.
Figure 1-3
Expansion chassis utilizing PCI paths
In Figure 1-3, a PCI path has been established by routing the PCI upstream connection from the PCI switch in
the Mac host through the Thunderbolt switch and across the cable to the Thunderbolt switch in the PCI chassis.
From the Thunderbolt switch inside the PCI chassis, the path is routed up to the PCI switch in the controller
chip. The PCI fanout from the switch provides 4x1 connectivity.

PCI paths can utilize fanout or can connect directly to PCI adapters in the Mac host. The host software controls
how the paths are established. Wherever possible, OS X and EFI attempt to use the same algorithms for path
selection to provide a similar user experience. Devices can also provide hints to software, indicating how much
and the type of bandwidth the device consumes in order for the software to make a more optimal selection.
An example of fanout is shown in Figure 1-4.
Figure 1-4
PCI fanout (two disk controllers)
The disk labeled Disk #1 has a PCI switch inside the controller chip used for fanout. In Figure 1-5, the host
software can connect Disk #2 directly to the host, instead of using fanout.
Figure 1-5
PCI direct routing (two disk controllers)
Only four PCI adapters are available through the Mac host. The host determines the optimal routes for all of
the attached devices.
Working with Thunderbolt Technology
This chapter provides guidelines for working with the Thunderbolt technology. Keep these guidelines in mind
when creating compatible devices.
Discovering Thunderbolt Devices

System software is used to discover Thunderbolt devices and to determine when a new Thunderbolt device
has been attached. The software discovers all DisplayPort and PCI devices, and creates Thunderbolt paths to
those devices using any available resources. Once the paths are created, the applicable OS driver(s) are notified
to the presence of the devices.
In the case of DisplayPort, graphics drivers may light up the screen. The screen is only lit up based on the
capabilities of the GPU.
In the case of PCI, the I/O PCI family (IOPCIFamily) within the I/O Kit is responsible for enumerating the PCI
bridge in the controller chip and any downstream bridges or devices.
Using PCI Resources Wisely

PCI resources (bus numbers, memory apertures) are scarce and programmers and device manufacturers should
do their best to present the true requirements of the device.
For example, consider a PCI device that has multiple functions, where only function 0 will be used in the
application. PCI functions other than function 0 should be hidden from the OS if possible, by returning
0xFFFFFFFF for a Configuration Read at offset 0 for that function. Another example is a device that requires
only 2 KB of aperture for Memory Mapped I/O (MMIO) space, but instead reports 64 KB of aperture during
enumeration. These devices should report accurate aperture values.
I/O space cannot be counted on when designing for the Thunderbolt interface. PCI limitations require an I/O
space allocation of 4 KB per PCI Bridge, and the total system-wide I/O space is only 64 KB. Therefore, use MMIO
instead of relying on I/O space allocations for devices. Using MMIO allows the operating system to load the
device driver, even if I/O space could not be properly allocated for the device.
10

Tolerating PCI Latency
Tolerating PCI Latency

Devices on a Thunderbolt bus exhibit higher latency than devices on internal slotsabout 1.5 microseconds
of round-trip latency per hop. This means that putting a device at the end of a Thunderbolt chain can add up
to 9 microseconds of round-trip latency. Most devices and drivers handle this additional communication latency
with no particular difficulty. In some cases, however, your driver or device firmware must be tweaked to tolerate
that latency.
For example:
If your driver uses timers to detect device failures, you may need to lengthen the timeout value.
If your driver needs to perform an operation the moment an interrupt occurs, you may have to find another
way to schedule the operation, such as telling the device to delay the action until the next interrupt or
using a software-defined phase-locked loop.
If your device expects the driver to send some data and then wait for the device to acknowledge it before
sending more data, then the device needs to be more tolerant of delays than it otherwise might need to
be, or else the driver needs to handle failure gracefully.
If your driver tells an isochronous endpoint to stop sending data, it may continue to send data longer than
usual.
And so on. The details are highly dependent on the driver.

Note: Interrupt latency on Thunderbolt can be slightly more problematic if your device does not
use MSI-based interrupts. Because legacy PCI interrupts are shared bus-wide, other devices can cause
serious performance problems. For this reason, Thunderbolt devices should always use MSI-based
interrupts.
Using a Registered Vendor ID

Each Thunderbolt device must have a Device ROM (DROM) that contains information about the vendor and
the device, as well as a 64-bit UID, which uniquely identifies the device. Each vendor must have a registered
vendor ID with the Thunderbolt naming authority (Intel).
Enabling PCI Device Drivers for Thunderbolt Operation

To declare support for Thunderbolt technology, the following key must be added to each of the personalities
in your drivers Info.plist file.
11

Enabling PCI Device Drivers for Thunderbolt Operation
Listing 2-1
PCI Driver Key (Thunderbolt Operation)
<key>IOPCITunnelCompatible</key>
<true/>
You must include the key in the IOKitPersonalities section of the Info.plist file or the IOPCIFamily will
not load the drivers for Thunderbolt connected PCI devices. This opt-in key protects consumers against older
drivers that have not yet been updated to support Thunderbolt technology. This method also makes it easy
for third party developers, who have properly updated their driver, to signify they have done so. (See example
driver Listing 2-2 (page 12).)
Note: A PCI device driver must not ship with this key set, unless it fully supports the requirements
set forth in this document. This opt-in key is specific to a personality for the driver, but the personality
can contain additional information in the form of key/value pairs in order to provide additional
context to the device driver should the IOPCITunnelCompatible key not be enough.
An example driver that includes this key in one of its personalities is AppleAHCIPort.kext, which is shown
in the PCI Driver Personality (Thunderbolt Key Set).
Listing 2-2
PCI Driver Personality (Thunderbolt Key Set)
<key>GenericAHCI</key>
<dict>
<key>CFBundleIdentifier</key>
<string>com.apple.driver.AppleAHCIPort</string>
<key>Chipset Name</key>
<string>AHCI Standard Controller</string>
<key>IOClass</key>
<string>AppleAHCI</string>
<key>IOPCIClassMatch</key>
<string>0x01060100&0xffffff00</string>
<key>IOProbeScore</key>
<integer>800</integer>
<key>IOProviderClass</key>
<string>IOPCIDevice</string>
<key>Vendor Name</key>
<string>Unknown</string>
12

Verifying Thunderbolt Connections in PCI Devices
key>IOPCITunnelCompatible/key>
<true/>
</dict>
Verifying Thunderbolt Connections in PCI Devices

PCI device drivers can determine if a Thunderbolt device is connected by recursively searching over parents
in the I/O Registry for the key IOPCITunnelled. (See code example Listing 2-3 (page 13) .)
Listing 2-3
Verifying Thunderbolt Connections
#include < IOKit/pci/IOPCIDevice.h>

bool tb = false;
if ( getProperty ( kIOPCITunnelledKey, gIOServicePlane, kIORegistryIterateRecursively
| kIORegistryIterateParents ) )
tb = true;
Additionally, PCI device drivers that present storage to the operating system must ensure the Physical
Interconnect Location key is set to indicate that the device is externally connected. (See physical interconnect
location key example Listing 2-4 (page 13) .)
Listing 2-4
PCI Driver: External Storage Key Set
setProperty ( kIOPropertyPhysicalInterconnectLocationKey,
kIOPropertyExternalKey );
Note: This key must be set before any devices are instantiated (that is, early in the drivers start()
routine or when it loads based on its IOKitPersonality in the Info.plist file).
13
Handling and Routing Interrupts
This chapter provides guidelines to help you support Thunderbolt devices using Message Signaled Interrupts
(MSI) and to understand how to respond to hardware devices that do not support MSI. There is also a section
on hot plug operations with PCI Devices that describes how to change PCI drivers so that they are able to deal
with unplanned disconnections.
Most modern PCI devices support flexibility when dealing with interrupt routing. With the advent of PCI-X (PCI
eXtended) and PCIe (PCI Express), Message Signaled Interrupts were introduced as an in-band mechanism for
asserting interrupts.
Building Only MSI-Capable Thunderbolt Devices

Apple strongly recommends building only MSI-capable Thunderbolt devices. Ensure your OS X drivers enable
MSI when supporting Thunderbolt devices. To ease development, the Mac Pro computers have PCIe slots that
support MSI.
Note: MSI are edge-triggered interrupts while legacy interrupts are typically level-triggered. The
hardware may behave differently depending on the interrupt type used. Ensure your software is
written so that it can handle the different types of interrupts properly. Whenever possible, Apple
recommends using MSI.
OS X services all interrupts on CPU 0 and uses the secondary interrupt context threads to defer work to task-level
interrupts rather than primary-interrupt levels. This method ensures that multiple drivers can run in parallel
on the available CPUs, that the OS can schedule real-time threads with accuracy, and that the system remains
responsive to user interaction. The best practice methods, described in I/O Kit Fundamentals , encourage the
use of the IOWorkLoop abstraction model for device drivers in order to defer work to the IOWorkLoop thread.
There may be a small number of devices that require work to be done in the primary interrupt context, however,
drivers should spend the least amount of time possible in the primary-interrupt context.
14

Handling Hardware Exceptions for MSI Support
Note: Since Message Signaled Interrupts are never shared, most device drivers can use the MSI
vector with an IOInterruptEventSource instance and avoid spending time in the primary interrupt
context.

The hardware should support MSI, but in the unlikely event the hardware device does not support MSI, the
following three scenarios are most likely to occur. These three scenarios usually occur in the drivers filter
routine and it is important to understand the ramifications of the results returned from the filter routine:
1.
The filter is called, but the device has not signaled an interrupt.
This may be due to another device sharing the same interrupt pin. The filter routine should return false.
2.
The filter is called and the device has signaled an interrupt.

The driver cannot mask the interrupt source, nor can it handle the interrupt at this level for locking or
performance reasons. The filter routine should return true.
Note: The I/O Kit interprets this true as needing to disable the entire interrupt pin at the
interrupt controller in the computer. This disables the interrupts for all devices that share the
same interrupt pin and induces additional latency in servicing each devices respective interrupts.
3.
The filter is called and the device has signaled an interrupt.

a.
The driver can handle the interrupt at this level and return the device to a non interrupting state. If
the interrupt action needs to be run, the filter routine should call signalInterrupt().
Note: The filter may be called again either before or during the execution of the interrupt
action. The filter routine should return false.
b.
The driver cannot handle the interrupt at this level, but it can prevent the device from signaling
another interrupt (for example, by manipulating an internal mask).
The filter routine should prevent the device from signaling another interrupt and then call
signalInterrupt() to cause the interrupt action to be run. The interrupt action should handle the
interrupt and return the device to a state where it can again signal an interrupt. The filter routine
should return false.
15

Note: The MMIO read cycles and PCI Configuration cycles of any type (read or write) are non-posted
transactions from the CPU, which turn into PCIe transactions and may take many thousands of clock
cycles to complete . These transactions should be minimized whenever possible.
Apple provides developers with tools that can track primary interrupt times and help developers to minimize
the time needed for primary interrupts.
Enabling MSI
To enable MSI, a device driver should do the following, assuming that the provider is an IOPCIDevice instance:
Note: The device driver should use source as the last argument when creating an
IOInterruptEventSource or IOFilterInterruptEventSource.
Listing 3-1
Enabling MSIs
int
index
= 0;
int
source = 0;
for ( index = 0; ; index++ )

{
IOReturn result
= kIOReturnSuccess;
int interruptType
= 0;
result = provider->getInterruptType ( index, &interruptType );

if ( result != kIOReturnSuccess )
break;
if ( interruptType & kIOInterruptTypePCIMessaged )
{
source = index;
break;
}
}
16

Using Hot Plug Operation with PCI Devices

PCI device drivers are typically developed with the expectation that the device will not be removed from the
PCI bus during its operation. However, Thunderbolt technology allows PCI data to be tunneled through a
Thunderbolt connection, and the Thunderbolt cables may be unplugged from the host or device at any time.
Therefore, the system must be able to cope with the removal of PCI devices by the user at any time.
Warning: To prevent data loss, storage devices should not be unplugged unless the user unmounts
the storage device first.
The user may freely unplug Thunderbolt devices at any time, with the exception of storage devices, and should
be able to sleep and wake the system with devices attached without causing any problems. For all devices,
the user must not be able to hang the system (computer) by unplugging a device or cable.
The PCI device drivers used with Thunderbolt devices may need to be updated in order to handle surprise or
unplanned removal. In particular, MMIO cycles and PCI Configuration accesses require special attention. When
a PCI device that is connected to a Thunderbolt port is detached from the system, the PCIe Root Port must
time out any outstanding transactions sent to the device, terminate the transaction as though an Unsupported
Request occurred on the bus, and return a value of 0xFFFFFFFF. The Root Port has a completion timeout
value that is many milliseconds long and varies depending on the system layout. Real-time scheduling,
particularly for audio and video threads, can be affected by transactions that must be timed out.
Note: The surprise removal of PCI devices mentioned in the previous paragraphs is similar to removal
of ExpressCard devices in the previous generation of MacBook Pro laptop products.
As a basic guideline, developers should modify their drivers to handle a return value of 0xFFFFFFFF. If any
thread, callback, interrupt filter, or code path in a driver receives 0xFFFFFFFF indicating the device has been
unplugged, then all threads, callbacks, interrupt filters, interrupt handlers, and other code paths in that driver
must cease MMIO reads and writes immediately and prepare for termination.
Note: Use an instance variable to enforce modifying the drivers to return a value of 0xFFFFFFFF
and prevent the driver from issuing additional MMIO or PCI Configuration cycles, which may cause
additional transaction timeouts.
If 0xFFFFFFFF is a legal value for a particular register offset, one additional read of a different register, which
is known to never return 0xFFFFFFFF is the preferred mechanism for determining if the device is still connected.
Finally, if I/O Kit has already performed termination and called the drivers willTerminate() method, no
further accesses should be performed.
17

Once it has been determined that a device is no longer connected, do not try to clean up or reset the hardware
as attempts to communicate with the hardware may lead to further delays.
Apple recommends auditing usage of MMIO writes when no further access should be performed. MMIO writes
are posted transactions and it is possible for device drivers to queue up multiple writes in a row.
Note: It is best to prevent large numbers (i.e. > 2) of writes without an intervening read access even
though writes are preferred over reads in PCI.
A typical way for a developer to solve this problem is to provide a single bottleneck routine for all MMIO reads
and have that routine check the status of the device before beginning the actual transaction. An example of
such a routine using little endian fields for its memory mapped apertures follows:
Listing 3-2
Single Bottleneck Routine (MMIO Read/Write)
class AppleSamplePCI
{
...
bool
fDeviceRemoved;
volatile unit8_t *
fBaseAddressRegister;
...
unit32_t
ReadRegister ( uint32_t
offset );
virtual bool
willTerminate ( IOService * provider, IOOptionBits options
);
};
unit32_t
AppleSamplePCI::ReadRegister ( unint32_t offset )
{
unint32_t
if
result = 0xFFFFFFFF;
( !fDeviceRemoved )
{
result = OSReadLittleInt32 ( fBaseAddressRegister, offset );
if ( result == 0xFFFFFFFF )
fDeviceRemoved = true;
18

}
return result;
}
bool
AppleSamplePCI::willTerminate ( IOService * provider, IOOptionBits options )
{
fDeviceRemoved = true;
return super::willTerminate ( provider, options );
}
Similar routines can be written for PCI Configuration cycle transactions, which may receive a return value of
0xFFFFFFFF and MMIO reads of smaller sizes.
Note: Be sure to initialize the variables fBaseAddressRegister and fDeviceRemoved before
use.
Supporting PCIe Pause

Because Thunderbolt allows the addition and removal of arbitrary numbers of peripherals connected in arbitrary
topologies, the task of dividing up the PCI trees address space can be challenging. Sometimes, particularly
when large numbers of devices are attached, it is possible to exhaust portions of that address space. When
this happens, a new device cannot be enabled without moving existing devices.
To solve this problem, OS X v10.9 supports PCIe Pausea special power management state in which all driver
and device operations are temporarily suspended. Whenever address space exhaustion occurs, OS X may ask
drivers to pause operations. After the drivers are paused, OS X changes the address space layout of the paused
devices to make room for new devices, and then tells the drivers to resume normal operation.
To support pause, you must add an additional Info.plist key and a new power management state
(kIOPCIDevicePausedState) to your driver, as described in the following sections.
Declare Support for Pausing In Your Info.plist File

To declare support for pausing, add the following key to each of the personalities in your drivers Info.plist
file:
<key>IOPCIPauseCompatible</key>
19

<true/>
Important: If your driver does not explicitly declare support for pausing, your driver will never receive
pause requests. As a result, devices may fail to appear when the user plugs in additional devices, particularly
on hardware with multiple Thunderbolt ports. For this reason, you are strongly encouraged to support this
functionality as soon as possible.
Add Support for the kIOPCIDevicePausedState Power State

When a driver for a IOPCIDevice provider registers for power management, it must provide a set of power
state definitions indicating which of its power states should be used for each power state of the IOPCIDevice
object itself. The inputPowerRequirement field of these power states is matched against masks of the PCI
power states outputPowerCharacter field.
If you do not add any power states to your driver, your driver is put into its off state during pause, because the
pause state does not include the kIOPMPowerOn flag. Any clients of the driver in the power management
hierarchy then must change their states to match.
Modified drivers can support pause explicitly by adding a power state with the kIOPMConfigRetained flag
set in the inputPowerRequirement field, causing that state to be selected by the power management system
when the device needs to enter a paused state. The outputPowerCharacter value of the drivers new power
state then dictates what the drivers power management clients see. Depending on this flag, clients of your
driver may or may not change states.
For example, your power state table might look like this:
static const IOPMPowerState powerStates[kYourDriverPowerStateCount] = {
// version, capabilityFlags, outputPowerCharacter,

// inputPowerRequirement, staticPower, unbudgetedPower
// powerToAttain timeToAttain settleUpTime
// timeToLower settleDownTime powerDomainBudget
// Device off; inputPowerRequirement is 0,

// which matches the flags for kIOPCIDeviceOffState.
{ kIOPMPowerStateVersion1, 0, 0,
0, 0, 0,
0, 0, 0,
20

0, 0, 0 },
// Sleep mode; inputPowerRequirement has kIOPMSoftSleep flag set,

// which matches the flags for kIOPCIDeviceDozeState.
{ kIOPMPowerStateVersion1, 0, kIOPMSoftSleep,
kIOPMSoftSleep, 0, 0,
0, 0, 0,
0, 0, 0 },
// Device paused; inputPowerRequirement has kIOPMConfigRetained flag set,

// which matches the flags for kIOPCIDevicePausedState.
{ kIOPMPowerStateVersion1, kIOPMConfigRetained, kIOPMConfigRetained,
kIOPMConfigRetained, 0, 0,
0, 0, 0,
0, 0, 0 },
// Device active; inputPowerRequirement has kIOPMPowerOn flag set,

// which matches the flags for kIOPCIDeviceOnState.
{ kIOPMPowerStateVersion1, kIOPMPowerOn | kIOPMUsable, kIOPMPowerOn,
kIOPMPowerOn, 0, 0,
0, 0, 0,
0, 0, 0 }
};
To avoid disrupting service, write your code in a way that makes entering and exiting the pause state as fast
as possible. In particular, you do not need to run all of the code that you would use for a
kIOPCIDeviceOffState/kIOPCIDeviceOnState transition, because the device remains powered on
through the state transition, making a full reinitialization unnecessary.
However, when OS X tells your driver to pause, your driver should still do many of the things that it would do
when the computer goes into safe sleepthat is, it should tell the device to stop issuing additional transactions
and then wait until all outstanding transactions have finished before telling OS X that the devices power state
has changed.
While the driver is paused:
The driver should not access the device using memory-mapped I/O or configuration space transactions
The device should not generate any interrupts, whether MSI or pin-based interrupts
21

The device should not generate any DMA requests
The device should not be the target of any DMA requests
Note: This quiesced (inactive) behavior is similar to what your driver should do in all of the other
inactive power states (that is, all power states other than kIOPCIDeviceOnState), so the driver
state should be very similar to the off and doze states. However, because the system is not actually
being powered down or going to sleep, the device does not receive any PCI broadcast messages,
such as PME_turn_off.
When a driver is resumed after a pause, the driver should act as though the computer just woke from safe
sleep, but without performing any unnecessary hardware initialization (because the device remained powered).
In particular, it must determine whether the device has changed addresses, and if it has, it must use the new
physical addresses for all future communication with the device. The following values may have changed:
The devices base address registers (BARs)
The devices bus number
Registry properties reflecting these values: "ranges", "assigned-addresses", and "reg"
The devices MSI capability register block values for address and value, but not the number of MSIs allocated
The following values will not change:
The PCI power management configuration block registers of the devicethat is, the device will not be
put into a device sleep state (runtime D3)
The virtual addresses of the BARs

Any mappings previously created by the driver with IOPCIDevice::mapDeviceMemoryWithRegister
or IOPCIDevice::mapDeviceMemoryWithIndex for memory-mapped I/O access to hardware are
automatically remapped with the same virtual address and the new physical address
Any other configuration registers
The set of available BAR resources

Any BAR resource present at pause is guaranteed to be reallocatedthat is, a device will never lose
resources across reconfiguration.
The registry hierarchy of the device
The number or kind of interrupt assignments (shared versus MSI)
22

Most drivers have few or no dependencies on the items above being changed. If they do, while entering the
state kIOPCIDeviceOnState (from any other state), these dependencies should be updated to the current
configuration of the device.
Returning I/O Operations as Errors

Drivers should ensure any I/Os that are in flight at the time of surprise removal are properly returned as errors
to the upper layers that issued the I/O requests. The exact manner in which this is done is I/O family specific.
23
Debugging Thunderbolt Drivers
This chapter contains debugging tips that may be useful when tracking down problems in Thunderbolt drivers.
Debugging VT-d I/O MMU Virtualization

Newer Thunderbolt-capable Macs provide support for I/O MMU virtualization (VT-d). This technology allows
virtual machines to have direct access to hardware. As a consequence of this support, when your device
performs DMA operations, the I/O addresses it uses may be different from the physical addresses as seen by
the OS X kernel.
The complete spec can be found here: Intel Virtualization Technology for Directed I/O Specification.
Ivy Bridge systems running OS X v10.8.2 and later enable the Intel VT-d unit as a DMA remapper. This functionality
is supported by the current APIs in much the same way as the DART controller on PowerMac G5 computers.
(To learn how this works at a high level, read Supporting DMA on 64-Bit System Architectures in I/O Kit
Fundamentals ).
When VT-d is enabled, some of the changes you'll see as a results of this are:
If you are writing DMA programs, although physical segments returned by IOMemoryDescriptor or
IODMACommand are contiguous ranges of I/O addresses, the I/O addresses for the segments may be
different than the CPU physical addresses, and the ranges are no longer necessarily contiguous in physical
memory from perspective of the CPU.
As a result, segments will usually be the same size as the full IOMemoryDescriptor size (for a virtually
contiguous IOMemoryDescriptor object with a single source buffer).
Note: This assumes you are using the default behavior for IODMACommand (mappingOptions
= kMapped) or getPhysicalSegment (without the kIOMapperNone option set). Specifying
nondefault options produces different behavior.
By default, requests for physically contiguous memory return memory that is contiguous only from the
perspective of I/O devices, and may not be physically contiguous in RAM.
24

Debugging VT-d I/O MMU Virtualization
If the memory must be contiguous to the CPU as well, you must pass the
kIOMemoryHostPhysicallyContiguous option to IOBufferMemoryDescriptor when you request
the memory. As before, such requests can fail if no physically contiguous blocks are available.
Transactions now fail if memory descriptors or DMA commands are not properly prepared for reading or
writing.
The prepare method adds a mapping entry into the I/O address translation table. You must call prepare
(and specify the correct direction) before you tell the device to read data from or write data to that region
of memory.
You must also balance each prepare call with a matching call to the complete method. If you do not,
your driver will eventually fill up the mapping table, and you will get a kernel panic the next time a driver
tries to add a mapping table entry.
Important: The I/O direction matters. If you prepare a descriptor or DMA command for reading and
your device then tries to write to that memory, the transaction will fail.
The prepare and complete methods have additional overhead, though this impact is partially
counterbalanced by shorter scatter-gather lists.
If you ask for unmapped physical addressesby calling an IOMemoryDescriptor objects

getPhysicalSegment method with the option kIOMapperNone or an IODMACommand objects
initWithSpecification method with mappingOptions set to something other than kMappedand try
to use them for DMA, your driver will break.
Enabling VT-d Panics

By default, VT-d faults are logged to /var/log/system.log. A VT-d fault looks like the following:
vtd[0] fault: device 13:0:0 reason 0x6 R:0x3dff000
In the example above, 13:0:0 is the bus, device, and function of the device that generated the fault. You can
determine if your device produced the fault by looking at the pci-debug field in the output of the ioreg or
in the IORegistryExplorer application and comparing the values.
For a list of reason codes, see the Intel Virtualization Technology for Directed I/O Specification.
The R indicates that a read operation triggered the fault. (A W indicates a write.) The final value is the address
that the device was trying to read or write (expressed as an I/O-space address).
When debugging faults, it can be useful to configure your kernel to panic whenever a fault occurs. To do this,
add the following flag in your kernel boot args:
25

Debugging PCIe Pause
pci=0x100
Disabling VT-d
When debugging PCIe device drivers, it is often useful to temporarily disable VT-d so that I/O addresses are
the same as the corresponding physical addresses. To disable VT-d, add the following to your kernel boot args:
dart=0x0
If the problem you are debugging goes away in this mode, it usually indicates one of the following mistakes:
Some part of your code is incorrectly passing a physical address in RAM to your device for DMA purposes
instead of an I/O address.
Your code failed to call prepare or called prepare incorrectly on an IOMemoryDescriptor object
before using it to perform DMA.
Debugging PCIe Pause

By default, the OS X kernel spreads out PCI address allocations. As a result, PCIe pause events occur infrequently,
which makes these events challenging to debug. You can make debugging easier by disabling this allocation
spreading behavior. With spreading disabled, nearly every hot plug event triggers a pause event.
To disable allocation spreading, add the following to your kernel boot-args string:
pci=0x200
Avoiding Memory Leaks

Drivers should ensure release of any and all resources acquired over the lifecycle of the driver. In particular,
memory mapped I/O ranges, memory, and objects should all be freed and double-checked for any leaks. Use
of tools such as ioclasscount can help identify some of these types of leaks. A common form of leaks is
introduced by references (that is, retain counts). Often, it is difficult to determine the source of these references.
One way to determine the source of these references is to override the taggedRetain() method and obtain
a backtrace of the caller, which you can then send to printf(), kprintf(), or IOTimeStampConstant().
26

Avoiding Memory Leaks
Listing 4-1
Searching for Memory Leaks
void
AppleSamplePCI::taggedRetain ( const void * tag ) const
{
void *
bt[16] = { 0 };
OSBacktrace ( &bt[0], sizeof ( bt ) / sizeof ( bt[0] ) );

super::taggedRetain ( tag );
}
Using standard symbolication tools allows you to determine which functions or methods caused the reference(s)
to be taken. Furthermore, you can override taggedRelease() and match the retains and releases to find
calls to taggedRetain(), which have no corresponding call to taggedRelease().
Note: The const keyword at the end of the method signatures is required to properly override
these methods.
27
Document Revision History
This table describes the changes to Thunderbolt Device Driver Programming Guide .
Date
Notes
2013-10-22
Updated with information about PCIe pause and tips on debugging

memory leaks.
2011-12-21
First version of a document that explains the Thunderbolt technology for

developers.
28
Apple Inc.
Copyright 2013 Apple Inc.
All rights reserved.
No part of this publication may be reproduced,
stored in a retrieval system, or transmitted, in any
form or by any means, mechanical, electronic,
photocopying, recording, or otherwise, without
prior written permission of Apple Inc., with the
following exceptions: Any person is hereby
authorized to store documentation on a single
computer for personal use only and to print
copies of documentation for personal use
provided that the documentation contains
Apples copyright notice.
No licenses, express or implied, are granted with
respect to any of the technology described in this
document. Apple retains all intellectual property
rights associated with the technology described
in this document. This document is intended to
assist application developers to develop
applications only for Apple-labeled computers.
Apple Inc.
1 Infinite Loop
Cupertino, CA 95014
408-996-1010
Apple, the Apple logo, FireWire, Leopard, Mac,
Mac Pro, MacBook, OS X, and Snow Leopard are
trademarks of Apple Inc., registered in the U.S.
and other countries.
Intel and Intel Core are registered trademarks of
Intel Corporation or its subsidiaries in the United
States and other countries.
Even though Apple has reviewed this document,
APPLE MAKES NO WARRANTY OR REPRESENTATION,
EITHER EXPRESS OR IMPLIED, WITH RESPECT TO THIS
DOCUMENT, ITS QUALITY, ACCURACY,
MERCHANTABILITY, OR FITNESS FOR A PARTICULAR
PURPOSE. AS A RESULT, THIS DOCUMENT IS PROVIDED
AS IS, AND YOU, THE READER, ARE ASSUMING THE
ENTIRE RISK AS TO ITS QUALITY AND ACCURACY.
IN NO EVENT WILL APPLE BE LIABLE FOR DIRECT,
INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL
DAMAGES RESULTING FROM ANY DEFECT OR
INACCURACY IN THIS DOCUMENT, even if advised of
the possibility of such damages.
THE WARRANTY AND REMEDIES SET FORTH ABOVE
ARE EXCLUSIVE AND IN LIEU OF ALL OTHERS, ORAL
OR WRITTEN, EXPRESS OR IMPLIED. No Apple dealer,
agent, or employee is authorized to make any
modification, extension, or addition to this warranty.
Some states do not allow the exclusion or limitation
of implied warranties or liability for incidental or
consequential damages, so the above limitation or
exclusion may not apply to you. This warranty gives
you specific legal rights, and you may also have other
rights which vary from state to state.

ThunderboltDevice Driver Programming Guide by Apple

Transféré par

Informations du document

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

ThunderboltDevice Driver Programming Guide by Apple

Transféré par

Droits d'auteur :

Formats disponibles

Thunderbolt Device

About the Thunderbolt Technology 4

Thunderbolt Technology Overview 6

Working with Thunderbolt Technology 10

Handling and Routing Interrupts 14

Debugging Thunderbolt Drivers 24

Document Revision History 28

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Figures and Listings

Thunderbolt Technology Overview 6

Intel 82524EF controller chip (host mode) 6

Working with Thunderbolt Technology 10

PCI Driver Key (Thunderbolt Operation) 12

Handling and Routing Interrupts 14

Debugging Thunderbolt Drivers 24

Searching for Memory Leaks 27

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

About the Thunderbolt Technology

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

About the Thunderbolt Technology

I/O Kit Fundamentals

64-Bit Transition Guide

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Thunderbolt Technology Overview

The Beginning: The Thunderbolt Controller Chip

Intel 82524EF controller chip (host mode)

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Thunderbolt Technology Overview

Intel 82524EF controller chip (endpoint mode)

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Thunderbolt Technology Overview

Using Thunderbolt Communication over Ports

Expansion chassis utilizing PCI paths

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Thunderbolt Technology Overview

PCI fanout (two disk controllers)

PCI direct routing (two disk controllers)

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Working with Thunderbolt Technology

Discovering Thunderbolt Devices

Using PCI Resources Wisely

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Working with Thunderbolt Technology

Tolerating PCI Latency

And so on. The details are highly dependent on the driver.

Using a Registered Vendor ID

Enabling PCI Device Drivers for Thunderbolt Operation

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Working with Thunderbolt Technology

PCI Driver Key (Thunderbolt Operation)

PCI Driver Personality (Thunderbolt Key Set)

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Working with Thunderbolt Technology

Verifying Thunderbolt Connections in PCI Devices

Verifying Thunderbolt Connections

#include < IOKit/pci/IOPCIDevice.h>

PCI Driver: External Storage Key Set

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Handling and Routing Interrupts

Building Only MSI-Capable Thunderbolt Devices

2013-10-22 | Copyright 2013 Apple Inc. All Rights Reserved.

Handling and Routing Interrupts

Handling Hardware Exceptions for MSI Support

The filter is called and the device has signaled an interrupt.

OSBacktrace ( &bt[0], sizeof ( bt ) / sizeof ( bt[0] ) );