xferring.cpp File Reference

#include "xhcidrv.h"
#include <debug.h>

Include dependency graph for xferring.cpp:

Go to the source code of this file.

Macros
#define	NDEBUG

Functions
	WDF_DECLARE_CONTEXT_TYPE_WITH_NAME (XhciTransferRing, XhciRingContext)
static ULONG NTAPI	XhciRingCountList (_In_ const LIST_ENTRY *Head)
static VOID NTAPI	XhciRingAppendList (_Inout_ PLIST_ENTRY To, _Inout_ PLIST_ENTRY From)
static VOID NTAPI	XhciRingPrependList (_Inout_ PLIST_ENTRY To, _Inout_ PLIST_ENTRY From)
static VOID NTAPI	XhciRingQueueCleanup (_In_ WDFOBJECT Object)
static VOID NTAPI	XhciRingGrowthWorkItem (_In_ WDFWORKITEM WorkItem)
static WDF_IO_FORWARD_PROGRESS_ACTION NTAPI	XhciRingExamineForwardProgress (_In_ WDFQUEUE Queue, _In_ PIRP Irp)
	_IRQL_requires_ (PASSIVE_LEVEL) NTSTATUS XhciTransferRing
	Sends a synchronous IOCTL to the specified device object.
	_IRQL_requires_max_ (DISPATCH_LEVEL) NTSTATUS XhciTransferRing

Variables
static const ULONG	XhciRingSegmentBytes = 0x200
static const ULONG	XhciRingWideSegmentBytes = 0x1000
static const ULONG	XhciRingDoubleBufferBytes = 0x200
static const ULONG	XhciRingSingleSegmentStage = (XhciRingSegmentBytes / sizeof(XHCI_TRB) - 4) * PAGE_SIZE
static const ULONG	XhciRingIdleWaitMs = 5000
static const ULONG	XhciRingIdleWaitUs = 100000
static const USBD_STATUS	XhciRingUsbdNotSet = (USBD_STATUS)0xFFFFFFFF
static EVT_WDF_OBJECT_CONTEXT_CLEANUP	XhciRingQueueCleanup
static EVT_WDF_WORKITEM	XhciRingGrowthWorkItem
static EVT_WDF_IO_WDM_IRP_FOR_FORWARD_PROGRESS	XhciRingExamineForwardProgress
static KSPIN_LOCK	XhciLiveRingLock
static LIST_ENTRY	XhciLiveRings = { &XhciLiveRings, &XhciLiveRings }

Macro Definition Documentation

NDEBUG

#define NDEBUG

Definition at line 10 of file xferring.cpp.

Function Documentation

_IRQL_requires_()

_IRQL_requires_

(

PASSIVE_LEVEL

)

Sends a synchronous IOCTL to the specified device object.

Parameters

[in]	IoControlCode	The IOCTL to send to the device.
[in]	DeviceObject	Pointer to the device object that will handle the IOCTL.
[in]	InputBuffer	Optional pointer to a buffer containing input data for the IOCTL. When specified, the buffer should be at least of InputBufferLength size.
[in]	InputBufferLength	Size in bytes, of the buffer pointed by InputBuffer.
[out]	OutputBuffer	Optional pointer to a buffer that will receive output data from the IOCTL. When specified, the buffer should be at least of OutputBufferLength size.
[in]	OutputBufferLength	Size in bytes, of the buffer pointed by OutputBuffer.
[in]	FileObject	Optional pointer to a file object that may be necessary for the IOCTL.

Returns

An NTSTATUS code indicating success or failure of this function.

Note

Must be called at PASSIVE_LEVEL with all APCs enabled.

Nothing is found unless the PCI bus driver publishes the properties.

The root hub name without its prefix; NULL when the PDO has no name.

Grows a private buffer to what the helper asks for.

Lets KMDF handle the IRP, then hands it back here still uncompleted.

Waits for the device machine to answer a PnP or power event; returns the PnP status.

Deletes the UCX endpoints of the disabled pipes, then the interface record.

One 4 byte value; the registry type is not checked.

Reread whenever user experience settings change; the open status is returned.

Sends a synchronous IOCTL to the specified device object.

STATUS_SUCCESS when the caller's token is an administrator's.

Sends a synchronous IOCTL to the specified device object.

Reads one value of a walk. A missing value only clears Present; FALSE means stop.

Reads up to Size bytes of a value into a zeroed buffer; the type is not checked.

Writes a REG_BINARY value into the device's usbflags subkey.

Sets or clears ExtPropertiesInstalled from ExtPropDescSemaphore in the hardware key.

Fails only when a port cannot be bound to the USB4 host router its ACPI node names.

Assigns device, hardware, compatible, container and instance ids to a new child PDO.

Answers IRP_MN_QUERY_DEVICE_TEXT on a child PDO; always completes the IRP.

Takes the connector map lock, needed around HubConnectorFindCompanion.

Device hardware key values read at PDO prepare hardware.

Saves the MS OS 2.0 set info in usbflags when the set asks for alternate enumeration.

MsOs20Flags and EnumerationRetryCount, written at PDO start.

Rereads the UXD record; UpdateRequired says whether the device id changes.

Replacement PnP id of a UXD reserved device, or an empty String. Always succeeds.

Telemetry values under the device's Ceip subkey.

BootPathSurpriseRemovalCount + 1.

This machine's dual role features for a dual role partner.

Tells UCX the WDM PDO of the device; call right after the child PDO was created.

IOCTL_INTERNAL_USB_GET_CONTROLLER_NAME: Argument1 and its length Argument2.

IOCTL_INTERNAL_USB_GET_BUS_INFO for the device: the 16 byte USB_BUS_NOTIFICATION at Argument1.

Registers the GUID_USB_WMI_NODE_INFO instance of a child PDO (not for hubs or composites).

Registers the boot device surprise removal event on an external, non hub boot PDO.

Fires the boot device surprise removal event, if one is registered.

Sends a synchronous IOCTL to the specified device object.

Publishes "<port>#<host>" for every bound port on the hub's PDO. QUIRK: with no bound port an older value is left in place.

Binds USB 3 ports to the host routers named by the USB4 DVSEC; an earlier ACPI mapping wins.

Waits on a PnP handshake event with no timeout, printing a watchdog line every minute so a stuck machine is visible in the debugger.

Copies up to Size bytes of a device property; a failed query leaves the field zero.

Only the controller name length is real; both bandwidth fields read 0.

The answer overlays a USB_HUB_NAME on TotalSize, so TotalSize comes back as the name length + 4.

Evaluates a method without arguments into Output.

Drops every MSI descriptor and keeps only lists that still offer a line interrupt.

Picks the list with the best MSI offer: MSI-X first, then the most messages.

Keeps message descriptors covering Wanted messages and removes the others.

Evaluates _DSM on the controller ACPI object with a buffer as Arg3; returns up to 8 result bytes, 0 on failure.

Definition at line 122 of file xferring.cpp.

{
    XhciController* Controller = Endpoint->Controller();
    WDF_IO_QUEUE_CONFIG QueueConfig;
    WDF_OBJECT_ATTRIBUTES Attributes;
    WDF_DPC_CONFIG DpcConfig;
    PFN_WDF_DPC CompletionDpc;
    XhciTransferRing* NewRing;
    WDFQUEUE Queue;
    ULONG TransferType;
    ULONG SegmentSize;
    ULONG MaxStageSize;
    BOOLEAN NoLinkInTd;
    NTSTATUS Status;
 
    *Ring = NULL;
 
    TransferType = Endpoint->TransferType();
    MaxStageSize = Controller->m_Buffers.MaxDmaSize() - PAGE_SIZE;
    NoLinkInTd = Controller->HasErrata(XhciErrata::RingKeepTdInOneSegment);
 
    if (TransferType == USB_ENDPOINT_TYPE_CONTROL)
    {
        WDF_IO_QUEUE_CONFIG_INIT(&QueueConfig, WdfIoQueueDispatchSequential);
        QueueConfig.EvtIoDefault = XhciControlRing::EvtIoDefault;
        QueueConfig.EvtIoCanceledOnQueue = XhciControlRing::EvtIoCanceledOnQueue;
        CompletionDpc = XhciControlRing::EvtCompletionDpc;
 
        /* A whole control transfer must fit one segment when no Link TRB may split it */
        SegmentSize = NoLinkInTd ? XhciRingWideSegmentBytes : XhciRingSegmentBytes;
    }
    else if (TransferType == USB_ENDPOINT_TYPE_BULK || TransferType == USB_ENDPOINT_TYPE_INTERRUPT)
    {
        WDF_IO_QUEUE_CONFIG_INIT(&QueueConfig, WdfIoQueueDispatchManual);
        QueueConfig.EvtIoCanceledOnQueue = XhciBulkRing::EvtIoCanceledOnQueue;
        CompletionDpc = XhciBulkRing::EvtCompletionDpc;
        SegmentSize = XhciRingSegmentBytes;
 
        if (NoLinkInTd)
            MaxStageSize = XhciRingSingleSegmentStage;
    }
    else if (TransferType == USB_ENDPOINT_TYPE_ISOCHRONOUS)
    {
        WDF_IO_QUEUE_CONFIG_INIT(&QueueConfig, WdfIoQueueDispatchManual);
        QueueConfig.EvtIoCanceledOnQueue = XhciIsochRing::EvtIoCanceledOnQueue;
        CompletionDpc = XhciIsochRing::EvtCompletionDpc;
        SegmentSize = XhciRingSegmentBytes;
    }
    else
    {
        DPRINT1("Endpoint DCI %lu has transfer type %lu, no transfer ring for it\n",
                Endpoint->Dci(), TransferType);
        return STATUS_UNSUCCESSFUL;
    }
 
    WDF_OBJECT_ATTRIBUTES_INIT_CONTEXT_TYPE(&Attributes, XhciTransferRing);
    Attributes.ParentObject = Parent;
    Attributes.EvtCleanupCallback = XhciRingQueueCleanup;
 
    Status = WdfIoQueueCreate(Controller->m_Device, &QueueConfig, &Attributes, &Queue);
    if (!NT_SUCCESS(Status))
    {
        DPRINT1("Queue for DCI %lu stream %lu not created, 0x%08lx\n",
                Endpoint->Dci(), StreamId, Status);
        return Status;
    }
 
    /* The context is zeroed by WDF, so cleanup copes with any step below failing */
    NewRing = new (XhciRingContext(Queue)) XhciTransferRing;
    KeInitializeSpinLock(&NewRing->m_Lock);
    InitializeListHead(&NewRing->m_FreeSegments);
    InitializeListHead(&NewRing->m_DoubleBuffers);
    XhciClearListEntry(&NewRing->m_LiveLink);
    NewRing->m_Device = Endpoint->Device();
    NewRing->m_Endpoint = Endpoint;
    NewRing->m_StreamId = StreamId;
    NewRing->m_TransferType = TransferType;
    NewRing->m_Queue = Queue;
    NewRing->m_MapState = static_cast<LONG>(XhciMapState::Halted);
    NewRing->m_SegmentSize = SegmentSize;
    NewRing->m_DoubleBufferSize = XhciRingDoubleBufferBytes;
    NewRing->m_MaxStageSize = MaxStageSize;
    NewRing->m_MapRegisterCount = Controller->m_Buffers.MapRegisterCount();
    NewRing->m_LastIndex = SegmentSize / sizeof(XHCI_TRB) - 1;
    NewRing->m_Cycle = 1;
    NewRing->m_Controller = Controller;
 
    if (!NewRing->IsControl())
    {
        WDF_WORKITEM_CONFIG WorkConfig;
 
        Status = WdfIoQueueReadyNotify(Queue,
                                       NewRing->IsIsoch() ? XhciIsochRing::ReadyNotification
                                                          : XhciBulkRing::ReadyNotification,
                                       NewRing);
        if (!NT_SUCCESS(Status))
        {
            DPRINT1("Ready notification for DCI %lu stream %lu not registered, 0x%08lx\n",
                    Endpoint->Dci(), StreamId, Status);
            return Status;
        }
 
        WDF_WORKITEM_CONFIG_INIT(&WorkConfig, XhciRingGrowthWorkItem);
        WorkConfig.AutomaticSerialization = FALSE;
        WDF_OBJECT_ATTRIBUTES_INIT(&Attributes);
        Attributes.ParentObject = Queue;
 
        Status = WdfWorkItemCreate(&WorkConfig, &Attributes, &NewRing->m_GrowthItem);
        if (!NT_SUCCESS(Status))
        {
            DPRINT1("Segment growth work item for DCI %lu stream %lu not created, 0x%08lx\n",
                    Endpoint->Dci(), StreamId, Status);
            return Status;
        }
    }
 
    WDF_DPC_CONFIG_INIT(&DpcConfig, CompletionDpc);
    DpcConfig.AutomaticSerialization = TRUE;
    WDF_OBJECT_ATTRIBUTES_INIT(&Attributes);
    Attributes.ParentObject = Queue;
 
    Status = WdfDpcCreate(&DpcConfig, &Attributes, &NewRing->m_CompletionDpc);
    if (!NT_SUCCESS(Status))
    {
        DPRINT1("Completion DPC for DCI %lu stream %lu not created, 0x%08lx\n",
                Endpoint->Dci(), StreamId, Status);
        return Status;
    }
 
    /* Take the first segment now so a later enable at DISPATCH_LEVEL never allocates */
    Status = NewRing->EnsureFreeSegments(1, FALSE);
    if (NT_SUCCESS(Status))
    {
        XhciSpinLockGuard Guard(&NewRing->m_Lock);
 
        if (IsListEmpty(&NewRing->m_FreeSegments))
        {
            Status = STATUS_INSUFFICIENT_RESOURCES;
        }
        else
        {
            NewRing->m_Segment = CONTAINING_RECORD(RemoveHeadList(&NewRing->m_FreeSegments),
                                                   XhciDmaBuffer, Link);
            XhciClearListEntry(&NewRing->m_Segment->Link);
            NewRing->m_Trbs = (PXHCI_TRB)NewRing->m_Segment->VirtualAddress;
        }
    }
 
    if (!NT_SUCCESS(Status))
    {
        DPRINT1("No initial segment for DCI %lu stream %lu, 0x%08lx\n",
                Endpoint->Dci(), StreamId, Status);
        return Status;
    }
 
    if (NewRing->IsControl())
        Status = XhciControlRing::Initialize(NewRing);
    else if (NewRing->IsIsoch())
        Status = XhciIsochRing::Initialize(NewRing);
    else
        Status = XhciBulkRing::Initialize(NewRing);
 
    if (!NT_SUCCESS(Status))
    {
        DPRINT1("Type setup for DCI %lu stream %lu failed, 0x%08lx\n",
                Endpoint->Dci(), StreamId, Status);
        return Status;
    }
 
    NewRing->m_TypeReady = TRUE;
    NewRing->MakeLive();
 
    DPRINT("Ring %p for DCI %lu stream %lu type %lu, segment 0x%lx, stage max 0x%lx\n",
           NewRing, Endpoint->Dci(), StreamId, TransferType, SegmentSize, MaxStageSize);
 
    *Ring = NewRing;
    return STATUS_SUCCESS;
}

_IRQL_requires_max_()

_IRQL_requires_max_

(

DISPATCH_LEVEL

)

IOCTL_INTERNAL_USB_SUBMIT_IDLE_NOTIFICATION; the idle machine owns the IRP afterwards.

Records hub over current in the hub hardware key, once per devnode.

First selective suspend of the device; the registry write runs in a work item.

"USB device not recognized" popup for a port.

A hub was attached beyond the allowed tier depth.

The selected configuration draws more power than the port can give.

The controller ran out of bandwidth for the device; fired by the out of bandwidth timer.

Definition at line 458 of file xferring.cpp.

{
    ASSERT(m_MapState == static_cast<LONG>(XhciMapState::Halted));
 
    if (m_Segment == NULL)
    {
        DPRINT1("Ring for DCI %lu stream %lu has no segment to enable\n",
                m_Endpoint->Dci(), m_StreamId);
        return STATUS_INVALID_DEVICE_STATE;
    }
 
    InitializeRing();
 
    if (IsControl())
        XhciControlRing::Enable(this);
    else if (IsIsoch())
        XhciIsochRing::Enable(this);
    else
        XhciBulkRing::Enable(this);
 
    DPRINT("Ring for DCI %lu stream %lu enabled\n", m_Endpoint->Dci(), m_StreamId);
    return STATUS_SUCCESS;
}

WDF_DECLARE_CONTEXT_TYPE_WITH_NAME()

WDF_DECLARE_CONTEXT_TYPE_WITH_NAME	(	XhciTransferRing	,
		XhciRingContext
	)

XhciRingAppendList()

static VOID NTAPI XhciRingAppendList ( _Inout_ PLIST_ENTRY  To, _Inout_ PLIST_ENTRY  From  )

static

Moves every entry of From to the tail of To, keeping their order.

Definition at line 60 of file xferring.cpp.

{
    while (!IsListEmpty(From))
        InsertTailList(To, RemoveHeadList(From));
}

Referenced by XhciTransferRing::Cleanup(), XhciTransferRing::EnsureFreeSegments(), and XhciTransferRing::ReturnSegments().

XhciRingCountList()

static ULONG NTAPI XhciRingCountList ( _In_ const LIST_ENTRY *  Head )

static

Definition at line 44 of file xferring.cpp.

{
    const LIST_ENTRY* Entry;
    ULONG Count = 0;
 
    for (Entry = Head->Flink; Entry != Head; Entry = Entry->Flink)
        Count++;
 
    return Count;
}

Referenced by XhciTransferRing::Cleanup(), XhciTransferRing::EnsureFreeSegments(), and XhciTransferRing::TakeSegments().

XhciRingExamineForwardProgress()

static WDF_IO_FORWARD_PROGRESS_ACTION NTAPI XhciRingExamineForwardProgress ( _In_ WDFQUEUE  Queue, _In_ PIRP  Irp  )

static

Definition at line 103 of file xferring.cpp.

{
    PURB Urb;
 
    UNREFERENCED_PARAMETER(Queue);
 
    /* Control and bulk URBs keep TransferFlags at the same offset */
    Urb = (PURB)IoGetCurrentIrpStackLocation(Irp)->Parameters.Others.Argument1;
    if (Urb != NULL &&
        (Urb->UrbBulkOrInterruptTransfer.TransferFlags & USB3_URB_RESERVED_RESOURCES))
    {
        return WdfIoForwardProgressActionUseReservedRequest;
    }
 
    return WdfIoForwardProgressActionFailRequest;
}

XhciRingGrowthWorkItem()

static VOID NTAPI XhciRingGrowthWorkItem ( _In_ WDFWORKITEM  WorkItem )

static

Definition at line 92 of file xferring.cpp.

{
    WDFQUEUE Queue = (WDFQUEUE)WdfWorkItemGetParentObject(WorkItem);
 
    XhciTransferRing::FromQueue(Queue)->GrowSegments();
}

XhciRingPrependList()

static VOID NTAPI XhciRingPrependList ( _Inout_ PLIST_ENTRY  To, _Inout_ PLIST_ENTRY  From  )

static

Moves every entry of From to the head of To, keeping their order.

Definition at line 72 of file xferring.cpp.

{
    while (!IsListEmpty(From))
        InsertHeadList(To, RemoveTailList(From));
}

Referenced by XhciTransferRing::ReturnSegments(), and XhciTransferRing::TakeSegments().

XhciRingQueueCleanup()

static VOID NTAPI XhciRingQueueCleanup ( _In_ WDFOBJECT  Object )

static

Definition at line 83 of file xferring.cpp.

{
    XhciRingContext(Object)->Cleanup();
}

Variable Documentation

XhciLiveRingLock

KSPIN_LOCK XhciLiveRingLock

static

Definition at line 38 of file xferring.cpp.

Referenced by XhciTransferRing::MakeLive(), XhciTransferRing::ReferenceLiveRing(), and XhciTransferRing::RetireLive().

XhciLiveRings

LIST_ENTRY XhciLiveRings = { &XhciLiveRings, &XhciLiveRings }

static

Definition at line 39 of file xferring.cpp.

Referenced by XhciTransferRing::MakeLive(), and XhciTransferRing::ReferenceLiveRing().

XhciRingDoubleBufferBytes

const ULONG XhciRingDoubleBufferBytes = 0x200

static

Definition at line 16 of file xferring.cpp.

Referenced by _IRQL_requires_().

XhciRingExamineForwardProgress

EVT_WDF_IO_WDM_IRP_FOR_FORWARD_PROGRESS XhciRingExamineForwardProgress

static

Definition at line 32 of file xferring.cpp.

XhciRingGrowthWorkItem

EVT_WDF_WORKITEM XhciRingGrowthWorkItem

static

Definition at line 31 of file xferring.cpp.

Referenced by _IRQL_requires_().

XhciRingIdleWaitMs

const ULONG XhciRingIdleWaitMs = 5000

static

Definition at line 22 of file xferring.cpp.

XhciRingIdleWaitUs

const ULONG XhciRingIdleWaitUs = 100000

static

Definition at line 23 of file xferring.cpp.

XhciRingQueueCleanup

EVT_WDF_OBJECT_CONTEXT_CLEANUP XhciRingQueueCleanup

static

Definition at line 30 of file xferring.cpp.

Referenced by _IRQL_requires_().

XhciRingSegmentBytes

const ULONG XhciRingSegmentBytes = 0x200

static

Definition at line 14 of file xferring.cpp.

Referenced by _IRQL_requires_().

XhciRingSingleSegmentStage

const ULONG XhciRingSingleSegmentStage = (XhciRingSegmentBytes / sizeof(XHCI_TRB) - 4) * PAGE_SIZE

static

Largest bulk stage when a TD may not cross a Link TRB: one segment less a few TRBs, one page each.

Definition at line 19 of file xferring.cpp.

Referenced by _IRQL_requires_().

XhciRingUsbdNotSet

const USBD_STATUS XhciRingUsbdNotSet = (USBD_STATUS)0xFFFFFFFF

static

USBD status meaning "no hardware result", for the caller to replace with its default.

Definition at line 26 of file xferring.cpp.

Referenced by XhciTransferRing::UsbdStatusFromCompletionCode().

XhciRingWideSegmentBytes

const ULONG XhciRingWideSegmentBytes = 0x1000

static

Definition at line 15 of file xferring.cpp.

Referenced by _IRQL_requires_().