[net-next-2.6.git] / Documentation / infiniband / core_locking.txt

INFINIBAND MIDLAYER LOCKING

  This guide is an attempt to make explicit the locking assumptions
  made by the InfiniBand midlayer.  It describes the requirements on
  both low-level drivers that sit below the midlayer and upper level
  protocols that use the midlayer.

Sleeping and interrupt context

  With the following exceptions, a low-level driver implementation of
  all of the methods in struct ib_device may sleep.  The exceptions
  are any methods from the list:

    create_ah
    modify_ah
    query_ah
    destroy_ah
    bind_mw
    post_send
    post_recv
    poll_cq
    req_notify_cq
    map_phys_fmr

  which may not sleep and must be callable from any context.

  The corresponding functions exported to upper level protocol
  consumers:

    ib_create_ah
    ib_modify_ah
    ib_query_ah
    ib_destroy_ah
    ib_bind_mw
    ib_post_send
    ib_post_recv
    ib_req_notify_cq
    ib_map_phys_fmr

  are therefore safe to call from any context.

  In addition, the function

    ib_dispatch_event

  used by low-level drivers to dispatch asynchronous events through
  the midlayer is also safe to call from any context.

Reentrancy

  All of the methods in struct ib_device exported by a low-level
  driver must be fully reentrant.  The low-level driver is required to
  perform all synchronization necessary to maintain consistency, even
  if multiple function calls using the same object are run
  simultaneously.

  The IB midlayer does not perform any serialization of function calls.

  Because low-level drivers are reentrant, upper level protocol
  consumers are not required to perform any serialization.  However,
  some serialization may be required to get sensible results.  For
  example, a consumer may safely call ib_poll_cq() on multiple CPUs
  simultaneously.  However, the ordering of the work completion
  information between different calls of ib_poll_cq() is not defined.

Callbacks

  A low-level driver must not perform a callback directly from the
  same callchain as an ib_device method call.  For example, it is not
  allowed for a low-level driver to call a consumer's completion event
  handler directly from its post_send method.  Instead, the low-level
  driver should defer this callback by, for example, scheduling a
  tasklet to perform the callback.

  The low-level driver is responsible for ensuring that multiple
  completion event handlers for the same CQ are not called
  simultaneously.  The driver must guarantee that only one CQ event
  handler for a given CQ is running at a time.  In other words, the
  following situation is not allowed:

        CPU1                                    CPU2

  low-level driver ->
    consumer CQ event callback:
      /* ... */
      ib_req_notify_cq(cq, ...);
                                        low-level driver ->
      /* ... */                           consumer CQ event callback:
                                            /* ... */
      return from CQ event handler

  The context in which completion event and asynchronous event
  callbacks run is not defined.  Depending on the low-level driver, it
  may be process context, softirq context, or interrupt context.
  Upper level protocol consumers may not sleep in a callback.

Hot-plug

  A low-level driver announces that a device is ready for use by
  consumers when it calls ib_register_device(), all initialization
  must be complete before this call.  The device must remain usable
  until the driver's call to ib_unregister_device() has returned.

  A low-level driver must call ib_register_device() and
  ib_unregister_device() from process context.  It must not hold any
  semaphores that could cause deadlock if a consumer calls back into
  the driver across these calls.

  An upper level protocol consumer may begin using an IB device as
  soon as the add method of its struct ib_client is called for that
  device.  A consumer must finish all cleanup and free all resources
  relating to a device before returning from the remove method.

  A consumer is permitted to sleep in its add and remove methods.
Commit	Line	Data
617b586b HR	1	INFINIBAND MIDLAYER LOCKING
	2
	3	This guide is an attempt to make explicit the locking assumptions
	4	made by the InfiniBand midlayer. It describes the requirements on
	5	both low-level drivers that sit below the midlayer and upper level
	6	protocols that use the midlayer.
	7
	8	Sleeping and interrupt context
	9
	10	With the following exceptions, a low-level driver implementation of
	11	all of the methods in struct ib_device may sleep. The exceptions
	12	are any methods from the list:
	13
	14	create_ah
	15	modify_ah
	16	query_ah
	17	destroy_ah
	18	bind_mw
	19	post_send
	20	post_recv
	21	poll_cq
	22	req_notify_cq
	23	map_phys_fmr
	24
	25	which may not sleep and must be callable from any context.
	26
	27	The corresponding functions exported to upper level protocol
	28	consumers:
	29
	30	ib_create_ah
	31	ib_modify_ah
	32	ib_query_ah
	33	ib_destroy_ah
	34	ib_bind_mw
	35	ib_post_send
	36	ib_post_recv
	37	ib_req_notify_cq
	38	ib_map_phys_fmr
	39
	40	are therefore safe to call from any context.
	41
	42	In addition, the function
	43
	44	ib_dispatch_event
	45
	46	used by low-level drivers to dispatch asynchronous events through
	47	the midlayer is also safe to call from any context.
	48
	49	Reentrancy
	50
	51	All of the methods in struct ib_device exported by a low-level
	52	driver must be fully reentrant. The low-level driver is required to
	53	perform all synchronization necessary to maintain consistency, even
	54	if multiple function calls using the same object are run
	55	simultaneously.
	56
	57	The IB midlayer does not perform any serialization of function calls.
	58
	59	Because low-level drivers are reentrant, upper level protocol
	60	consumers are not required to perform any serialization. However,
	61	some serialization may be required to get sensible results. For
	62	example, a consumer may safely call ib_poll_cq() on multiple CPUs
	63	simultaneously. However, the ordering of the work completion
	64	information between different calls of ib_poll_cq() is not defined.
65
66	Callbacks
67
68	A low-level driver must not perform a callback directly from the
69	same callchain as an ib_device method call. For example, it is not
70	allowed for a low-level driver to call a consumer's completion event
71	handler directly from its post_send method. Instead, the low-level
72	driver should defer this callback by, for example, scheduling a
73	tasklet to perform the callback.
74
75	The low-level driver is responsible for ensuring that multiple
76	completion event handlers for the same CQ are not called
77	simultaneously. The driver must guarantee that only one CQ event
78	handler for a given CQ is running at a time. In other words, the
79	following situation is not allowed:
80
81	CPU1 CPU2
82
83	low-level driver ->
84	consumer CQ event callback:
85	/* ... */
86	ib_req_notify_cq(cq, ...);
87	low-level driver ->
88	/* ... */ consumer CQ event callback:
89	/* ... */
90	return from CQ event handler
91
92	The context in which completion event and asynchronous event
93	callbacks run is not defined. Depending on the low-level driver, it
94	may be process context, softirq context, or interrupt context.
95	Upper level protocol consumers may not sleep in a callback.
96
97	Hot-plug
98
99	A low-level driver announces that a device is ready for use by
100	consumers when it calls ib_register_device(), all initialization
101	must be complete before this call. The device must remain usable
102	until the driver's call to ib_unregister_device() has returned.
103
104	A low-level driver must call ib_register_device() and
105	ib_unregister_device() from process context. It must not hold any
106	semaphores that could cause deadlock if a consumer calls back into
107	the driver across these calls.
108
109	An upper level protocol consumer may begin using an IB device as
110	soon as the add method of its struct ib_client is called for that
111	device. A consumer must finish all cleanup and free all resources
112	relating to a device before returning from the remove method.
113
114	A consumer is permitted to sleep in its add and remove methods.