-
Notifications
You must be signed in to change notification settings - Fork 7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
usb: device_next: new USB Video Class (UVC) implementation #76798
base: main
Are you sure you want to change the base?
Conversation
I am grateful for the work on the USB and Video stacks of Zephyr, as well as the entire Zephyr tree, on the shoulder of which this is built. |
Force push:
|
Force-push:
|
.dwMinBitRate = sys_cpu_to_le32(15360000), \ | ||
.dwMaxBitRate = sys_cpu_to_le32(15360000), \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Where are these values coming from?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
These are chosen arbitrarily, and I need to think about what should I do here: pick reasonable defaults? Deduce from other values? Let the user figure out by exposing a devicetree option?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In future work, it will be possible to ask the sensor for VIDEO_CID_PIXEL_RATE
in combination with video_bits_per_pixel()
to generate this field.
.bLength = sizeof(struct usb_ep_descriptor), \ | ||
.bDescriptorType = USB_DESC_ENDPOINT, \ | ||
.bEndpointAddress = 0x81, \ | ||
.bmAttributes = USB_EP_TYPE_BULK, \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I believe UVC can be BULK or ISO.... but I only see BULK supported here. Do you plan to have a way to select which type of EP?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I forgot to mention that only BULK is supported through this early version. I will need to spend more time investigating the device_next
APIs for how ISO is implemented.
I cannot guarantee ETAs as I would do this feature on free time, but will add it to the roadmap: this will be needed at some point.
Very grateful for your reviews @XenuIsWatching I will force-push with the changes as soon as I get a chance to do so. |
Force-push:
|
182ac7c
to
cec26cc
Compare
02c1087
to
4bdc1ea
Compare
force push: rebased on |
force push: changed the way descriptors are declared and introduce video controls at the descriptor level (no support for controls commands yet) |
force push: implemented the UVC controls for the supported Video class controls. I did not test this yet with the samples, but on the internal fork, we could get an IMX219 sensor with exposure and gain control from the host, format selection at startup (but not runtime, see [1]). I believe that the last step for me is to test the sample on several boards that support |
Known limitations:[1]: there is currently no [2]: [3]: [4]: [5]: There are missing implementations for selector unit, extensions unit, encoding unit. TODO. [6]: UVC introduces dependencies between some controls, i.e. auto-expopsure needs to be off for exposure to be accepted. TODO. [7]: [8]: [9]: [10]: Supports custom header size, but not passing custom header data yet. [11]: Still image capture (capturing one frame at full resolution) not supported. [12]: USB3CV compliance tests not all passing since last rebase. TODO. [13]: Announcing different resolutions/FPS for different connection speed not supported. [14]: Asynchronous controls (the host setting a control, and a notification interrupt alert of completion) not supported. Supported features:[A]: Class API and enumeration [B]: [C]: [D]: [E]: Handling of control commands end-to-end from host to Zephyr video device [F]: Zephyr native Video API that allows to enqueue/dequeue frames like any other Zephyr video device. [G]: Supports fragmented frames as discussed in #66994 and #72827 [H]: [I]: Supports querying the min/max/current values of every controls end-to-end from host to Zephyr video driver. |
a7cd6cb
to
ea60ec0
Compare
CI just finished. Thanks for waiting. Force-push:
|
#define SET_CUR 0x01 | ||
#define GET_CUR 0x81 | ||
#define GET_MIN 0x82 | ||
#define GET_MAX 0x83 | ||
#define GET_RES 0x84 | ||
#define GET_LEN 0x85 | ||
#define GET_INFO 0x86 | ||
#define GET_DEF 0x87 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suggest to prefix them with UVC_ to minimize the risk of colliding with whatever madness the vendor's HALs contain.
/** | ||
* @brief Get the currently selected format and frame descriptors. | ||
* | ||
* @param strm The VideoStreaming interface from which search the descriptors | ||
* @param format_desc Pointer to be set to a format descriptor pointer. | ||
* @param frame_desc Pointer to be set to a frame descriptor pointer. | ||
*/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please never use doxygen format in c or local header files.
case GET_RES: | ||
case GET_MIN: | ||
probe->bFormatIndex = 1; | ||
break; | ||
case GET_MAX: | ||
probe->bFormatIndex = max; | ||
break; | ||
case GET_CUR: | ||
probe->bFormatIndex = strm->format_id; | ||
break; | ||
case SET_CUR: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GET_/SET_ must not be mixed, but must be handled in separate handlers. See my comment below about net_buf pointer.
|
||
static void uvc_update(struct usbd_class_data *const c_data, uint8_t iface, uint8_t alternate) | ||
{ | ||
LOG_DBG("update"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOG_DBG("Select alternate %u for interface %u", alternate, iface);
uint32_t max_size = MAX(p, w) * h; | ||
|
||
LOG_DBG("Adding frame descriptor #%u for %ux%u", | ||
format_desc->format.bNumFrameDescriptors + 1, (unsigned int)w, (unsigned int)h); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LOG_DBG("Adding frame descriptor #%u for %ux%u",
format_desc->format.bNumFrameDescriptors + 1, w, h);
@@ -11,4 +11,5 @@ supported: | |||
- gpio | |||
- spi | |||
- i2c | |||
- usbd |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please move it to a separate commit "boards: nicla_vision: add usbd test feature"
Force-push:
This is an early preview that was not tested yet, only there to give an idea. |
Force-push:
Current limitations (likely not due to this refactoring):
High-level outline:
|
Kind of sad this won't make it in 4.1 😢 |
Then maybe this is the opportunity to complete this refactoring, and introduce a more stable API right from the start... |
uint32_t dwMaxBitRate; | ||
uint32_t dwMaxVideoFrameBufferSize; | ||
uint32_t dwDefaultFrameInterval; | ||
uint8_t bFrameIntervalType; | ||
uint32_t dwMinFrameInterval; | ||
uint32_t dwMaxFrameInterval; | ||
uint32_t dwFrameIntervalStep; | ||
} __packed; | ||
|
||
struct uvc_frame_discrete_descriptor { | ||
uint8_t bLength; | ||
uint8_t bDescriptorType; | ||
uint8_t bDescriptorSubtype; | ||
uint8_t bFrameIndex; | ||
uint8_t bmCapabilities; | ||
uint16_t wWidth; | ||
uint16_t wHeight; | ||
uint32_t dwMinBitRate; | ||
uint32_t dwMaxBitRate; | ||
uint32_t dwMaxVideoFrameBufferSize; | ||
uint32_t dwDefaultFrameInterval; | ||
uint8_t bFrameIntervalType; | ||
uint32_t dwFrameInterval[CONFIG_USBD_VIDEO_MAX_FRMIVAL]; | ||
} __packed; | ||
|
||
union uvc_stream_descriptor { | ||
struct uvc_format_descriptor format; | ||
struct uvc_format_uncomp_descriptor format_uncomp; | ||
struct uvc_format_mjpeg_descriptor format_mjpeg; | ||
struct uvc_frame_discrete_descriptor frame_discrete; | ||
struct uvc_frame_continuous_descriptor frame_continuous; | ||
} __packed; | ||
|
||
struct uvc_color_descriptor { | ||
uint8_t bLength; | ||
uint8_t bDescriptorType; | ||
uint8_t bDescriptorSubtype; | ||
uint8_t bColorPrimaries; | ||
uint8_t bTransferCharacteristics; | ||
uint8_t bMatrixCoefficients; | ||
} __packed; | ||
|
||
struct usbd_uvc_desc { | ||
struct usb_association_descriptor iad; | ||
struct usb_if_descriptor if_vc; | ||
struct uvc_control_header_descriptor if_vc_header; | ||
struct usb_desc_header nil_desc; | ||
}; | ||
|
||
struct usbd_uvc_strm_desc { | ||
struct uvc_camera_terminal_descriptor if_vc_ct; | ||
struct uvc_selector_unit_descriptor if_vc_su; | ||
struct uvc_processing_unit_descriptor if_vc_pu; | ||
struct uvc_extension_unit_descriptor if_vc_xu; | ||
struct uvc_output_terminal_descriptor if_vc_ot; | ||
|
||
struct usb_if_descriptor if_vs; | ||
struct uvc_stream_header_descriptor if_vs_header; | ||
union uvc_stream_descriptor *if_vs_formats; | ||
size_t if_vs_format_num; | ||
struct usb_ep_descriptor if_vs_ep_fs; | ||
struct usb_ep_descriptor if_vs_ep_hs; | ||
struct uvc_color_descriptor if_vs_color; | ||
}; | ||
|
||
struct uvc_probe { | ||
uint16_t bmHint; | ||
uint8_t bFormatIndex; | ||
uint8_t bFrameIndex; | ||
uint32_t dwFrameInterval; | ||
uint16_t wKeyFrameRate; | ||
uint16_t wPFrameRate; | ||
uint16_t wCompQuality; | ||
uint16_t wCompWindowSize; | ||
uint16_t wDelay; | ||
uint32_t dwMaxVideoFrameSize; | ||
uint32_t dwMaxPayloadTransferSize; | ||
uint32_t dwClockFrequency; | ||
uint8_t bmFramingInfo; | ||
#define UVC_BMFRAMING_INFO_FID BIT(0) | ||
#define UVC_BMFRAMING_INFO_EOF BIT(1) | ||
#define UVC_BMFRAMING_INFO_EOS BIT(2) | ||
uint8_t bPreferedVersion; | ||
uint8_t bMinVersion; | ||
uint8_t bMaxVersion; | ||
uint8_t bUsage; | ||
uint8_t bBitDepthLuma; | ||
uint8_t bmSettings; | ||
uint8_t bMaxNumberOfRefFramesPlus1; | ||
uint16_t bmRateControlModes; | ||
uint64_t bmLayoutPerStream; | ||
} __packed; | ||
|
||
struct uvc_payload_header { | ||
uint8_t bHeaderLength; | ||
uint8_t bmHeaderInfo; | ||
uint32_t dwPresentationTime; /* optional */ | ||
uint32_t scrSourceClockSTC; /* optional */ | ||
uint16_t scrSourceClockSOF; /* optional */ | ||
} __packed; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please consider moving these 400+ lines into the subsys/usb/device_next/class/uvc.h header. This could be used later to write uvc tests and perhaps host driver implementation.
|
||
struct uvc_guid_quirk { | ||
uint32_t fourcc; | ||
uint8_t guid[16]; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why 16? Please add comment.
uint8_t selector; | ||
uint8_t size; | ||
uint8_t bit; | ||
enum uvc_op op; | ||
uint32_t param; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
?
uint32_t param;
enum uvc_op op;
uint8_t selector;
uint8_t size;
uint8_t bit;
NET_BUF_POOL_VAR_DEFINE(uvc_pool, DT_NUM_INST_STATUS_OKAY(DT_DRV_COMPAT) * 16, | ||
512 * DT_NUM_INST_STATUS_OKAY(DT_DRV_COMPAT) * 16, | ||
sizeof(struct uvc_buf_info), NULL); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a problem. It is not a DCACHE-aware buffer pool. I will look in the next days how to provide a macro that also takes into account user data length argument.
{.size = 1, .bit = 1, .selector = UVC_CT_AE_MODE_CONTROL, | ||
.op = UVC_OP_VC_FIXED, .param = BIT(0)}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are only a few places with these squeezed initialization. This is sometimes okay, but it does not follow the usual style and is inconsistent, please rework them.
static const struct uvc_control_map uvc_control_map_ct[] = {
{
.size = 1,
.bit = 1,
.selector = UVC_CT_AE_MODE_CONTROL,
.op = UVC_OP_VC_FIXED,
.param = BIT(0)
},
I have noticed some, mostly minor, issues, but fixing them during the stabilization period would probably be too stressful. @josuah I apologize for not noticing them earlier. Let's get this merged right after the release. |
Thanks @jfischer-no - makes sense. |
On my side, I have no issue with delay as tinyVision.ai uses an SDK where patches are cooked. About other users wanting to try it, I am still glad that the review is comprehensive, it will help making things JustWork™ out of the box, and fit better in P.S.: more force-push coming |
248d236
to
7516ec4
Compare
force-push:
known-limitations and issues:
The feedback helped improve the quality and reduce memory usage a little bit. 🙏 |
4a9347f
to
568ce73
Compare
Force-push:
|
Introduce a new USB Video Class (UVC) implementation from scratch. It exposes a native Zephyr Video driver interface, allowing to call the video_enqueue()/video_dequeue() interface. It will query the attached video device to learn about the pipeline capabilities, and use this to configure the USB descriptors. At runtime, this UVC implementation will send this device all the control requests, which it can then dispatch to the rest of the pipeline. The application can poll the format currently selected by the host, but will not be alerted when the host configures a new format, as there is no video.h API for it yet. Signed-off-by: Josuah Demangeon <[email protected]>
The Arduino Nicla Vision board supports the new device_next USB stack and can be used for testing USB Device features such as UVC. Signed-off-by: Josuah Demangeon <[email protected]>
Following the addition of USB Video Class, this adds a sample that makes use of the &zephyr,camera chosen node of any board to stream the video source to the host. A fallback video-emul.overlay is provided for test and debugging purpose for devices without a camera. Signed-off-by: Josuah Demangeon <[email protected]>
How to test:
The method to test with any
device_next
board:The method to test with any device with a
dcmi
node:It will use a very small test pattern as video source, so no camera is required.
To connect a camera instead of a test pattern, connect a different video source through the devicetree.
The USB descriptors will be updated by querying the driver.
Dependencies:
drivers: video: introduce "GET" sub-operations #78603replaced by something else to comedrivers: video: allow allocation with a header preceding the buffer #79168not required anymoreThis is a preview of an USB Video Class implementation for Zephyr that was only tested on custom devices insofar.
Proper Zephyr samples will be provided on upcoming commits. The API is simply to submit frames to the UVC device like to any Zephyr video device.
There is an unsolved challenge around the Video API: there is no
set_format
because the Zephyr application cannot decide what the host uses, onlyget_format
for what the host does support. But there is a missing video API to allow the driver to warn the application about a forced format change requested by the host. I thought of maybe reusingset_signal()
to also warn about format changes and not just buffer events.I will now work on building examples for existing Zephyr devices, as this was built for a custom USB3 board.
Here is the devicetree configuration used insofar:
The sizes and FPS are to be selected by the developer. The USB descriptors get configured as described above, and the host will request a particular format. Once that is done, the USB Video Class driver can let the application know which of these was selected through the video.hget_format()
API.This is still a draft PR, but I am grateful for comments and suggestions. I am willing to do the work of refactoring this as much as needed.
[EDIT: see this comment for latest description]