Today, I used doca_dev.h API to enumerate PCI functions of the BlueField device.

  • Understand the architecture
  • DOCA Core primitives
    • Device enumerations with doca_devinfo
  • Run sample applications
    • Regex offload sample
    • Compression sample: doca_compress

To acquire the information of each interface of the device, first I had to call the function doca_devinfo_create_list. It creates an array of struct doca_devinfo. Then, while iterating through the array, I called doca_dev_open function to retrieve struct doca_dev of the device.

The struct struct doca_dev represents a host/local device that is visible to the current execution context. While on the other hand, there is a struct called struct doca_dev_rep which is a logical stand-in for a device that lives on the other side of the PCIe/DPU boundary.

To add more description to struct doca_dev_rep, Claude told me that struct doca_dev_rep is a proxy or a mirror of the remote endpoint that is accessible from the DPU.

So I made a small DOCA program that prints some information of the devices (struct doca_dev) and the representor devices (struct doca_dev_rep). This is the output:

ubuntu@localhost:~/learn$ sudo ./doca_dev
0000:03:00.0 p0 0.0.0.0 94:6D:AE:CF:D2:24 PF
-> 0000:3b:00.0 pf0hpf PF
-> 0000:03:00.0 en3f0pf0sf0 SF
0000:03:00.1 p1 0.0.0.0 94:6D:AE:CF:D2:25 PF
-> 0000:3b:00.1 pf1hpf PF
-> 0000:03:00.1 en3f1pf1sf0 SF
0000:03:00.0 enp3s0f0s0 0.0.0.0 02:7F:C2:BA:06:95 SF
0000:03:00.1 enp3s0f1s0 0.0.0.0 02:51:CE:BC:12:FE SF

As we can see, the interface p0 and p1 is present in this device, and their PCIe address is 03:00.0 and 03:00.1, respectively. The representor devices of p0 is pf0hpf, a PF (Physical Function), and en3f0pf0sf0, a SF (Sub-Function).

A physical function is an actual PCIe device exposed, and it is the real hardware endpoint, which means that one cannot create or destory PFs in software. However, a sub-function is a software-defined version of PF. Many SFs can be created per PF, and each SF gets its own dedicated hardware resources.