| 검색 | ?

DPDK(Data Plane Development Kit)

1.1. 개요

https://www.dpdk.org/about/
[PNG image (15.58 KB)]


DPDK(Data Plane Development Kit)는 EAL(Environment Abstraction Layer, 환경 추상화 계층)을 통해 다양한 CPU 아키텍처에서 실행되는 패킷 처리 계층을 가속화하기 위한 표준 인터페이스 라이브러리들로 구성된 프레임 워크입니다.

이 문서는 개인적으로 DPDK(Data Plane Development Kit)을 조사하면서 수집된 문서들을 바탕으로 이를 정리하는 것을 목표로 작성되었습니다. 경우에 따라서 개인적인 해석관점에서 작성되어 잘못된 내용이 있을 수 있다는 점은 염두하면서 읽어주시고 바로 잡아야 할 내용이 있으면 저에게 피드백을 요청드리면서 이 문서를 여러분과 함께 작성해보고자 하며 이미 선두에서 DPDK(Data Plane Development Kit)를 탄생시키고 학습하고 자료를 남겨주신 선배님들과 피드백을 주실 모든 분들에게 감사의 뜻을 전합니다.

현재 이 문서는 아직 학습 및 작성 중에 있는 문서입니다.

TODO: EAL, mempool, ring, mbuf, pmd, ...

1.2. DPDK 개발 환경

[https]Compilation of the DPDK[] 를 참고하실 수 있습니다.

기본 빌드 환경에 필요한 요구사항
  • GCC (version 5.0+) or Clang (version 3.6+)
  • pkg-config or pkgconf
  • Python 3.6+
  • Meson (version 0.57+)
  • pyelftools (version 0.22+)
  • libnuma library

Ubuntu/Debian Linux 환경 
$ apt install build-essential
$ apt install meson ninja-build
$ apt install python3-pyelftools
$ apt install libnuma-dev
$ apt install python3-sphinx python3-sphinx-rtd-theme doxygen


DPDK source 다운로드 및 빌드 실습 
$ git clone https://github.com/DPDK/dpdk.git MyDPDK
$ cd MyDPDK
$ meson setup -D enable_docs=true -D examples=all build
$ cd build
$ ninja
...
$ meson install --destdir "$(HOME)/MyRootFS"

1.3. DPDK 실행 환경

[https]Running DPDK Applications[] 를 참고하실 수 있습니다.

기본 실행 환경에 필요한 요구사항
  • Kernel version >= 4.19
    • HUGETLBFS
    • PROC_PAGE_MONITOR support
    • HPET and HPET_MMAP
  • glibc >= 2.7

1.4. EAL (Environment Abstraction Layer)

[https]Environment Abstraction Layer[] 를 참고하실 수 있습니다.

EAL (Environment Abstraction Layer, 환경 추상화 계층)은 DPDK의 Application 및 Library에서 실행되는 다양한 환경의 H/W 및 Memory space 같은 하위 수준의 자원(resources)등의 접근에 필요한 권한을 확보하도록 일반적인 추상화 인터페이스로 제공합니다.

EAL이 제공하는 기본적인 서비스는 다음과 같습니다.
  • DPDK loading and launching
  • Support for multi-process and multi-thread execution types
  • Core affinity/assignment procedures
  • System memory allocation/de-allocation
  • Atomic/lock operations
  • Time reference
  • PCI bus access
  • Trace and debug functions
  • CPU feature identification
  • Interrupt handling
  • Alarm operations
  • Memory management (malloc) 

int rc;

/* STEP: EAL 초기화 */
/* int rte_eal_init(int argc, char **argv); */
rc = rte_eal_init(argc, argv);
if (unlikely(rc < 0)) {
    /* void rte_exit(int exit_code, const char *format, ...); */
    rte_exit(EXIT_FAILURE, "ERROR: Invalid EAL parameters (errno=%d[\"%s\"])\n", rte_errno, rte_strerror(rte_errno));
}
/* STEP: EAL 초기화 완료 */
argc -= rc;
argv += rc;
/* STEP: EAL 옵션 이외의 옵션들을 처리 */

/*
 STEP: 이제 DPDK의 EAL 세상입니다... 관련 port / queue 등의 구성을 진행합니다.
*/

/* STEP: 이제 lcore 별 실행되는 자신만의 my_lcore_main 함수를 작성하고 Data 를 처리하는 lcore 별 main loop 를 작성합니다. */
/* int rte_eal_mp_remote_launch(lcore_function_t *f, void *arg, enum rte_rmt_call_main_t call_main); */
rc = rte_eal_mp_remote_launch(my_lcore_main /* int my_lcore_main(void *s_argument) */, my_arg, CALL_MAIN /* or SKIP_MAIN */);
if (rc == 0) {
    unsigned int lcore_id;

    /* 모든 lcore의 main 함수가 종료되기를 기다립니다. */
    RTE_LCORE_FOREACH_WORKER(lcore_id) {
        /* int rte_eal_wait_lcore(unsigned worker_id); */
        rc = rte_eal_wait_lcore(lcore_id);
        if (unlikely(rc < 0)) {
            RTE_LOG(ERR, EAL, "rte_eal_wait_lcore failed ! (lcore=%u)\n", lcore_id);
            break;
        }
    }
}

/* STEP: EAL 종료 */
/* int rte_eal_cleanup(void); */
rc = rte_eal_cleanup();
if (unlikely(rc != 0)) {
    rte_exit(EXIT_FAILURE, "ERROR: There was an error in releasing all resources (errno=%d[\"%s\"])\n", rte_errno, rte_strerror(rte_errno));
}

1.5. memzone

  • 기본 자료형 및 주요 함수 
    /* "lib/eal/include/rte_memzone.h" */

#define RTE_MEMZONE_2MB            0x00000001   /**< Use 2MB pages. */
#define RTE_MEMZONE_1GB            0x00000002   /**< Use 1GB pages. */
#define RTE_MEMZONE_16MB           0x00000100   /**< Use 16MB pages. */
#define RTE_MEMZONE_16GB           0x00000200   /**< Use 16GB pages. */
#define RTE_MEMZONE_256KB          0x00010000   /**< Use 256KB pages. */
#define RTE_MEMZONE_256MB          0x00020000   /**< Use 256MB pages. */
#define RTE_MEMZONE_512MB          0x00040000   /**< Use 512MB pages. */
#define RTE_MEMZONE_4GB            0x00080000   /**< Use 4GB pages. */
#define RTE_MEMZONE_SIZE_HINT_ONLY 0x00000004   /**< Use available page size */
#define RTE_MEMZONE_IOVA_CONTIG    0x00100000   /**< Ask for IOVA-contiguous memzone. */

struct rte_memzone {
#define RTE_MEMZONE_NAMESIZE 32       /**< Maximum length of memory zone name.*/
        char name[RTE_MEMZONE_NAMESIZE];  /**< Name of the memory zone. */
        rte_iova_t iova;                  /**< Start IO address. */
        union {
                void *addr;                   /**< Start virtual address. */
                uint64_t addr_64;             /**< Makes sure addr is always 64-bits */
        };
        size_t len;                       /**< Length of the memzone. */
        uint64_t hugepage_sz;             /**< The page size of underlying memory */
        int32_t socket_id;                /**< NUMA socket ID. */
        uint32_t flags;                   /**< Characteristics of this memzone. */
} __rte_packed;

/* 최대 memzone 영역 설정 또는 확인 */
int rte_memzone_max_set(size_t max);
size_t rte_memzone_max_get(void);

/* memzone 할당 및 해제 */
const struct rte_memzone *rte_memzone_reserve(const char *name, size_t len, int socket_id, unsigned flags);
const struct rte_memzone *rte_memzone_reserve_aligned(const char *name, size_t len, int socket_id, unsigned flags, unsigned align);
const struct rte_memzone *rte_memzone_reserve_bounded(const char *name, size_t len, int socket_id, unsigned flags, unsigned align, unsigned bound);
int rte_memzone_free(const struct rte_memzone *mz);

/* 이미 생성된 memzone 을 이름으로 찾기 */
const struct rte_memzone *rte_memzone_lookup(const char *name);

/* memzone 상황 dump */
void rte_memzone_dump(FILE *f);

/* 전체 memzone 들 순회 */
void rte_memzone_walk(void (*func)(const struct rte_memzone *, void *arg), void *arg);

1.6. mempool

  • 기본 자료형 및 주요 함수 
    /* "lib/mempool/rte_mempool.h" */
struct __rte_cache_aligned rte_mempool {
        char name[RTE_MEMPOOL_NAMESIZE]; /**< Name of mempool. */
        union {
                void *pool_data;         /**< Ring or pool to store objects. */
                uint64_t pool_id;        /**< External mempool identifier. */
        };
        void *pool_config;               /**< optional args for ops alloc. */
        const struct rte_memzone *mz;    /**< Memzone where pool is alloc'd. */
        unsigned int flags;              /**< Flags of the mempool. */
        int socket_id;                   /**< Socket id passed at create. */
        uint32_t size;                   /**< Max size of the mempool. */
        uint32_t cache_size;
        /**< Size of per-lcore default local cache. */

        uint32_t elt_size;               /**< Size of an element. */
        uint32_t header_size;            /**< Size of header (before elt). */
        uint32_t trailer_size;           /**< Size of trailer (after elt). */

        unsigned private_data_size;      /**< Size of private data. */
        /**
         * Index into rte_mempool_ops_table array of mempool ops
         * structs, which contain callback function pointers.
         * We're using an index here rather than pointers to the callbacks
         * to facilitate any secondary processes that may want to use
         * this mempool.
         */
        int32_t ops_index;

        struct rte_mempool_cache *local_cache; /**< Per-lcore local cache */

        uint32_t populated_size;         /**< Number of populated objects. */
        struct rte_mempool_objhdr_list elt_list; /**< List of objects in pool */
        uint32_t nb_mem_chunks;          /**< Number of memory chunks */
        struct rte_mempool_memhdr_list mem_list; /**< List of memory chunks */

#ifdef RTE_LIBRTE_MEMPOOL_STATS
        /** Per-lcore statistics.
         *
         * Plus one, for unregistered non-EAL threads.
         */
        struct rte_mempool_debug_stats stats[RTE_MAX_LCORE + 1];
#endif
};

typedef void (rte_mempool_obj_cb_t)(struct rte_mempool *mp, void *opaque, void *obj, unsigned obj_idx);
typedef void (rte_mempool_ctor_t)(struct rte_mempool *, void *);

/* mempool 생성/제거 */
struct rte_mempool *rte_mempool_create_empty(const char *name, unsigned n, unsigned elt_size, unsigned cache_size, unsigned private_data_size, int socket_id, unsigned flags);
struct rte_mempool *rte_mempool_create(const char *name, unsigned n, unsigned elt_size, unsigned cache_size, unsigned private_data_size, rte_mempool_ctor_t *mp_init, void *mp_init_arg, rte_mempool_obj_cb_t *obj_init, void *obj_init_arg, int socket_id, unsigned flags);
void rte_mempool_free(struct rte_mempool *mp);

int rte_mempool_populate_default(struct rte_mempool *mp);

/* mempool object 들에 대하여 호출 */
uint32_t rte_mempool_obj_iter(struct rte_mempool *mp, rte_mempool_obj_cb_t *obj_cb, void *obj_cb_arg);

/* 주어진 이름 또는 object로부터 mempool을 반환 */
struct rte_mempool *rte_mempool_lookup(const char *name);
static inline struct rte_mempool *rte_mempool_from_obj(void *obj);

/* mempool의 할당 상황 */ 
unsigned int rte_mempool_avail_count(const struct rte_mempool *mp);
unsigned int rte_mempool_in_use_count(const struct rte_mempool *mp);
static inline int rte_mempool_full(const struct rte_mempool *mp);
static inline int rte_mempool_empty(const struct rte_mempool *mp);

/* 주어진 n개만큼의 object를 할당 및 해제 */
static __rte_always_inline int rte_mempool_get_bulk(struct rte_mempool *mp, void **obj_table, unsigned int n);
static __rte_always_inline void rte_mempool_put_bulk(struct rte_mempool *mp, void * const *obj_table, unsigned int n);

/* 한개의 object를 할당 및 해제 */
static __rte_always_inline int rte_mempool_get(struct rte_mempool *mp, void **obj_p);
static __rte_always_inline void rte_mempool_put(struct rte_mempool *mp, void *obj);

/* 모든 mempool 을 순회하면서 호출 */
void rte_mempool_walk(void (*func)(struct rte_mempool *, void *arg), void *arg);

1.7. ring

  • 기본 자료형 및 주요 함수 
    /* "lib/ring/rte_ring_core.h" */
enum rte_ring_queue_behavior {
        RTE_RING_QUEUE_FIXED = 0,
        RTE_RING_QUEUE_VARIABLE
};
#define RTE_RING_MZ_PREFIX "RG_"
#define RTE_RING_NAMESIZE (RTE_MEMZONE_NAMESIZE - sizeof(RTE_RING_MZ_PREFIX) + 1)

enum rte_ring_sync_type {
        RTE_RING_SYNC_MT,     /**< multi-thread safe (default mode) */
        RTE_RING_SYNC_ST,     /**< single thread only */
        RTE_RING_SYNC_MT_RTS, /**< multi-thread relaxed tail sync */
        RTE_RING_SYNC_MT_HTS, /**< multi-thread head/tail sync */
};

...

struct rte_ring {
        alignas(RTE_CACHE_LINE_SIZE) char name[RTE_RING_NAMESIZE];
        int flags;               /**< Flags supplied at creation. */
        const struct rte_memzone *memzone; /**< Memzone, if any, containing the rte_ring */
        uint32_t size;           /**< Size of ring. */
        uint32_t mask;           /**< Mask (size-1) of ring. */
        uint32_t capacity;       /**< Usable size of ring */
        RTE_CACHE_GUARD;
        /** Ring producer status. */
        union __rte_cache_aligned {
                struct rte_ring_headtail prod;
                struct rte_ring_hts_headtail hts_prod;
                struct rte_ring_rts_headtail rts_prod;
        };
        RTE_CACHE_GUARD;
        /** Ring consumer status. */
        union __rte_cache_aligned {
                struct rte_ring_headtail cons;
                struct rte_ring_hts_headtail hts_cons;
                struct rte_ring_rts_headtail rts_cons;
        };
        RTE_CACHE_GUARD;
};

#define RING_F_SP_ENQ 0x0001 /**< The default enqueue is "single-producer". */
#define RING_F_SC_DEQ 0x0002 /**< The default dequeue is "single-consumer". */

#define RING_F_EXACT_SZ 0x0004
#define RTE_RING_SZ_MASK  (0x7fffffffU) /**< Ring size mask */

#define RING_F_MP_RTS_ENQ 0x0008 /**< The default enqueue is "MP RTS". */
#define RING_F_MC_RTS_DEQ 0x0010 /**< The default dequeue is "MC RTS". */

#define RING_F_MP_HTS_ENQ 0x0020 /**< The default enqueue is "MP HTS". */
#define RING_F_MC_HTS_DEQ 0x0040 /**< The default dequeue is "MC HTS". */

/* "lib/ring/rte_ring.h" */

ssize_t rte_ring_get_memsize(unsigned int count);

int rte_ring_init(struct rte_ring *r, const char *name, unsigned int count, unsigned int flags);
struct rte_ring *rte_ring_create(const char *name, unsigned int count, int socket_id, unsigned int flags);
void rte_ring_free(struct rte_ring *r);

void rte_ring_dump(FILE *f, const struct rte_ring *r);

static __rte_always_inline unsigned int rte_ring_mp_enqueue_bulk(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline unsigned int rte_ring_sp_enqueue_bulk(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline unsigned int rte_ring_enqueue_bulk(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline int rte_ring_mp_enqueue(struct rte_ring *r, void *obj)
static __rte_always_inline int rte_ring_sp_enqueue(struct rte_ring *r, void *obj)
static __rte_always_inline unsigned int rte_ring_mc_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)
static __rte_always_inline unsigned int rte_ring_sc_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)
static __rte_always_inline unsigned int rte_ring_dequeue_bulk(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)
static __rte_always_inline int rte_ring_mc_dequeue(struct rte_ring *r, void **obj_p)
static __rte_always_inline int rte_ring_sc_dequeue(struct rte_ring *r, void **obj_p)
static __rte_always_inline int rte_ring_dequeue(struct rte_ring *r, void **obj_p)

void rte_ring_reset(struct rte_ring *r);

static inline unsigned int rte_ring_count(const struct rte_ring *r)
static inline unsigned int rte_ring_free_count(const struct rte_ring *r)
static inline int rte_ring_full(const struct rte_ring *r)
static inline int rte_ring_empty(const struct rte_ring *r)
static inline unsigned int rte_ring_get_size(const struct rte_ring *r)
static inline unsigned int rte_ring_get_capacity(const struct rte_ring *r)
static inline enum rte_ring_sync_type rte_ring_get_prod_sync_type(const struct rte_ring *r)
static inline int rte_ring_is_prod_single(const struct rte_ring *r)
static inline enum rte_ring_sync_type rte_ring_get_cons_sync_type(const struct rte_ring *r)
static inline int rte_ring_is_cons_single(const struct rte_ring *r)

void rte_ring_list_dump(FILE *f);

struct rte_ring *rte_ring_lookup(const char *name);

static __rte_always_inline unsigned int rte_ring_mp_enqueue_burst(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline unsigned int rte_ring_sp_enqueue_burst(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline unsigned int rte_ring_enqueue_burst(struct rte_ring *r, void * const *obj_table, unsigned int n, unsigned int *free_space)
static __rte_always_inline unsigned int rte_ring_mc_dequeue_burst(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)
static __rte_always_inline unsigned int rte_ring_sc_dequeue_burst(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)
static __rte_always_inline unsigned int rte_ring_dequeue_burst(struct rte_ring *r, void **obj_table, unsigned int n, unsigned int *available)

1.8. mbuf

1.9. RCU(Read-Copy-Update)

1.10. hash

1.11. LPM(Longerst Prefix Match)

1.12. ACL(Access Control List)

1.13. ip_frag

1.14. compressdev

1.15. cryptodev

1.16. security

1.17. ipsec

1.18. ethdev

1.19. eventdev

1.20. 참고자료


Copyright ⓒ MINZKN.COM
All Rights Reserved.