Skip to content
代码片段 群组 项目

比较版本

更改显示为版本正在合并到目标版本。了解更多关于比较版本的信息。

来源

选择目标项目
No results found

目标

选择目标项目
  • wireguard/wireguard-linux-compat
  • apsara2825/wireguard-linux-compat
2 个结果
显示更改
源代码提交(35)
  • Jason A. Donenfeld's avatar
    allowedips: free empty intermediate nodes when removing single node · 383461db
    Jason A. Donenfeld 创作于
    
    When removing single nodes, it's possible that that node's parent is an
    empty intermediate node, in which case, it too should be removed.
    Otherwise the trie fills up and never is fully emptied, leading to
    gradual memory leaks over time for tries that are modified often. There
    was originally code to do this, but was removed during refactoring in
    2016 and never reworked. Now that we have proper parent pointers from
    the previous commits, we can implement this properly.
    
    In order to reduce branching and expensive comparisons, we want to keep
    the double pointer for parent assignment (which lets us easily chain up
    to the root), but we still need to actually get the parent's base
    address. So encode the bit number into the last two bits of the pointer,
    and pack and unpack it as needed. This is a little bit clumsy but is the
    fastest and less memory wasteful of the compromises. Note that we align
    the root struct here to a minimum of 4, because it's embedded into a
    larger struct, and we're relying on having the bottom two bits for our
    flag, which would only be 16-bit aligned on m68k.
    
    The existing macro-based helpers were a bit unwieldy for adding the bit
    packing to, so this commit replaces them with safer and clearer ordinary
    functions.
    
    We add a test to the randomized/fuzzer part of the selftests, to free
    the randomized tries by-peer, refuzz it, and repeat, until it's supposed
    to be empty, and then then see if that actually resulted in the whole
    thing being emptied. That combined with kmemcheck should hopefully make
    sure this commit is doing what it should. Along the way this resulted in
    various other cleanups of the tests and fixes for recent graphviz.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    383461db
  • Jason A. Donenfeld's avatar
    allowedips: add missing __rcu annotation to satisfy sparse · fd7a4621
    Jason A. Donenfeld 创作于
    
    A __rcu annotation got lost during refactoring, which caused sparse to
    become enraged.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    fd7a4621
  • Jason A. Donenfeld's avatar
    qemu: add disgusting hacks for RHEL 8 · 8f4414d3
    Jason A. Donenfeld 创作于
    
    Red Hat does awful things to their kernel for RHEL 8, such that it
    doesn't even compile in most configurations. This is utter craziness,
    and their response to me sending patches to fix this stuff has been to
    stonewall for months on end and then do nothing.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    8f4414d3
  • Jason A. Donenfeld's avatar
    qemu: increase default dmesg log size · fb4a0da6
    Jason A. Donenfeld 创作于
    
    The selftests currently parse the kernel log at the end to track
    potential memory leaks. With these tests now reading off the end of the
    buffer, due to recent optimizations, some creation messages were lost,
    making the tests think that there was a free without an alloc. Fix this
    by increasing the kernel log size.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    fb4a0da6
  • Jason A. Donenfeld's avatar
    version: bump · d378f930
    Jason A. Donenfeld 创作于
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    d378f930
  • Jason A. Donenfeld's avatar
    50dda8ce
  • Mathias Krause's avatar
    compat: account for grsecurity backports and changes · 29747255
    Mathias Krause 创作于
    
    grsecurity kernels tend to carry additional backports and changes, like
    commit b60b87fc2996 ("netlink: add ethernet address policy types") or
    the SYM_FUNC_* changes. RAP nowadays hooks the latter, therefore no
    diversion to RAP_ENTRY is needed any more.
    
    Instead of relying on the kernel version test, also test for the macros
    we're about to define to not already be defined to account for these
    additional changes in the grsecurity patch without breaking
    compatibility to the older public ones.
    
    Also test for CONFIG_PAX instead of RAP_PLUGIN for the timer API related
    changes as these don't depend on the RAP plugin to be enabled but just a
    PaX/grsecurity patch to be applied. While there is no preprocessor knob
    for the latter, use CONFIG_PAX as this will likely be enabled in every
    kernel that uses the patch.
    
    Signed-off-by: default avatarMathias Krause <minipli@grsecurity.net>
    [zx2c4: small changes to include a header nearby a macro def test]
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    29747255
  • Peter Georg's avatar
    compat: update for RHEL 8.5 · 2715e641
    Peter Georg 创作于
    
    RHEL 8.5 has been released. Replace all ISCENTOS8S checks with ISRHEL8.
    Increase RHEL_MINOR for CentOS 8 Stream detection to 6.
    
    Signed-off-by: default avatarPeter Georg <peter.georg@physik.uni-regensburg.de>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    2715e641
  • Jason A. Donenfeld's avatar
    netns: actually test for routing loops · cb001d45
    Jason A. Donenfeld 创作于
    
    We previously removed the restriction on looping to self, and then added
    a test to make sure the kernel didn't blow up during a routing loop. The
    kernel didn't blow up, thankfully, but on certain architectures where
    skb fragmentation is easier, such as ppc64, the skbs weren't actually
    being discarded after a few rounds through. But the test wasn't catching
    this. So actually test explicitly for massive increases in tx to see if
    we have a routing loop. Note that the actual loop problem will need to
    be addressed in a different commit.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    cb001d45
  • Randy Dunlap's avatar
    main: rename 'mod_init' & 'mod_exit' functions to be module-specific · ea3f5fbe
    Randy Dunlap 创作于
    
    Rename module_init & module_exit functions that are named
    "mod_init" and "mod_exit" so that they are unique in both the
    System.map file and in initcall_debug output instead of showing
    up as almost anonymous "mod_init".
    
    This is helpful for debugging and in determining how long certain
    module_init calls take to execute.
    
    Signed-off-by: default avatarRandy Dunlap <rdunlap@infradead.org>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    ea3f5fbe
  • Jason A. Donenfeld's avatar
    device: reset peer src endpoint when netns exits · 68abb1b9
    Jason A. Donenfeld 创作于
    
    Each peer's endpoint contains a dst_cache entry that takes a reference
    to another netdev. When the containing namespace exits, we take down the
    socket and prevent future sockets from being created (by setting
    creating_net to NULL), which removes that potential reference on the
    netns. However, it doesn't release references to the netns that a netdev
    cached in dst_cache might be taking, so the netns still might fail to
    exit. Since the socket is gimped anyway, we can simply clear all the
    dst_caches (by way of clearing the endpoint src), which will release all
    references.
    
    However, the current dst_cache_reset function only releases those
    references lazily. But it turns out that all of our usages of
    wg_socket_clear_peer_endpoint_src are called from contexts that are not
    exactly high-speed or bottle-necked. For example, when there's
    connection difficulty, or when userspace is reconfiguring the interface.
    And in particular for this patch, when the netns is exiting. So for
    those cases, it makes more sense to call dst_release immediately. For
    that, we add a small helper function to dst_cache.
    
    This patch also adds a test to netns.sh from Hangbin Liu to ensure this
    doesn't regress.
    
    Test-by: default avatarHangbin Liu <liuhangbin@gmail.com>
    Reported-by: default avatarXiumei Mu <xmu@redhat.com>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    68abb1b9
  • Jason A. Donenfeld's avatar
    receive: use ring buffer for incoming handshakes · 5707d38f
    Jason A. Donenfeld 创作于
    
    Apparently the spinlock on incoming_handshake's skb_queue is highly
    contended, and a torrent of handshake or cookie packets can bring the
    data plane to its knees, simply by virtue of enqueueing the handshake
    packets to be processed asynchronously. So, we try switching this to a
    ring buffer to hopefully have less lock contention. This alleviates the
    problem somewhat, though it still isn't perfect, so future patches will
    have to improve this further. However, it at least doesn't completely
    diminish the data plane.
    
    Reported-by: default avatarStreun Fabio <fstreun@student.ethz.ch>
    Reported-by: default avatarJoel Wanner <joel.wanner@inf.ethz.ch>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    5707d38f
  • Jason A. Donenfeld's avatar
    receive: drop handshakes if queue lock is contended · e44c78cb
    Jason A. Donenfeld 创作于
    
    If we're being delivered packets from multiple CPUs so quickly that the
    ring lock is contended for CPU tries, then it's safe to assume that the
    queue is near capacity anyway, so just drop the packet rather than
    spinning. This helps deal with multicore DoS that can interfere with
    data path performance. It _still_ does not completely fix the issue, but
    it again chips away at it.
    
    Reported-by: default avatarStreun Fabio <fstreun@student.ethz.ch>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    e44c78cb
  • Gustavo A. R. Silva's avatar
    ratelimiter: use kvcalloc() instead of kvzalloc() · 5325bc82
    Gustavo A. R. Silva 创作于
    
    Use 2-factor argument form kvcalloc() instead of kvzalloc().
    
    Signed-off-by: default avatarGustavo A. R. Silva <gustavoars@kernel.org>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    5325bc82
  • Arnd Bergmann's avatar
    compat: siphash: use _unaligned version by default · ea6b8e7b
    Arnd Bergmann 创作于
    On ARM v6 and later, we define CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
    because the ordinary load/store instructions (ldr, ldrh, ldrb) can
    tolerate any misalignment of the memory address. However, load/store
    double and load/store multiple instructions (ldrd, ldm) may still only
    be used on memory addresses that are 32-bit aligned, and so we have to
    use the CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS macro with care, or we
    may end up with a severe performance hit due to alignment traps that
    require fixups by the kernel. Testing shows that this currently happens
    with clang-13 but not gcc-11. In theory, any compiler version can
    produce this bug or other problems, as we are dealing with undefined
    behavior in C99 even on architectures that support this in hardware,
    see also https://gcc.gnu.org/bugzilla/show_bug.cgi?id=100363
    
    .
    
    Fortunately, the get_unaligned() accessors do the right thing: when
    building for ARMv6 or later, the compiler will emit unaligned accesses
    using the ordinary load/store instructions (but avoid the ones that
    require 32-bit alignment). When building for older ARM, those accessors
    will emit the appropriate sequence of ldrb/mov/orr instructions. And on
    architectures that can truly tolerate any kind of misalignment, the
    get_unaligned() accessors resolve to the leXX_to_cpup accessors that
    operate on aligned addresses.
    
    Since the compiler will in fact emit ldrd or ldm instructions when
    building this code for ARM v6 or later, the solution is to use the
    unaligned accessors unconditionally on architectures where this is
    known to be fast. The _aligned version of the hash function is
    however still needed to get the best performance on architectures
    that cannot do any unaligned access in hardware.
    
    This new version avoids the undefined behavior and should produce
    the fastest hash on all architectures we support.
    
    Reported-by: default avatarArd Biesheuvel <ard.biesheuvel@linaro.org>
    Signed-off-by: default avatarArnd Bergmann <arnd@arndb.de>
    Reviewed-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    Acked-by: default avatarArd Biesheuvel <ardb@kernel.org>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    ea6b8e7b
  • Jason A. Donenfeld's avatar
    compat: udp_tunnel: don't take reference to non-init namespace · 8e40dd62
    Jason A. Donenfeld 创作于
    
    The comment to sk_change_net is instructive:
    
      Kernel sockets, f.e. rtnl or icmp_socket, are a part of a namespace.
      They should not hold a reference to a namespace in order to allow
      to stop it.
      Sockets after sk_change_net should be released using sk_release_kernel
    
    We weren't following these rules before, and were instead using
    __sock_create, which means we kept a reference to the namespace, which
    in turn meant that interfaces were not cleaned up on namespace
    exit.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    8e40dd62
  • Mathias Krause's avatar
    crypto: curve25519-x86_64: solve register constraints with reserved registers · 3c9f3b69
    Mathias Krause 创作于
    
    The register constraints for the inline assembly in fsqr() and fsqr2()
    are pretty tight on what the compiler may assign to the remaining three
    register variables. The clobber list only allows the following to be
    used: RDI, RSI, RBP and R12. With RAP reserving R12 and a kernel having
    CONFIG_FRAME_POINTER=y, claiming RBP, there are only two registers left
    so the compiler rightfully complains about impossible constraints.
    
    Provide alternatives that'll allow a memory reference for 'out' to solve
    the allocation constraint dilemma for this configuration.
    
    Also make 'out' an input-only operand as it is only used as such. This
    not only allows gcc to optimize its usage further, but also works around
    older gcc versions, apparently failing to handle multiple alternatives
    correctly, as in failing to initialize the 'out' operand with its input
    value.
    
    Signed-off-by: default avatarMathias Krause <minipli@grsecurity.net>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    3c9f3b69
  • Jason A. Donenfeld's avatar
    version: bump · 743eef23
    Jason A. Donenfeld 创作于
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    743eef23
  • Jason A. Donenfeld's avatar
    compat: drop Ubuntu 14.04 · 4f4c0198
    Jason A. Donenfeld 创作于
    It's been over a year since we announced sunsetting this.
    
    Link: https://lore.kernel.org/wireguard/CAHmME9rckipsdZYW+LA=x6wCMybdFFA+VqoogFXnR=kHYiCteg@mail.gmail.com/T
    
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    4f4c0198
  • Jason A. Donenfeld's avatar
    crypto: curve25519-x86_64: use in/out register constraints more precisely · 273018b7
    Jason A. Donenfeld 创作于
    Rather than passing all variables as modified, pass ones that are only
    read into that parameter. This helps with old gcc versions when
    alternatives are additionally used, and lets gcc's codegen be a little
    bit more efficient. This also syncs up with the latest Vale/EverCrypt
    output.
    
    This also forward ports 3c9f3b69 ("crypto: curve25519-x86_64: solve
    register constraints with reserved registers").
    
    Cc: Aymeric Fromherz <aymeric.fromherz@inria.fr>
    Cc: Mathias Krause <minipli@grsecurity.net>
    Link: https://lore.kernel.org/wireguard/1554725710.1290070.1639240504281.JavaMail.zimbra@inria.fr/
    Link: https://github.com/project-everest/hacl-star/pull/501
    
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    273018b7
  • Jason A. Donenfeld's avatar
    queueing: use CFI-safe ptr_ring cleanup function · 4eff63d2
    Jason A. Donenfeld 创作于
    
    We make too nuanced use of ptr_ring to entirely move to the skb_array
    wrappers, but we at least should avoid the naughty function pointer cast
    when cleaning up skbs. Otherwise RAP/CFI will honk at us. This patch
    uses the __skb_array_destroy_skb wrapper for the cleanup, rather than
    directly providing kfree_skb, which is what other drivers in the same
    situation do too.
    
    Reported-by: default avatarPaX Team <pageexec@freemail.hu>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    4eff63d2
  • Jason A. Donenfeld's avatar
    qemu: simplify RNG seeding · ffb8cd62
    Jason A. Donenfeld 创作于
    
    We don't actualy need to write anything in the pool. Instead, we just
    force the total over 128, and we should be good to go for all old
    kernels. We also only need this on getrandom() kernels, which simplifies
    things too.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    ffb8cd62
  • Wang Hai's avatar
    socket: free skb in send6 when ipv6 is disabled · fa32671b
    Wang Hai 创作于
    
    I got a memory leak report:
    
    unreferenced object 0xffff8881191fc040 (size 232):
      comm "kworker/u17:0", pid 23193, jiffies 4295238848 (age 3464.870s)
      hex dump (first 32 bytes):
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  ................
      backtrace:
        [<ffffffff814c3ef4>] slab_post_alloc_hook+0x84/0x3b0
        [<ffffffff814c8977>] kmem_cache_alloc_node+0x167/0x340
        [<ffffffff832974fb>] __alloc_skb+0x1db/0x200
        [<ffffffff82612b5d>] wg_socket_send_buffer_to_peer+0x3d/0xc0
        [<ffffffff8260e94a>] wg_packet_send_handshake_initiation+0xfa/0x110
        [<ffffffff8260ec81>] wg_packet_handshake_send_worker+0x21/0x30
        [<ffffffff8119c558>] process_one_work+0x2e8/0x770
        [<ffffffff8119ca2a>] worker_thread+0x4a/0x4b0
        [<ffffffff811a88e0>] kthread+0x120/0x160
        [<ffffffff8100242f>] ret_from_fork+0x1f/0x30
    
    In function wg_socket_send_buffer_as_reply_to_skb() or wg_socket_send_
    buffer_to_peer(), the semantics of send6() is required to free skb. But
    when CONFIG_IPV6 is disable, kfree_skb() is missing. This patch adds it
    to fix this bug.
    
    Signed-off-by: default avatarWang Hai <wanghai38@huawei.com>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    fa32671b
  • Jason A. Donenfeld's avatar
    socket: ignore v6 endpoints when ipv6 is disabled · ec89ca64
    Jason A. Donenfeld 创作于
    
    The previous commit fixed a memory leak on the send path in the event
    that IPv6 is disabled at compile time, but how did a packet even arrive
    there to begin with? It turns out we have previously allowed IPv6
    endpoints even when IPv6 support is disabled at compile time. This is
    awkward and inconsistent. Instead, let's just ignore all things IPv6,
    the same way we do other malformed endpoints, in the case where IPv6 is
    disabled.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    ec89ca64
  • Jason A. Donenfeld's avatar
    qemu: enable ACPI for SMP · f909532a
    Jason A. Donenfeld 创作于
    
    It turns out that by having CONFIG_ACPI=n, we've been failing to boot
    additional CPUs, and so these systems were functionally UP. The code
    bloat is unfortunate for build times, but I don't see an alternative. So
    this commit sets CONFIG_ACPI=y for x86_64 and i686 configs.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    f909532a
  • Nikolay Aleksandrov's avatar
    device: check for metadata_dst with skb_valid_dst() · f9d9b4db
    Nikolay Aleksandrov 创作于
    When we try to transmit an skb with md_dst attached through wireguard
    we hit a null pointer dereference in wg_xmit() due to the use of
    dst_mtu() which calls into dst_blackhole_mtu() which in turn tries to
    dereference dst->dev.
    
    Since wireguard doesn't use md_dsts we should use skb_valid_dst(), which
    checks for DST_METADATA flag, and if it's set, then falls back to
    wireguard's device mtu. That gives us the best chance of transmitting
    the packet; otherwise if the blackhole netdev is used we'd get
    ETH_MIN_MTU.
    
     [  263.693506] BUG: kernel NULL pointer dereference, address: 00000000000000e0
     [  263.693908] #PF: supervisor read access in kernel mode
     [  263.694174] #PF: error_code(0x0000) - not-present page
     [  263.694424] PGD 0 P4D 0
     [  263.694653] Oops: 0000 [#1] PREEMPT SMP NOPTI
     [  263.694876] CPU: 5 PID: 951 Comm: mausezahn Kdump: loaded Not tainted 5.18.0-rc1+ #522
     [  263.695190] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1.fc35 04/01/2014
     [  263.695529] RIP: 0010:dst_blackhole_mtu+0x17/0x20
     [  263.695770] Code: 00 00 00 0f 1f 44 00 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 8b 47 10 48 83 e0 fc 8b 40 04 85 c0 75 09 48 8b 07 <8b> 80 e0 00 00 00 c3 66 90 0f 1f 44 00 00 48 89 d7 be 01 00 00 00
     [  263.696339] RSP: 0018:ffffa4a4422fbb28 EFLAGS: 00010246
     [  263.696600] RAX: 0000000000000000 RBX: ffff8ac9c3553000 RCX: 0000000000000000
     [  263.696891] RDX: 0000000000000401 RSI: 00000000fffffe01 RDI: ffffc4a43fb48900
     [  263.697178] RBP: ffffa4a4422fbb90 R08: ffffffff9622635e R09: 0000000000000002
     [  263.697469] R10: ffffffff9b69a6c0 R11: ffffa4a4422fbd0c R12: ffff8ac9d18b1a00
     [  263.697766] R13: ffff8ac9d0ce1840 R14: ffff8ac9d18b1a00 R15: ffff8ac9c3553000
     [  263.698054] FS:  00007f3704c337c0(0000) GS:ffff8acaebf40000(0000) knlGS:0000000000000000
     [  263.698470] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
     [  263.698826] CR2: 00000000000000e0 CR3: 0000000117a5c000 CR4: 00000000000006e0
     [  263.699214] Call Trace:
     [  263.699505]  <TASK>
     [  263.699759]  wg_xmit+0x411/0x450
     [  263.700059]  ? bpf_skb_set_tunnel_key+0x46/0x2d0
     [   263.700382]  ? dev_queue_xmit_nit+0x31/0x2b0
     [  263.700719]  dev_hard_start_xmit+0xd9/0x220
     [  263.701047]  __dev_queue_xmit+0x8b9/0xd30
     [  263.701344]  __bpf_redirect+0x1a4/0x380
     [  263.701664]  __dev_queue_xmit+0x83b/0xd30
     [  263.701961]  ? packet_parse_headers+0xb4/0xf0
     [  263.702275]  packet_sendmsg+0x9a8/0x16a0
     [  263.702596]  ? _raw_spin_unlock_irqrestore+0x23/0x40
     [  263.702933]  sock_sendmsg+0x5e/0x60
     [  263.703239]  __sys_sendto+0xf0/0x160
     [  263.703549]  __x64_sys_sendto+0x20/0x30
     [  263.703853]  do_syscall_64+0x3b/0x90
     [  263.704162]  entry_SYSCALL_64_after_hwframe+0x44/0xae
     [  263.704494] RIP: 0033:0x7f3704d50506
     [  263.704789] Code: 48 c7 c0 ff ff ff ff eb b7 66 2e 0f 1f 84 00 00 00 00 00 90 41 89 ca 64 8b 04 25 18 00 00 00 85 c0 75 11 b8 2c 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 72 c3 90 55 48 83 ec 30 44 89 4c 24 2c 4c 89
     [  263.705652] RSP: 002b:00007ffe954b0b88 EFLAGS: 00000246 ORIG_RAX: 000000000000002c
     [  263.706141] RAX: ffffffffffffffda RBX: 0000558bb259b490 RCX: 00007f3704d50506
     [  263.706544] RDX: 000000000000004a RSI: 0000558bb259b7b2 RDI: 0000000000000003
     [  263.706952] RBP: 0000000000000000 R08: 00007ffe954b0b90 R09: 0000000000000014
     [  263.707339] R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffe954b0b90
     [  263.707735] R13: 000000000000004a R14: 0000558bb259b7b2 R15: 0000000000000001
     [  263.708132]  </TASK>
     [  263.708398] Modules linked in: bridge netconsole bonding [last unloaded: bridge]
     [  263.708942] CR2: 00000000000000e0
    
    Link: https://github.com/cilium/cilium/issues/19428
    
    
    Reported-by: default avatarMartynas Pumputis <m@lambda.lt>
    Signed-off-by: default avatarNikolay Aleksandrov <razor@blackwall.org>
    Acked-by: default avatarDaniel Borkmann <daniel@iogearbox.net>
    [Jason: polyfilled for < 4.3]
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    f9d9b4db
  • Jason A. Donenfeld's avatar
    netns: make routing loop test non-fatal · f8886735
    Jason A. Donenfeld 创作于
    I hate to do this, but I still do not have a good solution to actually
    fix this bug across architectures. So just disable it for now, so that
    the CI can still deliver actionable results. This commit adds a large
    red warning, so that at least the failure isn't lost forever, and
    hopefully this can be revisited down the line.
    
    Link: https://lore.kernel.org/netdev/CAHmME9pv1x6C4TNdL6648HydD8r+txpV4hTUXOBVkrapBXH4QQ@mail.gmail.com/
    Link: https://lore.kernel.org/netdev/YmszSXueTxYOC41G@zx2c4.com/
    Link: https://lore.kernel.org/wireguard/CAHmME9rNnBiNvBstb7MPwK-7AmAN0sOfnhdR=eeLrowWcKxaaQ@mail.gmail.com/
    
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    f8886735
  • Jason A. Donenfeld's avatar
    netns: limit parallelism to $(nproc) tests at once · 894152a5
    Jason A. Donenfeld 创作于
    
    The parallel tests were added to catch queueing issues from multiple
    cores. But what happens in reality when testing tons of processes is
    that these separate threads wind up fighting with the scheduler, and we
    wind up with contention in places we don't care about that decrease the
    chances of hitting a bug. So just do a test with the number of CPU
    cores, rather than trying to scale up arbitrarily.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    894152a5
  • Jason A. Donenfeld's avatar
    qemu: use vports on arm · 33c87a11
    Jason A. Donenfeld 创作于
    
    Rather than having to hack up QEMU, just use the virtio serial device.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    33c87a11
  • Jason A. Donenfeld's avatar
    qemu: set panic_on_warn=1 from cmdline · c7560fd0
    Jason A. Donenfeld 创作于
    
    Rather than setting this once init is running, set panic_on_warn from
    the kernel command line, so that it catches splats from WireGuard
    initialization code and the various crypto selftests.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    c7560fd0
  • Jason A. Donenfeld's avatar
    qemu: give up on RHEL8 in CI · ba45dd6f
    Jason A. Donenfeld 创作于
    
    They keep breaking their kernel and being difficult when I send patches
    to fix it, so just give up on trying to support this in the CI. It'll
    bitrot and people will complain and we'll see what happens at that
    point.
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    ba45dd6f
  • Jason A. Donenfeld's avatar
    3ec3e822
  • Jason A. Donenfeld's avatar
    version: bump · 18fbcd68
    Jason A. Donenfeld 创作于
    
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    18fbcd68
  • Jason A. Donenfeld's avatar
    compat: do not backport ktime_get_coarse_boottime_ns to c8s · 99935b07
    Jason A. Donenfeld 创作于
    
    Also bump the c8s version stamp.
    
    Reported-by: default avatarVladimír Beneš <vbenes@redhat.com>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    99935b07
  • Jason A. Donenfeld's avatar
    compat: drop CentOS 8 Stream support · 3d3c92b4
    Jason A. Donenfeld 创作于
    Nobody uses this and it's impossible to maintain given the current CI
    situation.
    
    RHEL 7 and 8 release remain for now, though that might not always be the
    case. See the link for details.
    
    Link: https://lists.zx2c4.com/pipermail/wireguard/2022-June/007664.html
    
    
    Suggested-by: default avatarPhilip J. Perry <phil@elrepo.org>
    Signed-off-by: default avatarJason A. Donenfeld <Jason@zx2c4.com>
    3d3c92b4
显示
759 个添加452 个删除
...@@ -30,8 +30,11 @@ static void copy_and_assign_cidr(struct allowedips_node *node, const u8 *src, ...@@ -30,8 +30,11 @@ static void copy_and_assign_cidr(struct allowedips_node *node, const u8 *src,
node->bitlen = bits; node->bitlen = bits;
memcpy(node->bits, src, bits / 8U); memcpy(node->bits, src, bits / 8U);
} }
#define CHOOSE_NODE(parent, key) \
parent->bit[(key[parent->bit_at_a] >> parent->bit_at_b) & 1] static inline u8 choose(struct allowedips_node *node, const u8 *key)
{
return (key[node->bit_at_a] >> node->bit_at_b) & 1;
}
static void push_rcu(struct allowedips_node **stack, static void push_rcu(struct allowedips_node **stack,
struct allowedips_node __rcu *p, unsigned int *len) struct allowedips_node __rcu *p, unsigned int *len)
...@@ -112,7 +115,7 @@ static struct allowedips_node *find_node(struct allowedips_node *trie, u8 bits, ...@@ -112,7 +115,7 @@ static struct allowedips_node *find_node(struct allowedips_node *trie, u8 bits,
found = node; found = node;
if (node->cidr == bits) if (node->cidr == bits)
break; break;
node = rcu_dereference_bh(CHOOSE_NODE(node, key)); node = rcu_dereference_bh(node->bit[choose(node, key)]);
} }
return found; return found;
} }
...@@ -144,8 +147,7 @@ static bool node_placement(struct allowedips_node __rcu *trie, const u8 *key, ...@@ -144,8 +147,7 @@ static bool node_placement(struct allowedips_node __rcu *trie, const u8 *key,
u8 cidr, u8 bits, struct allowedips_node **rnode, u8 cidr, u8 bits, struct allowedips_node **rnode,
struct mutex *lock) struct mutex *lock)
{ {
struct allowedips_node *node = rcu_dereference_protected(trie, struct allowedips_node *node = rcu_dereference_protected(trie, lockdep_is_held(lock));
lockdep_is_held(lock));
struct allowedips_node *parent = NULL; struct allowedips_node *parent = NULL;
bool exact = false; bool exact = false;
...@@ -155,13 +157,24 @@ static bool node_placement(struct allowedips_node __rcu *trie, const u8 *key, ...@@ -155,13 +157,24 @@ static bool node_placement(struct allowedips_node __rcu *trie, const u8 *key,
exact = true; exact = true;
break; break;
} }
node = rcu_dereference_protected(CHOOSE_NODE(parent, key), node = rcu_dereference_protected(parent->bit[choose(parent, key)], lockdep_is_held(lock));
lockdep_is_held(lock));
} }
*rnode = parent; *rnode = parent;
return exact; return exact;
} }
static inline void connect_node(struct allowedips_node __rcu **parent, u8 bit, struct allowedips_node *node)
{
node->parent_bit_packed = (unsigned long)parent | bit;
rcu_assign_pointer(*parent, node);
}
static inline void choose_and_connect_node(struct allowedips_node *parent, struct allowedips_node *node)
{
u8 bit = choose(parent, node->bits);
connect_node(&parent->bit[bit], bit, node);
}
static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key, static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
u8 cidr, struct wg_peer *peer, struct mutex *lock) u8 cidr, struct wg_peer *peer, struct mutex *lock)
{ {
...@@ -177,8 +190,7 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key, ...@@ -177,8 +190,7 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
RCU_INIT_POINTER(node->peer, peer); RCU_INIT_POINTER(node->peer, peer);
list_add_tail(&node->peer_list, &peer->allowedips_list); list_add_tail(&node->peer_list, &peer->allowedips_list);
copy_and_assign_cidr(node, key, cidr, bits); copy_and_assign_cidr(node, key, cidr, bits);
rcu_assign_pointer(node->parent_bit, trie); connect_node(trie, 2, node);
rcu_assign_pointer(*trie, node);
return 0; return 0;
} }
if (node_placement(*trie, key, cidr, bits, &node, lock)) { if (node_placement(*trie, key, cidr, bits, &node, lock)) {
...@@ -197,10 +209,10 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key, ...@@ -197,10 +209,10 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
if (!node) { if (!node) {
down = rcu_dereference_protected(*trie, lockdep_is_held(lock)); down = rcu_dereference_protected(*trie, lockdep_is_held(lock));
} else { } else {
down = rcu_dereference_protected(CHOOSE_NODE(node, key), lockdep_is_held(lock)); const u8 bit = choose(node, key);
down = rcu_dereference_protected(node->bit[bit], lockdep_is_held(lock));
if (!down) { if (!down) {
rcu_assign_pointer(newnode->parent_bit, &CHOOSE_NODE(node, key)); connect_node(&node->bit[bit], bit, newnode);
rcu_assign_pointer(CHOOSE_NODE(node, key), newnode);
return 0; return 0;
} }
} }
...@@ -208,15 +220,11 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key, ...@@ -208,15 +220,11 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
parent = node; parent = node;
if (newnode->cidr == cidr) { if (newnode->cidr == cidr) {
rcu_assign_pointer(down->parent_bit, &CHOOSE_NODE(newnode, down->bits)); choose_and_connect_node(newnode, down);
rcu_assign_pointer(CHOOSE_NODE(newnode, down->bits), down); if (!parent)
if (!parent) { connect_node(trie, 2, newnode);
rcu_assign_pointer(newnode->parent_bit, trie); else
rcu_assign_pointer(*trie, newnode); choose_and_connect_node(parent, newnode);
} else {
rcu_assign_pointer(newnode->parent_bit, &CHOOSE_NODE(parent, newnode->bits));
rcu_assign_pointer(CHOOSE_NODE(parent, newnode->bits), newnode);
}
return 0; return 0;
} }
...@@ -229,17 +237,12 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key, ...@@ -229,17 +237,12 @@ static int add(struct allowedips_node __rcu **trie, u8 bits, const u8 *key,
INIT_LIST_HEAD(&node->peer_list); INIT_LIST_HEAD(&node->peer_list);
copy_and_assign_cidr(node, newnode->bits, cidr, bits); copy_and_assign_cidr(node, newnode->bits, cidr, bits);
rcu_assign_pointer(down->parent_bit, &CHOOSE_NODE(node, down->bits)); choose_and_connect_node(node, down);
rcu_assign_pointer(CHOOSE_NODE(node, down->bits), down); choose_and_connect_node(node, newnode);
rcu_assign_pointer(newnode->parent_bit, &CHOOSE_NODE(node, newnode->bits)); if (!parent)
rcu_assign_pointer(CHOOSE_NODE(node, newnode->bits), newnode); connect_node(trie, 2, node);
if (!parent) { else
rcu_assign_pointer(node->parent_bit, trie); choose_and_connect_node(parent, node);
rcu_assign_pointer(*trie, node);
} else {
rcu_assign_pointer(node->parent_bit, &CHOOSE_NODE(parent, node->bits));
rcu_assign_pointer(CHOOSE_NODE(parent, node->bits), node);
}
return 0; return 0;
} }
...@@ -297,7 +300,8 @@ int wg_allowedips_insert_v6(struct allowedips *table, const struct in6_addr *ip, ...@@ -297,7 +300,8 @@ int wg_allowedips_insert_v6(struct allowedips *table, const struct in6_addr *ip,
void wg_allowedips_remove_by_peer(struct allowedips *table, void wg_allowedips_remove_by_peer(struct allowedips *table,
struct wg_peer *peer, struct mutex *lock) struct wg_peer *peer, struct mutex *lock)
{ {
struct allowedips_node *node, *child, *tmp; struct allowedips_node *node, *child, **parent_bit, *parent, *tmp;
bool free_parent;
if (list_empty(&peer->allowedips_list)) if (list_empty(&peer->allowedips_list))
return; return;
...@@ -307,19 +311,29 @@ void wg_allowedips_remove_by_peer(struct allowedips *table, ...@@ -307,19 +311,29 @@ void wg_allowedips_remove_by_peer(struct allowedips *table,
RCU_INIT_POINTER(node->peer, NULL); RCU_INIT_POINTER(node->peer, NULL);
if (node->bit[0] && node->bit[1]) if (node->bit[0] && node->bit[1])
continue; continue;
child = rcu_dereference_protected( child = rcu_dereference_protected(node->bit[!rcu_access_pointer(node->bit[0])],
node->bit[!rcu_access_pointer(node->bit[0])], lockdep_is_held(lock));
lockdep_is_held(lock));
if (child) if (child)
child->parent_bit = node->parent_bit; child->parent_bit_packed = node->parent_bit_packed;
*rcu_dereference_protected(node->parent_bit, lockdep_is_held(lock)) = child; parent_bit = (struct allowedips_node **)(node->parent_bit_packed & ~3UL);
*parent_bit = child;
parent = (void *)parent_bit -
offsetof(struct allowedips_node, bit[node->parent_bit_packed & 1]);
free_parent = !rcu_access_pointer(node->bit[0]) &&
!rcu_access_pointer(node->bit[1]) &&
(node->parent_bit_packed & 3) <= 1 &&
!rcu_access_pointer(parent->peer);
if (free_parent)
child = rcu_dereference_protected(
parent->bit[!(node->parent_bit_packed & 1)],
lockdep_is_held(lock));
call_rcu(&node->rcu, node_free_rcu); call_rcu(&node->rcu, node_free_rcu);
if (!free_parent)
/* TODO: Note that we currently don't walk up and down in order to continue;
* free any potential filler nodes. This means that this function if (child)
* doesn't free up as much as it could, which could be revisited child->parent_bit_packed = parent->parent_bit_packed;
* at some point. *(struct allowedips_node **)(parent->parent_bit_packed & ~3UL) = child;
*/ call_rcu(&parent->rcu, node_free_rcu);
} }
} }
......
...@@ -19,7 +19,7 @@ struct allowedips_node { ...@@ -19,7 +19,7 @@ struct allowedips_node {
u8 bits[16] __aligned(__alignof(u64)); u8 bits[16] __aligned(__alignof(u64));
/* Keep rarely used members at bottom to be beyond cache line. */ /* Keep rarely used members at bottom to be beyond cache line. */
struct allowedips_node *__rcu *parent_bit; unsigned long parent_bit_packed;
union { union {
struct list_head peer_list; struct list_head peer_list;
struct rcu_head rcu; struct rcu_head rcu;
...@@ -30,7 +30,7 @@ struct allowedips { ...@@ -30,7 +30,7 @@ struct allowedips {
struct allowedips_node __rcu *root4; struct allowedips_node __rcu *root4;
struct allowedips_node __rcu *root6; struct allowedips_node __rcu *root6;
u64 seq; u64 seq;
}; } __aligned(4); /* We pack the lower 2 bits of &root, but m68k only gives 16-bit alignment. */
void wg_allowedips_init(struct allowedips *table); void wg_allowedips_init(struct allowedips *table);
void wg_allowedips_free(struct allowedips *table, struct mutex *mutex); void wg_allowedips_free(struct allowedips *table, struct mutex *mutex);
......
...@@ -12,6 +12,10 @@ ifeq ($(wildcard $(srctree)/include/linux/ptr_ring.h),) ...@@ -12,6 +12,10 @@ ifeq ($(wildcard $(srctree)/include/linux/ptr_ring.h),)
ccflags-y += -I$(kbuild-dir)/compat/ptr_ring/include ccflags-y += -I$(kbuild-dir)/compat/ptr_ring/include
endif endif
ifeq ($(wildcard $(srctree)/include/linux/skb_array.h),)
ccflags-y += -I$(kbuild-dir)/compat/skb_array/include
endif
ifeq ($(wildcard $(srctree)/include/linux/siphash.h),) ifeq ($(wildcard $(srctree)/include/linux/siphash.h),)
ccflags-y += -I$(kbuild-dir)/compat/siphash/include ccflags-y += -I$(kbuild-dir)/compat/siphash/include
wireguard-y += compat/siphash/siphash.o wireguard-y += compat/siphash/siphash.o
...@@ -65,6 +69,10 @@ ifeq ($(wildcard $(srctree)/arch/arm64/include/asm/neon.h)$(CONFIG_ARM64),y) ...@@ -65,6 +69,10 @@ ifeq ($(wildcard $(srctree)/arch/arm64/include/asm/neon.h)$(CONFIG_ARM64),y)
ccflags-y += -I$(kbuild-dir)/compat/neon-arm/include ccflags-y += -I$(kbuild-dir)/compat/neon-arm/include
endif endif
ifeq ($(wildcard $(srctree)/include/net/dst_metadata.h),)
ccflags-y += -I$(kbuild-dir)/compat/dstmetadata/include
endif
ifeq ($(CONFIG_X86_64),y) ifeq ($(CONFIG_X86_64),y)
ifeq ($(ssse3_instr),) ifeq ($(ssse3_instr),)
ssse3_instr := $(call as-instr,pshufb %xmm0$(comma)%xmm0,-DCONFIG_AS_SSSE3=1) ssse3_instr := $(call as-instr,pshufb %xmm0$(comma)%xmm0,-DCONFIG_AS_SSSE3=1)
......
...@@ -15,14 +15,11 @@ ...@@ -15,14 +15,11 @@
#define ISRHEL7 #define ISRHEL7
#elif RHEL_MAJOR == 8 #elif RHEL_MAJOR == 8
#define ISRHEL8 #define ISRHEL8
#if RHEL_MINOR >= 4
#define ISCENTOS8S
#endif
#endif #endif
#endif #endif
/* PaX compatibility */ /* PaX compatibility */
#if defined(RAP_PLUGIN) #if defined(RAP_PLUGIN) && defined(RAP_ENTRY)
#undef ENTRY #undef ENTRY
#define ENTRY RAP_ENTRY #define ENTRY RAP_ENTRY
#endif #endif
...@@ -51,7 +48,7 @@ ...@@ -51,7 +48,7 @@
#undef pull #undef pull
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 4, 76) && !defined(ISCENTOS8S) #if LINUX_VERSION_CODE < KERNEL_VERSION(5, 4, 76) && !defined(ISRHEL8) && !defined(SYM_FUNC_START)
#define SYM_FUNC_START ENTRY #define SYM_FUNC_START ENTRY
#define SYM_FUNC_END ENDPROC #define SYM_FUNC_END ENDPROC
#endif #endif
......
...@@ -16,15 +16,10 @@ ...@@ -16,15 +16,10 @@
#define ISRHEL7 #define ISRHEL7
#elif RHEL_MAJOR == 8 #elif RHEL_MAJOR == 8
#define ISRHEL8 #define ISRHEL8
#if RHEL_MINOR >= 4
#define ISCENTOS8S
#endif
#endif #endif
#endif #endif
#ifdef UTS_UBUNTU_RELEASE_ABI #ifdef UTS_UBUNTU_RELEASE_ABI
#if LINUX_VERSION_CODE == KERNEL_VERSION(3, 13, 11) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)
#define ISUBUNTU1404
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 5, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 4, 0)
#define ISUBUNTU1604 #define ISUBUNTU1604
#elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 15, 0) #elif LINUX_VERSION_CODE < KERNEL_VERSION(4, 16, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 15, 0)
#define ISUBUNTU1804 #define ISUBUNTU1804
...@@ -219,7 +214,7 @@ static inline void skb_scrub_packet(struct sk_buff *skb, bool xnet) ...@@ -219,7 +214,7 @@ static inline void skb_scrub_packet(struct sk_buff *skb, bool xnet)
#define skb_scrub_packet(a, b) skb_scrub_packet(a) #define skb_scrub_packet(a, b) skb_scrub_packet(a)
#endif #endif
#if ((LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0)) || LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 63) || defined(ISUBUNTU1404)) && !defined(ISRHEL7) #if ((LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0)) || LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 63)) && !defined(ISRHEL7)
#include <linux/random.h> #include <linux/random.h>
static inline u32 __compat_prandom_u32_max(u32 ep_ro) static inline u32 __compat_prandom_u32_max(u32 ep_ro)
{ {
...@@ -268,7 +263,7 @@ static inline u32 __compat_prandom_u32_max(u32 ep_ro) ...@@ -268,7 +263,7 @@ static inline u32 __compat_prandom_u32_max(u32 ep_ro)
#endif #endif
#endif #endif
#if (LINUX_VERSION_CODE < KERNEL_VERSION(3, 17, 3) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 17, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 16, 35) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 15, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 24) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0) && !defined(ISUBUNTU1404)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 33) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 11, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 10, 60) && !defined(ISRHEL7)) #if (LINUX_VERSION_CODE < KERNEL_VERSION(3, 17, 3) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 17, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 16, 35) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 15, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 14, 24) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 33) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 11, 0)) || (LINUX_VERSION_CODE < KERNEL_VERSION(3, 10, 60) && !defined(ISRHEL7))
static inline void memzero_explicit(void *s, size_t count) static inline void memzero_explicit(void *s, size_t count)
{ {
memset(s, 0, count); memset(s, 0, count);
...@@ -281,7 +276,7 @@ static const struct in6_addr __compat_in6addr_any = IN6ADDR_ANY_INIT; ...@@ -281,7 +276,7 @@ static const struct in6_addr __compat_in6addr_any = IN6ADDR_ANY_INIT;
#define in6addr_any __compat_in6addr_any #define in6addr_any __compat_in6addr_any
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 13, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 2, 0) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 13, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 2, 0) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 9, 320))
#include <linux/completion.h> #include <linux/completion.h>
#include <linux/random.h> #include <linux/random.h>
#include <linux/errno.h> #include <linux/errno.h>
...@@ -325,7 +320,7 @@ static inline int wait_for_random_bytes(void) ...@@ -325,7 +320,7 @@ static inline int wait_for_random_bytes(void)
} }
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 2, 0) && !defined(ISRHEL8) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 19, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 2, 0) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 15, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 14, 285)) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 9, 320)) && !defined(ISRHEL8)
#include <linux/random.h> #include <linux/random.h>
#include <linux/slab.h> #include <linux/slab.h>
struct rng_is_initialized_callback { struct rng_is_initialized_callback {
...@@ -377,7 +372,7 @@ static inline bool rng_is_initialized(void) ...@@ -377,7 +372,7 @@ static inline bool rng_is_initialized(void)
} }
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 13, 0) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 13, 0) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 9, 320))
static inline int get_random_bytes_wait(void *buf, int nbytes) static inline int get_random_bytes_wait(void *buf, int nbytes)
{ {
int ret = wait_for_random_bytes(); int ret = wait_for_random_bytes();
...@@ -502,7 +497,7 @@ static inline void *__compat_kvzalloc(size_t size, gfp_t flags) ...@@ -502,7 +497,7 @@ static inline void *__compat_kvzalloc(size_t size, gfp_t flags)
#define kvzalloc __compat_kvzalloc #define kvzalloc __compat_kvzalloc
#endif #endif
#if ((LINUX_VERSION_CODE < KERNEL_VERSION(3, 15, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0)) || LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 41)) && !defined(ISUBUNTU1404) #if ((LINUX_VERSION_CODE < KERNEL_VERSION(3, 15, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(3, 13, 0)) || LINUX_VERSION_CODE < KERNEL_VERSION(3, 12, 41))
#include <linux/vmalloc.h> #include <linux/vmalloc.h>
#include <linux/mm.h> #include <linux/mm.h>
static inline void __compat_kvfree(const void *addr) static inline void __compat_kvfree(const void *addr)
...@@ -515,6 +510,28 @@ static inline void __compat_kvfree(const void *addr) ...@@ -515,6 +510,28 @@ static inline void __compat_kvfree(const void *addr)
#define kvfree __compat_kvfree #define kvfree __compat_kvfree
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 12, 0)
#include <linux/vmalloc.h>
#include <linux/mm.h>
static inline void *__compat_kvmalloc_array(size_t n, size_t size, gfp_t flags)
{
if (n != 0 && SIZE_MAX / n < size)
return NULL;
return kvmalloc(n * size, flags);
}
#define kvmalloc_array __compat_kvmalloc_array
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 18, 0)
#include <linux/vmalloc.h>
#include <linux/mm.h>
static inline void *__compat_kvcalloc(size_t n, size_t size, gfp_t flags)
{
return kvmalloc_array(n, size, flags | __GFP_ZERO);
}
#define kvcalloc __compat_kvcalloc
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 11, 9) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 11, 9)
#include <linux/netdevice.h> #include <linux/netdevice.h>
#define priv_destructor destructor #define priv_destructor destructor
...@@ -704,7 +721,7 @@ static inline void *skb_put_data(struct sk_buff *skb, const void *data, unsigned ...@@ -704,7 +721,7 @@ static inline void *skb_put_data(struct sk_buff *skb, const void *data, unsigned
#endif #endif
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 17, 0) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 17, 0) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 15, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 14, 285)) && (LINUX_VERSION_CODE >= KERNEL_VERSION(4, 10, 0) || LINUX_VERSION_CODE < KERNEL_VERSION(4, 9, 320))
static inline void le32_to_cpu_array(u32 *buf, unsigned int words) static inline void le32_to_cpu_array(u32 *buf, unsigned int words)
{ {
while (words--) { while (words--) {
...@@ -757,7 +774,7 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2, ...@@ -757,7 +774,7 @@ static inline void crypto_xor_cpy(u8 *dst, const u8 *src1, const u8 *src2,
#define hlist_add_behind(a, b) hlist_add_after(b, a) #define hlist_add_behind(a, b) hlist_add_after(b, a)
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 0, 0) && !defined(ISCENTOS8S) #if LINUX_VERSION_CODE < KERNEL_VERSION(5, 0, 0) && !defined(ISRHEL8)
#define totalram_pages() totalram_pages #define totalram_pages() totalram_pages
#endif #endif
...@@ -831,10 +848,16 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb) ...@@ -831,10 +848,16 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb)
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(4, 20, 0) && !defined(ISRHEL8) #if LINUX_VERSION_CODE < KERNEL_VERSION(4, 20, 0) && !defined(ISRHEL8)
#include <net/netlink.h>
#ifndef NLA_POLICY_EXACT_LEN
#define NLA_POLICY_EXACT_LEN(_len) { .type = NLA_UNSPEC, .len = _len } #define NLA_POLICY_EXACT_LEN(_len) { .type = NLA_UNSPEC, .len = _len }
#endif #endif
#endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 2, 0) && !defined(ISRHEL8) #if LINUX_VERSION_CODE < KERNEL_VERSION(5, 2, 0) && !defined(ISRHEL8)
#include <net/netlink.h>
#ifndef NLA_POLICY_MIN_LEN
#define NLA_POLICY_MIN_LEN(_len) { .type = NLA_UNSPEC, .len = _len } #define NLA_POLICY_MIN_LEN(_len) { .type = NLA_UNSPEC, .len = _len }
#endif
#define COMPAT_CANNOT_INDIVIDUAL_NETLINK_OPS_POLICY #define COMPAT_CANNOT_INDIVIDUAL_NETLINK_OPS_POLICY
#endif #endif
...@@ -849,7 +872,7 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb) ...@@ -849,7 +872,7 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb)
#endif #endif
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 5, 0) #if LINUX_VERSION_CODE < KERNEL_VERSION(5, 5, 0) && !defined(ISRHEL8)
#define genl_dumpit_info(cb) ({ \ #define genl_dumpit_info(cb) ({ \
struct { struct nlattr **attrs; } *a = (void *)((u8 *)cb->args + offsetofend(struct dump_ctx, next_allowedip)); \ struct { struct nlattr **attrs; } *a = (void *)((u8 *)cb->args + offsetofend(struct dump_ctx, next_allowedip)); \
BUILD_BUG_ON(sizeof(cb->args) < offsetofend(struct dump_ctx, next_allowedip) + sizeof(*a)); \ BUILD_BUG_ON(sizeof(cb->args) < offsetofend(struct dump_ctx, next_allowedip) + sizeof(*a)); \
...@@ -869,11 +892,13 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb) ...@@ -869,11 +892,13 @@ static inline void skb_mark_not_on_list(struct sk_buff *skb)
#endif #endif
#endif #endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 5, 0) #if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 4, 200) || (LINUX_VERSION_CODE < KERNEL_VERSION(4, 20, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 19, 249)) || (LINUX_VERSION_CODE < KERNEL_VERSION(4, 15, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 14, 285)) || (LINUX_VERSION_CODE < KERNEL_VERSION(4, 10, 0) && LINUX_VERSION_CODE >= KERNEL_VERSION(4, 9, 320))
#define blake2s_init zinc_blake2s_init #define blake2s_init zinc_blake2s_init
#define blake2s_init_key zinc_blake2s_init_key #define blake2s_init_key zinc_blake2s_init_key
#define blake2s_update zinc_blake2s_update #define blake2s_update zinc_blake2s_update
#define blake2s_final zinc_blake2s_final #define blake2s_final zinc_blake2s_final
#endif
#if LINUX_VERSION_CODE >= KERNEL_VERSION(5, 5, 0)
#define blake2s_hmac zinc_blake2s_hmac #define blake2s_hmac zinc_blake2s_hmac
#define chacha20 zinc_chacha20 #define chacha20 zinc_chacha20
#define hchacha20 zinc_hchacha20 #define hchacha20 zinc_hchacha20
...@@ -1096,6 +1121,37 @@ static const struct header_ops ip_tunnel_header_ops = { .parse_protocol = ip_tun ...@@ -1096,6 +1121,37 @@ static const struct header_ops ip_tunnel_header_ops = { .parse_protocol = ip_tun
#endif #endif
#endif #endif
#if LINUX_VERSION_CODE < KERNEL_VERSION(5, 16, 0)
#include <net/dst_cache.h>
struct dst_cache_pcpu {
unsigned long refresh_ts;
struct dst_entry *dst;
u32 cookie;
union {
struct in_addr in_saddr;
struct in6_addr in6_saddr;
};
};
#define COMPAT_HAS_DEFINED_DST_CACHE_PCPU
static inline void dst_cache_reset_now(struct dst_cache *dst_cache)
{
int i;
if (!dst_cache->cache)
return;
dst_cache->reset_ts = jiffies;
for_each_possible_cpu(i) {
struct dst_cache_pcpu *idst = per_cpu_ptr(dst_cache->cache, i);
struct dst_entry *dst = idst->dst;
idst->cookie = 0;
idst->dst = NULL;
dst_release(dst);
}
}
#endif
#if defined(ISUBUNTU1604) || defined(ISRHEL7) #if defined(ISUBUNTU1604) || defined(ISRHEL7)
#include <linux/siphash.h> #include <linux/siphash.h>
#ifndef _WG_LINUX_SIPHASH_H #ifndef _WG_LINUX_SIPHASH_H
...@@ -1127,7 +1183,7 @@ static const struct header_ops ip_tunnel_header_ops = { .parse_protocol = ip_tun ...@@ -1127,7 +1183,7 @@ static const struct header_ops ip_tunnel_header_ops = { .parse_protocol = ip_tun
#undef __read_mostly #undef __read_mostly
#define __read_mostly #define __read_mostly
#endif #endif
#if (defined(RAP_PLUGIN) || defined(CONFIG_CFI_CLANG)) && LINUX_VERSION_CODE < KERNEL_VERSION(4, 15, 0) #if (defined(CONFIG_PAX) || defined(CONFIG_CFI_CLANG)) && LINUX_VERSION_CODE < KERNEL_VERSION(4, 15, 0)
#include <linux/timer.h> #include <linux/timer.h>
#define wg_expired_retransmit_handshake(a) wg_expired_retransmit_handshake(unsigned long timer) #define wg_expired_retransmit_handshake(a) wg_expired_retransmit_handshake(unsigned long timer)
#define wg_expired_send_keepalive(a) wg_expired_send_keepalive(unsigned long timer) #define wg_expired_send_keepalive(a) wg_expired_send_keepalive(unsigned long timer)
......
...@@ -27,6 +27,7 @@ static inline u32 rt6_get_cookie(const struct rt6_info *rt) ...@@ -27,6 +27,7 @@ static inline u32 rt6_get_cookie(const struct rt6_info *rt)
#endif #endif
#include <uapi/linux/in.h> #include <uapi/linux/in.h>
#ifndef COMPAT_HAS_DEFINED_DST_CACHE_PCPU
struct dst_cache_pcpu { struct dst_cache_pcpu {
unsigned long refresh_ts; unsigned long refresh_ts;
struct dst_entry *dst; struct dst_entry *dst;
...@@ -36,6 +37,7 @@ struct dst_cache_pcpu { ...@@ -36,6 +37,7 @@ struct dst_cache_pcpu {
struct in6_addr in6_saddr; struct in6_addr in6_saddr;
}; };
}; };
#endif
static void dst_cache_per_cpu_dst_set(struct dst_cache_pcpu *dst_cache, static void dst_cache_per_cpu_dst_set(struct dst_cache_pcpu *dst_cache,
struct dst_entry *dst, u32 cookie) struct dst_entry *dst, u32 cookie)
......
#ifndef skb_valid_dst
#define skb_valid_dst(skb) (!!skb_dst(skb))
#endif
...@@ -22,9 +22,7 @@ typedef struct { ...@@ -22,9 +22,7 @@ typedef struct {
} siphash_key_t; } siphash_key_t;
u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key); u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key);
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key); u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key);
#endif
u64 siphash_1u64(const u64 a, const siphash_key_t *key); u64 siphash_1u64(const u64 a, const siphash_key_t *key);
u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key); u64 siphash_2u64(const u64 a, const u64 b, const siphash_key_t *key);
...@@ -77,10 +75,9 @@ static inline u64 ___siphash_aligned(const __le64 *data, size_t len, ...@@ -77,10 +75,9 @@ static inline u64 ___siphash_aligned(const __le64 *data, size_t len,
static inline u64 siphash(const void *data, size_t len, static inline u64 siphash(const void *data, size_t len,
const siphash_key_t *key) const siphash_key_t *key)
{ {
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
if (!IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT)) !IS_ALIGNED((unsigned long)data, SIPHASH_ALIGNMENT))
return __siphash_unaligned(data, len, key); return __siphash_unaligned(data, len, key);
#endif
return ___siphash_aligned(data, len, key); return ___siphash_aligned(data, len, key);
} }
...@@ -91,10 +88,8 @@ typedef struct { ...@@ -91,10 +88,8 @@ typedef struct {
u32 __hsiphash_aligned(const void *data, size_t len, u32 __hsiphash_aligned(const void *data, size_t len,
const hsiphash_key_t *key); const hsiphash_key_t *key);
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u32 __hsiphash_unaligned(const void *data, size_t len, u32 __hsiphash_unaligned(const void *data, size_t len,
const hsiphash_key_t *key); const hsiphash_key_t *key);
#endif
u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key); u32 hsiphash_1u32(const u32 a, const hsiphash_key_t *key);
u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key); u32 hsiphash_2u32(const u32 a, const u32 b, const hsiphash_key_t *key);
...@@ -130,10 +125,9 @@ static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len, ...@@ -130,10 +125,9 @@ static inline u32 ___hsiphash_aligned(const __le32 *data, size_t len,
static inline u32 hsiphash(const void *data, size_t len, static inline u32 hsiphash(const void *data, size_t len,
const hsiphash_key_t *key) const hsiphash_key_t *key)
{ {
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS if (IS_ENABLED(CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS) ||
if (!IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT)) !IS_ALIGNED((unsigned long)data, HSIPHASH_ALIGNMENT))
return __hsiphash_unaligned(data, len, key); return __hsiphash_unaligned(data, len, key);
#endif
return ___hsiphash_aligned(data, len, key); return ___hsiphash_aligned(data, len, key);
} }
......
...@@ -57,6 +57,7 @@ ...@@ -57,6 +57,7 @@
SIPROUND; \ SIPROUND; \
return (v0 ^ v1) ^ (v2 ^ v3); return (v0 ^ v1) ^ (v2 ^ v3);
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key) u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
{ {
const u8 *end = data + len - (len % sizeof(u64)); const u8 *end = data + len - (len % sizeof(u64));
...@@ -76,19 +77,19 @@ u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key) ...@@ -76,19 +77,19 @@ u64 __siphash_aligned(const void *data, size_t len, const siphash_key_t *key)
bytemask_from_count(left))); bytemask_from_count(left)));
#else #else
switch (left) { switch (left) {
case 7: b |= ((u64)end[6]) << 48; case 7: b |= ((u64)end[6]) << 48; fallthrough;
case 6: b |= ((u64)end[5]) << 40; case 6: b |= ((u64)end[5]) << 40; fallthrough;
case 5: b |= ((u64)end[4]) << 32; case 5: b |= ((u64)end[4]) << 32; fallthrough;
case 4: b |= le32_to_cpup(data); break; case 4: b |= le32_to_cpup(data); break;
case 3: b |= ((u64)end[2]) << 16; case 3: b |= ((u64)end[2]) << 16; fallthrough;
case 2: b |= le16_to_cpup(data); break; case 2: b |= le16_to_cpup(data); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
#endif #endif
POSTAMBLE POSTAMBLE
} }
#endif
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
{ {
const u8 *end = data + len - (len % sizeof(u64)); const u8 *end = data + len - (len % sizeof(u64));
...@@ -108,18 +109,17 @@ u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key) ...@@ -108,18 +109,17 @@ u64 __siphash_unaligned(const void *data, size_t len, const siphash_key_t *key)
bytemask_from_count(left))); bytemask_from_count(left)));
#else #else
switch (left) { switch (left) {
case 7: b |= ((u64)end[6]) << 48; case 7: b |= ((u64)end[6]) << 48; fallthrough;
case 6: b |= ((u64)end[5]) << 40; case 6: b |= ((u64)end[5]) << 40; fallthrough;
case 5: b |= ((u64)end[4]) << 32; case 5: b |= ((u64)end[4]) << 32; fallthrough;
case 4: b |= get_unaligned_le32(end); break; case 4: b |= get_unaligned_le32(end); break;
case 3: b |= ((u64)end[2]) << 16; case 3: b |= ((u64)end[2]) << 16; fallthrough;
case 2: b |= get_unaligned_le16(end); break; case 2: b |= get_unaligned_le16(end); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
#endif #endif
POSTAMBLE POSTAMBLE
} }
#endif
/** /**
* siphash_1u64 - compute 64-bit siphash PRF value of a u64 * siphash_1u64 - compute 64-bit siphash PRF value of a u64
...@@ -250,6 +250,7 @@ u64 siphash_3u32(const u32 first, const u32 second, const u32 third, ...@@ -250,6 +250,7 @@ u64 siphash_3u32(const u32 first, const u32 second, const u32 third,
HSIPROUND; \ HSIPROUND; \
return (v0 ^ v1) ^ (v2 ^ v3); return (v0 ^ v1) ^ (v2 ^ v3);
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
{ {
const u8 *end = data + len - (len % sizeof(u64)); const u8 *end = data + len - (len % sizeof(u64));
...@@ -268,19 +269,19 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) ...@@ -268,19 +269,19 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
bytemask_from_count(left))); bytemask_from_count(left)));
#else #else
switch (left) { switch (left) {
case 7: b |= ((u64)end[6]) << 48; case 7: b |= ((u64)end[6]) << 48; fallthrough;
case 6: b |= ((u64)end[5]) << 40; case 6: b |= ((u64)end[5]) << 40; fallthrough;
case 5: b |= ((u64)end[4]) << 32; case 5: b |= ((u64)end[4]) << 32; fallthrough;
case 4: b |= le32_to_cpup(data); break; case 4: b |= le32_to_cpup(data); break;
case 3: b |= ((u64)end[2]) << 16; case 3: b |= ((u64)end[2]) << 16; fallthrough;
case 2: b |= le16_to_cpup(data); break; case 2: b |= le16_to_cpup(data); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
#endif #endif
HPOSTAMBLE HPOSTAMBLE
} }
#endif
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u32 __hsiphash_unaligned(const void *data, size_t len, u32 __hsiphash_unaligned(const void *data, size_t len,
const hsiphash_key_t *key) const hsiphash_key_t *key)
{ {
...@@ -300,18 +301,17 @@ u32 __hsiphash_unaligned(const void *data, size_t len, ...@@ -300,18 +301,17 @@ u32 __hsiphash_unaligned(const void *data, size_t len,
bytemask_from_count(left))); bytemask_from_count(left)));
#else #else
switch (left) { switch (left) {
case 7: b |= ((u64)end[6]) << 48; case 7: b |= ((u64)end[6]) << 48; fallthrough;
case 6: b |= ((u64)end[5]) << 40; case 6: b |= ((u64)end[5]) << 40; fallthrough;
case 5: b |= ((u64)end[4]) << 32; case 5: b |= ((u64)end[4]) << 32; fallthrough;
case 4: b |= get_unaligned_le32(end); break; case 4: b |= get_unaligned_le32(end); break;
case 3: b |= ((u64)end[2]) << 16; case 3: b |= ((u64)end[2]) << 16; fallthrough;
case 2: b |= get_unaligned_le16(end); break; case 2: b |= get_unaligned_le16(end); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
#endif #endif
HPOSTAMBLE HPOSTAMBLE
} }
#endif
/** /**
* hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32 * hsiphash_1u32 - compute 64-bit hsiphash PRF value of a u32
...@@ -412,6 +412,7 @@ u32 hsiphash_4u32(const u32 first, const u32 second, const u32 third, ...@@ -412,6 +412,7 @@ u32 hsiphash_4u32(const u32 first, const u32 second, const u32 third,
HSIPROUND; \ HSIPROUND; \
return v1 ^ v3; return v1 ^ v3;
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
{ {
const u8 *end = data + len - (len % sizeof(u32)); const u8 *end = data + len - (len % sizeof(u32));
...@@ -425,14 +426,14 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key) ...@@ -425,14 +426,14 @@ u32 __hsiphash_aligned(const void *data, size_t len, const hsiphash_key_t *key)
v0 ^= m; v0 ^= m;
} }
switch (left) { switch (left) {
case 3: b |= ((u32)end[2]) << 16; case 3: b |= ((u32)end[2]) << 16; fallthrough;
case 2: b |= le16_to_cpup(data); break; case 2: b |= le16_to_cpup(data); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
HPOSTAMBLE HPOSTAMBLE
} }
#endif
#ifndef CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS
u32 __hsiphash_unaligned(const void *data, size_t len, u32 __hsiphash_unaligned(const void *data, size_t len,
const hsiphash_key_t *key) const hsiphash_key_t *key)
{ {
...@@ -447,13 +448,12 @@ u32 __hsiphash_unaligned(const void *data, size_t len, ...@@ -447,13 +448,12 @@ u32 __hsiphash_unaligned(const void *data, size_t len,
v0 ^= m; v0 ^= m;
} }
switch (left) { switch (left) {
case 3: b |= ((u32)end[2]) << 16; case 3: b |= ((u32)end[2]) << 16; fallthrough;
case 2: b |= get_unaligned_le16(end); break; case 2: b |= get_unaligned_le16(end); break;
case 1: b |= end[0]; case 1: b |= end[0];
} }
HPOSTAMBLE HPOSTAMBLE
} }
#endif
/** /**
* hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32 * hsiphash_1u32 - compute 32-bit hsiphash PRF value of a u32
......
#ifndef _WG_SKB_ARRAY_H
#define _WG_SKB_ARRAY_H
#include <linux/skbuff.h>
static void __skb_array_destroy_skb(void *ptr)
{
kfree_skb(ptr);
}
#endif
...@@ -38,9 +38,10 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg, ...@@ -38,9 +38,10 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg,
struct socket *sock = NULL; struct socket *sock = NULL;
struct sockaddr_in udp_addr; struct sockaddr_in udp_addr;
err = __sock_create(net, AF_INET, SOCK_DGRAM, 0, &sock, 1); err = sock_create_kern(AF_INET, SOCK_DGRAM, 0, &sock);
if (err < 0) if (err < 0)
goto error; goto error;
sk_change_net(sock->sk, net);
udp_addr.sin_family = AF_INET; udp_addr.sin_family = AF_INET;
udp_addr.sin_addr = cfg->local_ip; udp_addr.sin_addr = cfg->local_ip;
...@@ -72,7 +73,7 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg, ...@@ -72,7 +73,7 @@ int udp_sock_create4(struct net *net, struct udp_port_cfg *cfg,
error: error:
if (sock) { if (sock) {
kernel_sock_shutdown(sock, SHUT_RDWR); kernel_sock_shutdown(sock, SHUT_RDWR);
sock_release(sock); sk_release_kernel(sock->sk);
} }
*sockp = NULL; *sockp = NULL;
return err; return err;
...@@ -229,7 +230,7 @@ void udp_tunnel_sock_release(struct socket *sock) ...@@ -229,7 +230,7 @@ void udp_tunnel_sock_release(struct socket *sock)
{ {
rcu_assign_sk_user_data(sock->sk, NULL); rcu_assign_sk_user_data(sock->sk, NULL);
kernel_sock_shutdown(sock, SHUT_RDWR); kernel_sock_shutdown(sock, SHUT_RDWR);
sock_release(sock); sk_release_kernel(sock->sk);
} }
#if IS_ENABLED(CONFIG_IPV6) #if IS_ENABLED(CONFIG_IPV6)
...@@ -254,9 +255,10 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg, ...@@ -254,9 +255,10 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg,
int err; int err;
struct socket *sock = NULL; struct socket *sock = NULL;
err = __sock_create(net, AF_INET6, SOCK_DGRAM, 0, &sock, 1); err = sock_create_kern(AF_INET6, SOCK_DGRAM, 0, &sock);
if (err < 0) if (err < 0)
goto error; goto error;
sk_change_net(sock->sk, net);
if (cfg->ipv6_v6only) { if (cfg->ipv6_v6only) {
int val = 1; int val = 1;
...@@ -301,7 +303,7 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg, ...@@ -301,7 +303,7 @@ int udp_sock_create6(struct net *net, struct udp_port_cfg *cfg,
error: error:
if (sock) { if (sock) {
kernel_sock_shutdown(sock, SHUT_RDWR); kernel_sock_shutdown(sock, SHUT_RDWR);
sock_release(sock); sk_release_kernel(sock->sk);
} }
*sockp = NULL; *sockp = NULL;
return err; return err;
......
...@@ -19,6 +19,7 @@ ...@@ -19,6 +19,7 @@
#include <linux/if_arp.h> #include <linux/if_arp.h>
#include <linux/icmp.h> #include <linux/icmp.h>
#include <linux/suspend.h> #include <linux/suspend.h>
#include <net/dst_metadata.h>
#include <net/icmp.h> #include <net/icmp.h>
#include <net/rtnetlink.h> #include <net/rtnetlink.h>
#include <net/ip_tunnels.h> #include <net/ip_tunnels.h>
...@@ -106,6 +107,7 @@ static int wg_stop(struct net_device *dev) ...@@ -106,6 +107,7 @@ static int wg_stop(struct net_device *dev)
{ {
struct wg_device *wg = netdev_priv(dev); struct wg_device *wg = netdev_priv(dev);
struct wg_peer *peer; struct wg_peer *peer;
struct sk_buff *skb;
mutex_lock(&wg->device_update_lock); mutex_lock(&wg->device_update_lock);
list_for_each_entry(peer, &wg->peer_list, peer_list) { list_for_each_entry(peer, &wg->peer_list, peer_list) {
...@@ -116,7 +118,9 @@ static int wg_stop(struct net_device *dev) ...@@ -116,7 +118,9 @@ static int wg_stop(struct net_device *dev)
wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake); wg_noise_reset_last_sent_handshake(&peer->last_sent_handshake);
} }
mutex_unlock(&wg->device_update_lock); mutex_unlock(&wg->device_update_lock);
skb_queue_purge(&wg->incoming_handshakes); while ((skb = ptr_ring_consume(&wg->handshake_queue.ring)) != NULL)
kfree_skb(skb);
atomic_set(&wg->handshake_queue_len, 0);
wg_socket_reinit(wg, NULL, NULL); wg_socket_reinit(wg, NULL, NULL);
return 0; return 0;
} }
...@@ -157,7 +161,7 @@ static netdev_tx_t wg_xmit(struct sk_buff *skb, struct net_device *dev) ...@@ -157,7 +161,7 @@ static netdev_tx_t wg_xmit(struct sk_buff *skb, struct net_device *dev)
goto err_peer; goto err_peer;
} }
mtu = skb_dst(skb) ? dst_mtu(skb_dst(skb)) : dev->mtu; mtu = skb_valid_dst(skb) ? dst_mtu(skb_dst(skb)) : dev->mtu;
__skb_queue_head_init(&packets); __skb_queue_head_init(&packets);
if (!skb_is_gso(skb)) { if (!skb_is_gso(skb)) {
...@@ -243,14 +247,13 @@ static void wg_destruct(struct net_device *dev) ...@@ -243,14 +247,13 @@ static void wg_destruct(struct net_device *dev)
destroy_workqueue(wg->handshake_receive_wq); destroy_workqueue(wg->handshake_receive_wq);
destroy_workqueue(wg->handshake_send_wq); destroy_workqueue(wg->handshake_send_wq);
destroy_workqueue(wg->packet_crypt_wq); destroy_workqueue(wg->packet_crypt_wq);
wg_packet_queue_free(&wg->decrypt_queue); wg_packet_queue_free(&wg->handshake_queue, true);
wg_packet_queue_free(&wg->encrypt_queue); wg_packet_queue_free(&wg->decrypt_queue, false);
wg_packet_queue_free(&wg->encrypt_queue, false);
rcu_barrier(); /* Wait for all the peers to be actually freed. */ rcu_barrier(); /* Wait for all the peers to be actually freed. */
wg_ratelimiter_uninit(); wg_ratelimiter_uninit();
memzero_explicit(&wg->static_identity, sizeof(wg->static_identity)); memzero_explicit(&wg->static_identity, sizeof(wg->static_identity));
skb_queue_purge(&wg->incoming_handshakes);
free_percpu(dev->tstats); free_percpu(dev->tstats);
free_percpu(wg->incoming_handshakes_worker);
kvfree(wg->index_hashtable); kvfree(wg->index_hashtable);
kvfree(wg->peer_hashtable); kvfree(wg->peer_hashtable);
mutex_unlock(&wg->device_update_lock); mutex_unlock(&wg->device_update_lock);
...@@ -312,7 +315,6 @@ static int wg_newlink(struct net *src_net, struct net_device *dev, ...@@ -312,7 +315,6 @@ static int wg_newlink(struct net *src_net, struct net_device *dev,
init_rwsem(&wg->static_identity.lock); init_rwsem(&wg->static_identity.lock);
mutex_init(&wg->socket_update_lock); mutex_init(&wg->socket_update_lock);
mutex_init(&wg->device_update_lock); mutex_init(&wg->device_update_lock);
skb_queue_head_init(&wg->incoming_handshakes);
wg_allowedips_init(&wg->peer_allowedips); wg_allowedips_init(&wg->peer_allowedips);
wg_cookie_checker_init(&wg->cookie_checker, wg); wg_cookie_checker_init(&wg->cookie_checker, wg);
INIT_LIST_HEAD(&wg->peer_list); INIT_LIST_HEAD(&wg->peer_list);
...@@ -330,16 +332,10 @@ static int wg_newlink(struct net *src_net, struct net_device *dev, ...@@ -330,16 +332,10 @@ static int wg_newlink(struct net *src_net, struct net_device *dev,
if (!dev->tstats) if (!dev->tstats)
goto err_free_index_hashtable; goto err_free_index_hashtable;
wg->incoming_handshakes_worker =
wg_packet_percpu_multicore_worker_alloc(
wg_packet_handshake_receive_worker, wg);
if (!wg->incoming_handshakes_worker)
goto err_free_tstats;
wg->handshake_receive_wq = alloc_workqueue("wg-kex-%s", wg->handshake_receive_wq = alloc_workqueue("wg-kex-%s",
WQ_CPU_INTENSIVE | WQ_FREEZABLE, 0, dev->name); WQ_CPU_INTENSIVE | WQ_FREEZABLE, 0, dev->name);
if (!wg->handshake_receive_wq) if (!wg->handshake_receive_wq)
goto err_free_incoming_handshakes; goto err_free_tstats;
wg->handshake_send_wq = alloc_workqueue("wg-kex-%s", wg->handshake_send_wq = alloc_workqueue("wg-kex-%s",
WQ_UNBOUND | WQ_FREEZABLE, 0, dev->name); WQ_UNBOUND | WQ_FREEZABLE, 0, dev->name);
...@@ -361,10 +357,15 @@ static int wg_newlink(struct net *src_net, struct net_device *dev, ...@@ -361,10 +357,15 @@ static int wg_newlink(struct net *src_net, struct net_device *dev,
if (ret < 0) if (ret < 0)
goto err_free_encrypt_queue; goto err_free_encrypt_queue;
ret = wg_ratelimiter_init(); ret = wg_packet_queue_init(&wg->handshake_queue, wg_packet_handshake_receive_worker,
MAX_QUEUED_INCOMING_HANDSHAKES);
if (ret < 0) if (ret < 0)
goto err_free_decrypt_queue; goto err_free_decrypt_queue;
ret = wg_ratelimiter_init();
if (ret < 0)
goto err_free_handshake_queue;
ret = register_netdevice(dev); ret = register_netdevice(dev);
if (ret < 0) if (ret < 0)
goto err_uninit_ratelimiter; goto err_uninit_ratelimiter;
...@@ -381,18 +382,18 @@ static int wg_newlink(struct net *src_net, struct net_device *dev, ...@@ -381,18 +382,18 @@ static int wg_newlink(struct net *src_net, struct net_device *dev,
err_uninit_ratelimiter: err_uninit_ratelimiter:
wg_ratelimiter_uninit(); wg_ratelimiter_uninit();
err_free_handshake_queue:
wg_packet_queue_free(&wg->handshake_queue, false);
err_free_decrypt_queue: err_free_decrypt_queue:
wg_packet_queue_free(&wg->decrypt_queue); wg_packet_queue_free(&wg->decrypt_queue, false);
err_free_encrypt_queue: err_free_encrypt_queue:
wg_packet_queue_free(&wg->encrypt_queue); wg_packet_queue_free(&wg->encrypt_queue, false);
err_destroy_packet_crypt: err_destroy_packet_crypt:
destroy_workqueue(wg->packet_crypt_wq); destroy_workqueue(wg->packet_crypt_wq);
err_destroy_handshake_send: err_destroy_handshake_send:
destroy_workqueue(wg->handshake_send_wq); destroy_workqueue(wg->handshake_send_wq);
err_destroy_handshake_receive: err_destroy_handshake_receive:
destroy_workqueue(wg->handshake_receive_wq); destroy_workqueue(wg->handshake_receive_wq);
err_free_incoming_handshakes:
free_percpu(wg->incoming_handshakes_worker);
err_free_tstats: err_free_tstats:
free_percpu(dev->tstats); free_percpu(dev->tstats);
err_free_index_hashtable: err_free_index_hashtable:
...@@ -412,6 +413,7 @@ static struct rtnl_link_ops link_ops __read_mostly = { ...@@ -412,6 +413,7 @@ static struct rtnl_link_ops link_ops __read_mostly = {
static void wg_netns_pre_exit(struct net *net) static void wg_netns_pre_exit(struct net *net)
{ {
struct wg_device *wg; struct wg_device *wg;
struct wg_peer *peer;
rtnl_lock(); rtnl_lock();
list_for_each_entry(wg, &device_list, device_list) { list_for_each_entry(wg, &device_list, device_list) {
...@@ -421,6 +423,8 @@ static void wg_netns_pre_exit(struct net *net) ...@@ -421,6 +423,8 @@ static void wg_netns_pre_exit(struct net *net)
mutex_lock(&wg->device_update_lock); mutex_lock(&wg->device_update_lock);
rcu_assign_pointer(wg->creating_net, NULL); rcu_assign_pointer(wg->creating_net, NULL);
wg_socket_reinit(wg, NULL, NULL); wg_socket_reinit(wg, NULL, NULL);
list_for_each_entry(peer, &wg->peer_list, peer_list)
wg_socket_clear_peer_endpoint_src(peer);
mutex_unlock(&wg->device_update_lock); mutex_unlock(&wg->device_update_lock);
} }
} }
......
...@@ -39,21 +39,18 @@ struct prev_queue { ...@@ -39,21 +39,18 @@ struct prev_queue {
struct wg_device { struct wg_device {
struct net_device *dev; struct net_device *dev;
struct crypt_queue encrypt_queue, decrypt_queue; struct crypt_queue encrypt_queue, decrypt_queue, handshake_queue;
struct sock __rcu *sock4, *sock6; struct sock __rcu *sock4, *sock6;
struct net __rcu *creating_net; struct net __rcu *creating_net;
struct noise_static_identity static_identity; struct noise_static_identity static_identity;
struct workqueue_struct *handshake_receive_wq, *handshake_send_wq; struct workqueue_struct *packet_crypt_wq,*handshake_receive_wq, *handshake_send_wq;
struct workqueue_struct *packet_crypt_wq;
struct sk_buff_head incoming_handshakes;
int incoming_handshake_cpu;
struct multicore_worker __percpu *incoming_handshakes_worker;
struct cookie_checker cookie_checker; struct cookie_checker cookie_checker;
struct pubkey_hashtable *peer_hashtable; struct pubkey_hashtable *peer_hashtable;
struct index_hashtable *index_hashtable; struct index_hashtable *index_hashtable;
struct allowedips peer_allowedips; struct allowedips peer_allowedips;
struct mutex device_update_lock, socket_update_lock; struct mutex device_update_lock, socket_update_lock;
struct list_head device_list, peer_list; struct list_head device_list, peer_list;
atomic_t handshake_queue_len;
unsigned int num_peers, device_update_gen; unsigned int num_peers, device_update_gen;
u32 fwmark; u32 fwmark;
u16 incoming_port; u16 incoming_port;
......
PACKAGE_NAME="wireguard" PACKAGE_NAME="wireguard"
PACKAGE_VERSION="1.0.20210424" PACKAGE_VERSION="1.0.20220627"
AUTOINSTALL=yes AUTOINSTALL=yes
BUILT_MODULE_NAME="wireguard" BUILT_MODULE_NAME="wireguard"
......
...@@ -17,7 +17,7 @@ ...@@ -17,7 +17,7 @@
#include <linux/genetlink.h> #include <linux/genetlink.h>
#include <net/rtnetlink.h> #include <net/rtnetlink.h>
static int __init mod_init(void) static int __init wg_mod_init(void)
{ {
int ret; int ret;
...@@ -65,7 +65,7 @@ err_allowedips: ...@@ -65,7 +65,7 @@ err_allowedips:
return ret; return ret;
} }
static void __exit mod_exit(void) static void __exit wg_mod_exit(void)
{ {
wg_genetlink_uninit(); wg_genetlink_uninit();
wg_device_uninit(); wg_device_uninit();
...@@ -73,8 +73,8 @@ static void __exit mod_exit(void) ...@@ -73,8 +73,8 @@ static void __exit mod_exit(void)
wg_allowedips_slab_uninit(); wg_allowedips_slab_uninit();
} }
module_init(mod_init); module_init(wg_mod_init);
module_exit(mod_exit); module_exit(wg_mod_exit);
MODULE_LICENSE("GPL v2"); MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("WireGuard secure network tunnel"); MODULE_DESCRIPTION("WireGuard secure network tunnel");
MODULE_AUTHOR("Jason A. Donenfeld <Jason@zx2c4.com>"); MODULE_AUTHOR("Jason A. Donenfeld <Jason@zx2c4.com>");
......
...@@ -4,6 +4,7 @@ ...@@ -4,6 +4,7 @@
*/ */
#include "queueing.h" #include "queueing.h"
#include <linux/skb_array.h>
struct multicore_worker __percpu * struct multicore_worker __percpu *
wg_packet_percpu_multicore_worker_alloc(work_func_t function, void *ptr) wg_packet_percpu_multicore_worker_alloc(work_func_t function, void *ptr)
...@@ -38,11 +39,11 @@ int wg_packet_queue_init(struct crypt_queue *queue, work_func_t function, ...@@ -38,11 +39,11 @@ int wg_packet_queue_init(struct crypt_queue *queue, work_func_t function,
return 0; return 0;
} }
void wg_packet_queue_free(struct crypt_queue *queue) void wg_packet_queue_free(struct crypt_queue *queue, bool purge)
{ {
free_percpu(queue->worker); free_percpu(queue->worker);
WARN_ON(!__ptr_ring_empty(&queue->ring)); WARN_ON(!purge && !__ptr_ring_empty(&queue->ring));
ptr_ring_cleanup(&queue->ring, NULL); ptr_ring_cleanup(&queue->ring, purge ? __skb_array_destroy_skb : NULL);
} }
#define NEXT(skb) ((skb)->prev) #define NEXT(skb) ((skb)->prev)
......
...@@ -23,7 +23,7 @@ struct sk_buff; ...@@ -23,7 +23,7 @@ struct sk_buff;
/* queueing.c APIs: */ /* queueing.c APIs: */
int wg_packet_queue_init(struct crypt_queue *queue, work_func_t function, int wg_packet_queue_init(struct crypt_queue *queue, work_func_t function,
unsigned int len); unsigned int len);
void wg_packet_queue_free(struct crypt_queue *queue); void wg_packet_queue_free(struct crypt_queue *queue, bool purge);
struct multicore_worker __percpu * struct multicore_worker __percpu *
wg_packet_percpu_multicore_worker_alloc(work_func_t function, void *ptr); wg_packet_percpu_multicore_worker_alloc(work_func_t function, void *ptr);
......
...@@ -188,12 +188,12 @@ int wg_ratelimiter_init(void) ...@@ -188,12 +188,12 @@ int wg_ratelimiter_init(void)
(1U << 14) / sizeof(struct hlist_head))); (1U << 14) / sizeof(struct hlist_head)));
max_entries = table_size * 8; max_entries = table_size * 8;
table_v4 = kvzalloc(table_size * sizeof(*table_v4), GFP_KERNEL); table_v4 = kvcalloc(table_size, sizeof(*table_v4), GFP_KERNEL);
if (unlikely(!table_v4)) if (unlikely(!table_v4))
goto err_kmemcache; goto err_kmemcache;
#if IS_ENABLED(CONFIG_IPV6) #if IS_ENABLED(CONFIG_IPV6)
table_v6 = kvzalloc(table_size * sizeof(*table_v6), GFP_KERNEL); table_v6 = kvcalloc(table_size, sizeof(*table_v6), GFP_KERNEL);
if (unlikely(!table_v6)) { if (unlikely(!table_v6)) {
kvfree(table_v4); kvfree(table_v4);
goto err_kmemcache; goto err_kmemcache;
......
...@@ -117,8 +117,8 @@ static void wg_receive_handshake_packet(struct wg_device *wg, ...@@ -117,8 +117,8 @@ static void wg_receive_handshake_packet(struct wg_device *wg,
return; return;
} }
under_load = skb_queue_len(&wg->incoming_handshakes) >= under_load = atomic_read(&wg->handshake_queue_len) >=
MAX_QUEUED_INCOMING_HANDSHAKES / 8; MAX_QUEUED_INCOMING_HANDSHAKES / 8;
if (under_load) { if (under_load) {
last_under_load = ktime_get_coarse_boottime_ns(); last_under_load = ktime_get_coarse_boottime_ns();
} else if (last_under_load) { } else if (last_under_load) {
...@@ -213,13 +213,14 @@ static void wg_receive_handshake_packet(struct wg_device *wg, ...@@ -213,13 +213,14 @@ static void wg_receive_handshake_packet(struct wg_device *wg,
void wg_packet_handshake_receive_worker(struct work_struct *work) void wg_packet_handshake_receive_worker(struct work_struct *work)
{ {
struct wg_device *wg = container_of(work, struct multicore_worker, struct crypt_queue *queue = container_of(work, struct multicore_worker, work)->ptr;
work)->ptr; struct wg_device *wg = container_of(queue, struct wg_device, handshake_queue);
struct sk_buff *skb; struct sk_buff *skb;
while ((skb = skb_dequeue(&wg->incoming_handshakes)) != NULL) { while ((skb = ptr_ring_consume_bh(&queue->ring)) != NULL) {
wg_receive_handshake_packet(wg, skb); wg_receive_handshake_packet(wg, skb);
dev_kfree_skb(skb); dev_kfree_skb(skb);
atomic_dec(&wg->handshake_queue_len);
cond_resched(); cond_resched();
} }
} }
...@@ -562,22 +563,28 @@ void wg_packet_receive(struct wg_device *wg, struct sk_buff *skb) ...@@ -562,22 +563,28 @@ void wg_packet_receive(struct wg_device *wg, struct sk_buff *skb)
case cpu_to_le32(MESSAGE_HANDSHAKE_INITIATION): case cpu_to_le32(MESSAGE_HANDSHAKE_INITIATION):
case cpu_to_le32(MESSAGE_HANDSHAKE_RESPONSE): case cpu_to_le32(MESSAGE_HANDSHAKE_RESPONSE):
case cpu_to_le32(MESSAGE_HANDSHAKE_COOKIE): { case cpu_to_le32(MESSAGE_HANDSHAKE_COOKIE): {
int cpu; int cpu, ret = -EBUSY;
if (skb_queue_len(&wg->incoming_handshakes) > if (unlikely(!rng_is_initialized()))
MAX_QUEUED_INCOMING_HANDSHAKES || goto drop;
unlikely(!rng_is_initialized())) { if (atomic_read(&wg->handshake_queue_len) > MAX_QUEUED_INCOMING_HANDSHAKES / 2) {
if (spin_trylock_bh(&wg->handshake_queue.ring.producer_lock)) {
ret = __ptr_ring_produce(&wg->handshake_queue.ring, skb);
spin_unlock_bh(&wg->handshake_queue.ring.producer_lock);
}
} else
ret = ptr_ring_produce_bh(&wg->handshake_queue.ring, skb);
if (ret) {
drop:
net_dbg_skb_ratelimited("%s: Dropping handshake packet from %pISpfsc\n", net_dbg_skb_ratelimited("%s: Dropping handshake packet from %pISpfsc\n",
wg->dev->name, skb); wg->dev->name, skb);
goto err; goto err;
} }
skb_queue_tail(&wg->incoming_handshakes, skb); atomic_inc(&wg->handshake_queue_len);
/* Queues up a call to packet_process_queued_handshake_ cpu = wg_cpumask_next_online(&wg->handshake_queue.last_cpu);
* packets(skb): /* Queues up a call to packet_process_queued_handshake_packets(skb): */
*/
cpu = wg_cpumask_next_online(&wg->incoming_handshake_cpu);
queue_work_on(cpu, wg->handshake_receive_wq, queue_work_on(cpu, wg->handshake_receive_wq,
&per_cpu_ptr(wg->incoming_handshakes_worker, cpu)->work); &per_cpu_ptr(wg->handshake_queue.worker, cpu)->work);
break; break;
} }
case cpu_to_le32(MESSAGE_DATA): case cpu_to_le32(MESSAGE_DATA):
......