**** BEGIN LOGGING AT Tue Nov 26 02:59:57 2019 Nov 26 07:19:34 stintel: just adjust CI to build with x86/64 SDK as well? in openwrt-ci I wanted to have arm/mips, little/big endian and 32/64 bit coverage so it builds with ath79, malta-be, imx6, mvebu-cortexa53 SDKs + natively on x86/64 (gcc-7, gcc-8, gcc-9, clang-10) and I'm sure, that openwrt-ci could be refactored little bit to support building of packages from feeds as well Nov 26 07:21:58 it currently works with GitLab, but adding support for CircleCI should be doable Nov 26 07:22:48 although it would probably take ages to complete for some packages... Nov 26 07:28:07 hexa-: no idea at all, sorry Nov 26 07:28:20 I'm afaraid you need to debug it yourself, I am unable to reproduce it Nov 26 07:28:48 start with make package/iwinfo/{clean,compile} V=s and see if it is built with nl80211 support in the first place Nov 26 07:36:28 KanjiMonster: https://gitlab.com/ynezz/openwrt-libubox/commit/a68360b99259188904729a82d8dc2280b04c323c can you test it? Nov 26 08:14:29 has anyone started working on kernel 5.x yet? Nov 26 08:30:07 nitroshift: xback has I believe Nov 26 08:34:27 mangix, thanks, i believe he has an offline repo he works on, there's nothing on openwrt git Nov 26 08:34:45 i mean in his staging tree Nov 26 08:45:53 ynezz: I'll do some more testing locally with the issues I'm currently seeing. if they turn out to be "host leakage" related I might just do that Nov 26 08:46:17 jow: that was wrt https://bugs.openwrt.org/index.php?do=details&task_id=2629 which your patch seems to have fixed Nov 26 08:48:48 mangix: I know about the mitigations, but In the article it was not stated that the slowdown were caused by only that Nov 26 08:49:33 Maybe we should add an option to allow setting the mitigations levels. Which are all enabled by default? Nov 26 08:49:49 if a user want extra speed at the cost of security, they can at their own risk then Nov 26 08:50:57 nitroshift: I just started yesterday working on it yesterday morning. Currently focusing on x86. If it's in a working state I'll push to staging for testing Nov 26 08:51:21 xback, thanks Nov 26 08:51:27 ldir: thank you for the patch Nov 26 08:58:23 xback: thanks for the nudge to look into possible bitrot - which I found. It's been compile tested 4.19 & 4.14 & run tested 4.19 x86. Not run tested 4.14... I don't anticipate any problems but....... :-) Nov 26 09:01:32 xback: we shouldn't encourage bad practices by adding options to mitigate mitigations :) Nov 26 09:05:18 Do we disable hyperthreading on Intel by default? Nov 26 09:06:59 if you have malicious processes on your openwrt this mitigations won't help any more... Nov 26 09:07:30 as openwrt isn't am multiuser system most of the vulnerabilities are likely much less critical Nov 26 09:07:31 so i don't think all those meltdown/spectre like problems belong to openwrt Nov 26 09:11:36 Yet it is multiuser system by many processes... does not need explicit "useradd" or so program.. Nov 26 09:14:01 One compiling openwrt oneself, or alike, can do whatever one desires, for default we really shouldn't suddenly start doing bad practices with this one Nov 26 09:17:20 sure, secure by default is a good aim. especially since x86 devices are mostly beefy enough anyway Nov 26 09:18:01 "Security" is always hard in a sense.. million things needs to go proper and often only one needs to fail and "done for" Nov 26 09:22:44 isn't it just matter of time, till someone exploits this cache/branch prediction/SMT bugs over other external side channel, like ethernet,wireless,usb for example? I mean it looks almost impossible, but what is impossible today might be doable tomorrow, right? Nov 26 09:29:05 Often times that is the case Nov 26 09:36:59 ldir: if we disable HT as OpenBSD does, shouldn't we as well switch to OpenSSH on !SMALL_FLASH devices? :p Nov 26 09:38:21 Why not Postgres Nov 26 09:39:01 Just don't run Intel :-) Nov 26 09:40:40 olmari: Dropbear -> OpenSSH Nov 26 09:41:59 Triple node Postgres to store root access, and then LDAP login ;) Nov 26 09:46:47 are there security-problems in dropbear? Nov 26 09:47:36 supporting more than one service for the same task raises the complexity which may cause other security holes... Nov 26 09:50:51 Dropbear is way smaller.. hence general default ssh provider Nov 26 09:51:15 smaller means less lines of code. less lines of code have potentially less bugs Nov 26 09:55:10 whats the problem with just applying the kernel.org default mitigation choices? Nov 26 09:55:39 grub is easily reachable on x86, people can stuff their preferred magic bootargs in there if needed Nov 26 10:12:39 stintel: I'm on v19.07.0-rc1-94-ga0897f8a46 on an archer c7 v2 (qca988x) with ath10k-ct and an intel 8265/8275 client with mfp enabled. I get pretty consistent 450 Mbit/s rn. Nov 26 10:13:32 hexa-: both directions? Nov 26 10:14:41 stintel: yes both directions, with wpa2-eap (ttls/pap) Nov 26 10:15:10 maybe it's related to your intel firmware? Nov 26 10:15:16 interesting. what kernel are you running on the client? Nov 26 10:15:20 and firmware ? Nov 26 10:15:21 5.3.11 Nov 26 10:15:30 good question :) Nov 26 10:15:37 ethtool shows it afaik Nov 26 10:15:43 * stintel grabs his laptop Nov 26 10:15:58 Settings for wlp2s0: Nov 26 10:16:00 Link detected: yes Nov 26 10:16:02 yay :D Nov 26 10:20:27 36.77d01142.0 via dmesg | grep "firmware version" Nov 26 10:24:20 also the exact card is Intel Corporation Wireless 8265 / 8275 [8086:24fd] (rev 78) Nov 26 10:35:31 version: 5.3.12-gentoo Nov 26 10:35:31 firmware-version: 36.77d01142.0 Nov 26 10:36:01 that kernel is one reboot away, but probably not the culprit Nov 26 10:36:01 sorry had to unclutter my desk for a moment I was going insane Nov 26 10:36:12 no, as I have this issue at least since september Nov 26 10:36:39 so I've seen other issues with intel 8265 on many 6th gen and newer thinkpads Nov 26 10:37:04 this is an "old" 8265 in an old XPS13 (9343) Nov 26 10:37:12 they would connect but have heavy packet loss on the first hop already Nov 26 10:37:24 after Ben fixed MFP in the ath10k-ct (or the firmware), I had similar speeds in both directions Nov 26 10:37:31 this happens with unifi ac-lite/mesh with original firmware, which is why I'm tryin to migrate them to openwrt now Nov 26 10:38:03 as soon as the thinkpads would be connected to power or ethernet the loss would go away Nov 26 10:38:14 like grounding was missing before Nov 26 10:38:28 otoh it could be power saving related Nov 26 10:39:19 also the archer c7 v2 maxes out the cpu with 98% softirq at 450 Mbit/s, that feels wrong, but it doesn't have vlan offload … so rip. Nov 26 10:40:37 lol. I try to reproduce it now and I can't Nov 26 10:40:38 wtf Nov 26 10:41:23 ~160Mbps from the client to the AP and ~213Mbps from the AP to the client (actually to/from a wired host behind the AP) Nov 26 10:42:15 let's see if the firmware was recently updated Nov 26 10:42:43 according to journal I have mine since at least october 29th Nov 26 10:42:50 looks like it's going to be that Nov 26 10:43:09 ill reboot back into nixos 19.03 and check what that has. brb Nov 26 10:44:23 hmm no I am on this firmware since sep 26 19:38:08 sylvester kernel: iwlwifi 0000:02:00.0: loaded firmware version 36.77d01142.0 op_mode iwlmvm Nov 26 10:44:50 and I had the problem earlier. sep 25 11:23:13 sylvester kernel: iwlwifi 0000:02:00.0: loaded firmware version 36.8fd77bb3.0 op_mode iwlmvm Nov 26 10:45:00 and now I don't have it on 36.77d01142.0 Nov 26 10:45:02 ah but wait Nov 26 10:45:13 MFP: no Nov 26 10:45:18 right, I disabled MFP in NetworkManager Nov 26 10:46:07 meh, I garbage collected 19.03 :3 Nov 26 10:46:30 hexa-: just to be sure, you verified MFP is actually enabled with `iw wlanX station dump` on either the AP or client ? Nov 26 10:46:36 or maybe even check it on both, who knows Nov 26 10:46:42 yep Nov 26 10:46:44 ap Nov 26 10:47:38 i'll reconfirm Nov 26 10:48:40 yes, both sides say "MFP: yes" Nov 26 10:49:09 interesting. now even with with "MFP: yes" on both sides I cannot reproduce Nov 26 10:49:20 https://bpaste.net/show/ZIR54 Nov 26 10:50:21 I am 100% sure I had this problem more than once. every time I travel from my home country back to my current home I take a bunch of data with me and it has been uploading at a terribly slow rate several times Nov 26 10:52:17 * stintel checks kernel changelogs Nov 26 10:56:19 and I accidentally wiped all my previous kernels on the laptop so cannot check the older ones easily either (gentoo on ultrabook, kernels are the only thing I compile locally and it takes a long time) Nov 26 10:57:22 hmm now this is from a cold boot (battery died overnight) and no suspend. maybe it happens only after suspend Nov 26 11:04:35 ¯\_(ツ)_/¯ Nov 26 11:04:55 so do we conclude: not openwrt's fault? Nov 26 11:04:57 ship it! :) Nov 26 11:57:17 hexa-: inconclusive. the fact that Hauke is seeing it too means there is something wrong, we just need to make sure if it is the client or OpenWrt Nov 26 12:15:46 * xback is getting a headache from going through netfilter code all morning Nov 26 12:16:01 sjeezes .. what a complex beast Nov 26 12:28:38 I presume you looked at the kernel side? Nov 26 12:40:49 is there a change to activate hardware ip accelleration of lantiq soc as in the fritzbox 3370? Nov 26 12:40:56 with openwrt Nov 26 12:41:52 xback: reminds me of libipset Nov 26 12:42:06 probably not near as complex as netfilter but for a simple mind like me ... jeebus Nov 26 12:53:31 jow: correct Nov 26 14:20:14 stintel: which one (qca9886 or qca9888?) do you see that low RX issue with? Nov 26 14:29:36 pepe2k: no clue, dmesg says 988x - also I cannot reproduce it right now Nov 26 14:30:21 [ 13.905196] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000 Nov 26 14:31:59 I have on desk AP with qca9886 and Intel card in PC, could double check it Nov 26 14:33:22 well I think we should get to the bottom of this before we can release 19.07 Nov 26 14:33:42 40Mbps upload is dog slow if you know it should be able to do 400 Nov 26 14:38:02 not that I care that much, but is it really necessary to block whole release just because of this? I mean you can fix it in the next minor release 19.07.1 Nov 26 14:39:00 anyone can come up with some important regression and block the release indefinitely Nov 26 14:40:29 I mean, there are many issues we're not aware about yet Nov 26 14:40:34 abysmal performance between probably the most used 11ac chip/driver used in AP and probably the most used 11ac chip/driver in clients? Nov 26 14:40:58 I'd say for a distribution targeting mostly wireless routers this is pretty damn important Nov 26 14:41:41 I agree with both of you :) Nov 26 14:43:08 but it probably makes more sense to ship release now with information that we know about the problem than delaying it again Nov 26 14:46:49 why, we're wayyyy over any sort of deadline anyway, what would be the point in artificually releasing it in such a state? Nov 26 14:47:54 nbd: https://github.com/csyuanc/mt76/commits/master, is safe to try it? :) Nov 26 14:50:25 karlp: to get wider feedback and allow unaffected users to use it? if this problem really applies only to a specific combination of AP and STA radios, even if they are popular, there are other users which won't be affected Nov 26 14:50:56 (and it can't be reproduced reliably) Nov 26 14:51:05 I wouldn't call this problem a critical one and if I read the mail correctly, there is a workaround Nov 26 14:51:15 ie hexa doesnt have this issue with similar combination Nov 26 14:51:59 could it be Intel fw specific? Nov 26 14:52:22 environment specific, it's wifi Nov 26 14:52:47 doubt it, my Intel firmware did not change between good and bad attempts Nov 26 14:53:45 and environment specific ... I've had both bad and good attempts in the same location. and 5GHz should be much less prone to interference Nov 26 14:54:30 or that fw could contain some bug, triggered by certain step Nov 26 14:54:40 so it might be client specific as well Nov 26 14:54:41 stintel: ynezz: I could probably setup RF cable tests, it's just 2x2 so not that many connections ;) but no earlier than over weekend Nov 26 14:55:00 and for the workaround .. disabling MFP could have just "fixed" it due to reconnect Nov 26 14:56:32 anyway, what I would mostly like to avoid is that people blame this on ath10k-ct and cause such a big fuzz we decide to go back to ath10k Nov 26 14:57:58 stintel: I don't think anyone would want to go back to ath10k :) Nov 26 14:58:10 and nobody has provided so far any credible evidence, that it's a -ct related issue Nov 26 14:58:41 people will see this as the big change and blame it automatically Nov 26 14:58:52 and I believe, that ben could fix it if we can bisect it Nov 26 15:00:22 there is -rc2 planned anyway, right? today if I'm not mistaken, so probably some time before release Nov 26 15:00:54 sure -rc2 sounds fine, probably mention these issues in the announcement and call for testing Nov 26 15:00:57 ynezz: I think we all were thinking about rc2, not the final one, correct? Nov 26 15:01:26 and we should probably also write something about WPA3 ... Nov 26 15:01:50 because I'm 100% sure people try to enable WPA3, but have clients that don't connect due to that, and then also blame it on something else Nov 26 15:02:14 I cannot enable WPA3, it breaks several clients, even in mixed mode with wpa2 Nov 26 15:02:18 release notes are on the wiki, feel free to add missing bits :) Nov 26 15:04:03 Then again for me SAE-mixed has worked 100% for old clients too, at least connectivity wise, but in theory can't rule its effect on some case slow downstream speeds either Nov 26 15:04:54 I'm still on case it happends but haven't got time to debug properly nor even half assed Nov 26 15:07:56 olmari: SAE-mixed breaks connectivity for some clients here Nov 26 15:18:54 I also think I tried to use it on RPi0 running OpenWrt as STA and it did not work Nov 26 15:19:32 maybe it could be useful to create an overview of devices that were tested with WPA3? Nov 26 15:19:49 including devices that do not work Nov 26 15:56:07 s Nov 26 16:37:50 Hi people I tryed wpa 3 out on my wrt3200acm and it did not work. Nov 26 16:38:25 I set it to mixt moad and a laptop with a intel wifi card and a GS8 android phone all so would not work. Nov 26 16:39:08 My soni android TV would connect but then streems would slow down so mutch that they would drop Nov 26 17:44:51 adrianschmutzler: ynezz: when you do cherry picking, do you show somehow that the change wasn't cleanly applied and some modifications had to be done? Nov 26 17:45:00 I usually use "cherry picked from..." for clean ones and "backported from..." for other but I'm not sure about this approach Nov 26 18:44:53 I just use `git cherry-pick -x` Nov 26 23:36:14 hmm, we switched ath10k-ct using the non-htt -ct firmware Nov 26 23:36:26 I am using the htt variant Nov 27 01:10:09 greearb_: do you recommend using the htt-mgt variant or not ? Nov 27 01:11:55 htt-mgt is suggested with ath10k-ct driver, if using stock driver, must use non-htt-mgt version Nov 27 01:12:34 so since we switched to ath10k-ct by default, it makes more sense to enable the htt-mgt firmwares by default Nov 27 01:13:33 ath10k-ct will try to load a ct-named firmware first, so the htt version could be called that, and non-htt version could be added at old location if wanted..then it will work no matter what driver you use. Nov 27 01:13:59 ynezz: Hauke: jow: ^ Nov 27 01:14:48 greearb_: thanks Nov 27 01:15:11 I forget the exact names, can look it up sometime later, or just look at the ath10k-ct driver where it tries to load fw Nov 27 01:21:12 hmmm looks like I have the slow upload on my AP with an older version of the htt-mgt firmware: ath10k-firmware-qca988x-ct-htt - 2019-06-28-7651f5bb-1 Nov 27 01:21:34 on my AP with this one it's fine: ath10k-firmware-qca988x-ct-htt - 2019-10-03-d622d160-1 Nov 27 01:22:30 weird though, I thought I just sysupgraded that one with the old firmware Nov 27 01:24:41 * stintel grabs the console cable to see wtf sysupgrade is doing Nov 27 01:32:40 it definitely did not upgrade :/ Nov 27 01:34:50 great, even the console port is broken Nov 27 01:41:09 ah, it's the problem russell-- found and fixed Nov 27 01:41:15 thanks for that russell-- Nov 27 01:41:38 what? Nov 27 01:41:54 sysupgrade missing jshn.sh or so in stage2 Nov 27 01:42:02 ah, yeah Nov 27 01:42:05 how do you even figure that out if you don't have serial console Nov 27 01:42:07 fucking sysupgrade Nov 27 01:42:13 sigh Nov 27 01:42:16 serial console++ Nov 27 01:42:28 apparently the one on my DAP-2695 doesn't work anymore Nov 27 01:43:02 you can fix it manually Nov 27 01:43:05 I did Nov 27 01:43:09 yeah Nov 27 01:43:27 but I had to /sysupgrade in git log Nov 27 01:43:30 that's guesswork Nov 27 01:43:49 there should really be a way to figure this out without serial console Nov 27 01:43:59 maybe sysupgrade should write somewhere its failed attempts with more info Nov 27 01:45:24 well, the other way of looking at it is: never be broken Nov 27 01:45:42 good luck with that Nov 27 01:45:46 lol Nov 27 01:47:45 sooo interesting, when my 8265 is connected to my ath79 AP upload is fine, when my 8265 is connected to my ar71xx AP upload from client to AP is only 10% of what I get on the ath79 AP Nov 27 01:47:50 both have [ 13.953930] ath10k_pci 0000:00:00.0: qca988x hw2.0 target 0x4100016c chip_id 0x043202ff sub 0000:0000 Nov 27 01:48:18 and both have ath10k-firmware-qca988x-ct-htt - 2019-10-03-d622d160-1 now Nov 27 01:49:17 Hauke: is your BTHH5A ar71xx or ath79? Nov 27 01:49:52 oh, neither? Nov 27 01:50:58 the difference between ath79 and ar71xx is kernel 4.19 vs kernel 4.14 Nov 27 01:52:24 aha, lantiq is 4.14 on 19.07 Nov 27 01:52:33 so maybe the problem is related to the kernel Nov 27 01:54:51 anyway, zzz **** ENDING LOGGING AT Wed Nov 27 02:59:57 2019