**** BEGIN LOGGING AT Sun Nov 03 02:59:59 2019 Nov 03 03:20:28 oh cool, I just hosed my linux install simply by using btrfs Nov 03 03:20:31 lesson learned Nov 03 03:48:37 jow: I changed the cdn cache header to 60 minutes instead of 7 days, I had the problem that snapshot files on the cdn would be outdated, I hope that was the right option to fix that. Nov 03 06:29:20 build #145 of sunxi/cortexa7 is complete: Failure [failed pkgbuild] Build details are at http://buildbot.openwrt.org/master/images/builders/sunxi%2Fcortexa7/builds/145 blamelist: Rasim Kalimullin , Sebastian Kemper , Yousong Zhou , Paul Spooren , Adrian Schmutzler Nov 03 06:29:20 , Roger Pueyo Centelles , Daniel Danzberger , Chih-Wei Chen , Kyle Copperfield , Michal Cieslakiewicz , Alexander Couzens Nov 03 10:21:50 strongswan-mod-eap-mschapv2 package seems to be broken lol Nov 03 10:22:45 ipsec statusall doesn't list mschapv2 as loaded plugin Nov 03 10:39:40 m0f0: ipsec listplugins | grep -i chap Nov 03 10:41:54 last time I used it it worked just fine, but this was end 2018 Nov 03 10:43:41 m0f0: works just fine, did you restart strongswan after installing the plugin? Nov 03 10:44:38 auto-restarting in package post-install is a bad idea as it will bring down established tunnels Nov 03 10:44:58 yes, I did strongswan restart (via /etc/init.d script) Nov 03 10:45:12 "ipsec listplugins | grep -i chap" gives nothing Nov 03 10:45:24 interesting, I just installed it here, restarted strongswan and the plugin shows Nov 03 10:45:50 stintel: maybe it's broken only on my platform? x86 geode Nov 03 10:45:55 m0f0: I'm on x86/64 Nov 03 10:46:10 weird... more or less same platform Nov 03 10:46:11 m0f0: `opkg files strongswan-mod-eap-mschapv2` shows /etc/strongswan.d/charon/eap-mschapv2.conf and /usr/lib/ipsec/plugins/libstrongswan-eap-mschapv2.so ? Nov 03 10:46:26 one moment please Nov 03 10:46:43 yeah, just those two Nov 03 10:47:00 ok, and /etc/strongswan.d/charon/eap-mschapv2.conf has load = yes ? Nov 03 10:47:05 yup Nov 03 10:47:55 stintel: maybe I should reboot the router? (but I'd rather avoid it) Nov 03 10:48:05 ok, maybe the restart didn't work for some reason, can you /etc/init.d/ipsec stop, verify if all processes are down ? Nov 03 10:48:21 one moment Nov 03 10:48:29 if not, kill them manually then restart it via init script Nov 03 10:48:46 ...and processes are *not* down Nov 03 10:49:03 maybe check logs for pointers why not, I've seen it on a few occasions Nov 03 10:49:49 oh yeah, it helped :))) Nov 03 10:49:58 eap-mschapv2: EAP_SERVER:MSCHAPV2 EAP_CLIENT:MSCHAPV2 Nov 03 10:50:15 yay Nov 03 10:50:35 strangely strongswan still fails to connect to my VPN server Nov 03 10:51:00 (which is weird because Apple devices connect, Android phone connects too and even Linux desktop does) Nov 03 10:51:03 you might need other modules, I vaguely remember something like that from when I was using it Nov 03 10:51:22 let me see if I can still find documentation about that in our company wiki Nov 03 10:52:00 can you maybe post the output somewhere? what you get when trying to initiate the connection ? Nov 03 10:52:29 loaded plugins: charon des md4 random nonce x509 pubkey pkcs1 pkcs8 pem openssl curve25519 attr kernel-netlink socket-default connmark farp stroke vici updown eap-identity eap-mschapv2 xauth-generic unity Nov 03 10:52:42 do you have eap-identity and xauth-generic installed ? Nov 03 10:54:30 https://paste.mgst.eu/?318fbb7cad86b448#5U4VQRYEeaRrQGAB7Mx9TB3T76vdZNkWWcSgb7krgVfg that's output of "ipsec up nyx" Nov 03 10:54:30 well I would try each one of those I listed, maybe pkcs1 after trying eap-identity, restart every time and make sure the processes are actually gone Nov 03 10:54:55 received netlink error: No such file or directory (2) Nov 03 10:55:25 hmmm, you're missing some kernel module, or your kernel vs module versions are mismatched Nov 03 10:55:57 these are matched for sure (18.06.4 installed from scratch two weeks ago or something like this) Nov 03 10:56:00 that's a major pita to debug, when you installed strongswan, did it try to install kernel modules ? Nov 03 10:56:19 ah, maybe you simply need to modprobe some modules Nov 03 10:56:20 yeah, I've got some of them Nov 03 10:56:28 not sure if they are loaded automatically Nov 03 10:56:34 after opkg install Nov 03 10:56:41 as far as I can see they do Nov 03 10:56:53 you have esp4 loaded for example ? Nov 03 10:56:57 (at least modules for LTE modem to work) Nov 03 10:57:29 https://paste.mgst.eu/?8d317c981af6254b#9Gk5f3VnTd3wxLCyU4Ea6xjsSks9dGY3HANpTcVPmR5y here's my lsmod output Nov 03 10:57:57 esp4 already loaded Nov 03 10:57:59 indeed Nov 03 10:58:59 what's your esp= ? Nov 03 10:59:30 can't see any proposals in your paste fron ipsec up nyx Nov 03 11:00:01 there's no esp= statement in ipsec.conf on this device Nov 03 11:00:06 should there be one? Nov 03 11:00:31 ok, that's probably not the best idea. we have no clue what the server is proposing and it's not logged Nov 03 11:00:49 "ike=aes128-sha256-modp1024" to try using Geode's AES acceleration Nov 03 11:00:52 m0f0: set this: Nov 03 11:00:53 ike=aes256-sha1-modp2048! Nov 03 11:00:53 esp=aes256-sha1-modp2048! Nov 03 11:01:08 with the ! it will not allow anything else, but should clearly log our proposal vs remote end proposal Nov 03 11:01:11 I'll show you what's set on server Nov 03 11:01:14 ah Nov 03 11:01:21 you maintain the server ? Nov 03 11:01:21 (it's mine too lol) Nov 03 11:01:23 ok Nov 03 11:01:23 yes Nov 03 11:02:25 nothing set as esp= on the server Nov 03 11:02:45 ike=chacha20poly1305-prfsha512-ecp512bp,aes256gcm16-prfsha512-ecp512bp,aes128-sha256-modp1024 this set as ike= on the server Nov 03 11:03:38 m0f0: just try setting esp=aes256-sha1-modp2048! on the client, ipsec update && ipsec up nix Nov 03 11:03:44 and pastebin the logs again Nov 03 11:03:56 I've tried with AES128 because Geode theoretically supports hardware assistance of it Nov 03 11:04:11 AES256 would just destroy that 500MHz CPU Nov 03 11:04:20 but ok, let's try that Nov 03 11:04:35 well replace it with 128, it's just to hopefully get usefull logging Nov 03 11:05:10 I would also recommend to configure both ike and esp on both ends, and limit to what you configured by ending these config lines with ! Nov 03 11:05:39 it might need some fine-tuning if you're using android, apple and linux Nov 03 11:05:49 but explicit > implicit Nov 03 11:09:50 https://paste.mgst.eu/?b49351eebc3454ae#5LjpG43mRyVr1uuDkRJfoL8BBLvAnqUj2Q8Bo3bfraQh not much more :( Nov 03 11:10:09 m0f0: can you check `logread` ? Nov 03 11:10:47 might give more info Nov 03 11:11:24 https://paste.mgst.eu/?ec9ef672c9466ece#6CWHVDvegfgqRsTu8mQsnwB86gK38FD2foP4Vix8SoPh I'm reading those last lines Nov 03 11:12:05 Sun Nov 3 11:49:34 2019 kern.err kernel: [904216.612992] Error allocating fallback algo cbc(aes) Nov 03 11:12:23 have not seen this before, might be related to the geode hw crypto Nov 03 11:12:48 @FRA and there is open wifi Nov 03 11:12:58 like not even an eula landing page Nov 03 11:13:05 just open telekom wifi Nov 03 11:13:20 traceroute shows I am actually 1 hop away from a dtag noc ip Nov 03 11:13:30 first thought this is rouge, but looks legit Nov 03 11:13:36 welcome to the new world Nov 03 11:13:45 blogic: who said Germans were unwelcoming ;) Nov 03 11:13:47 stintel: not really, afaics it's broken (entirely, it's just computing AES stuff using CPU cycles) Nov 03 11:14:04 Sun Nov 3 12:09:03 2019 daemon.info : 08[IKE] received NO_PROPOSAL_CHOSEN notify, no CHILD_SA built Nov 03 11:14:50 m0f0: it's best to set ike and esp on both ends. you have aes128-sha256-modp1024 on the server ike, set this on the client: ike=aes128-sha256-modp1024! Nov 03 11:15:59 and maybe set the same for esp on both ends (just to experiment right now) Nov 03 11:16:14 you should avoid modp <=2048 for example, they are no longer considered secure Nov 03 11:16:32 ike= has to be the same thing as esp=, right? Nov 03 11:16:52 nope Nov 03 11:16:59 it's completely unrelated Nov 03 11:21:41 ok, updated to correct modp and generally matching settings on client and server Nov 03 11:21:54 any difference in output ? Nov 03 11:22:18 none :( Nov 03 11:22:49 and can you check the server output ? grep -i proposal ? Nov 03 11:24:10 or follow the server log while trying to connect and pastebin the stuff that appeared Nov 03 11:24:32 I'm doing that atm Nov 03 11:24:35 (the latter) Nov 03 11:25:18 ...and output on client mysteriously changed, wtf Nov 03 11:25:32 ipsec is ... fun Nov 03 11:26:28 I'm really tempted to fsck it and just drop PPTP tunnel Nov 03 11:26:40 dead easy, always works but insecure as hell lol Nov 03 11:26:43 anyway, I'm starting to suspect it's geode related after all Nov 03 11:27:02 I can uninstall hw support module and reboot Nov 03 11:27:15 w8, let me paste something Nov 03 11:27:25 I was considering asking that. I googled a bit and it seems it could have problems with aes in cbc mode Nov 03 11:27:44 so maybe you can just change aes128 in esp to aes128gcm16 Nov 03 11:27:48 what the actual fsck, 4 minutes time drift between server and client Nov 03 11:27:59 wait let me verify that Nov 03 11:28:31 aes128gcm128 Nov 03 11:28:37 or aes128gcm16 - basicly the same Nov 03 11:28:46 yikees Nov 03 11:28:49 try that, it will not use cbc mode then Nov 03 11:29:10 i am still suffering from debugging the mt7623 ipsec crypto offload crap last month Nov 03 11:29:14 blogic: ;) Nov 03 11:29:24 it resulted in a kernel patch :-) Nov 03 11:29:28 ipsec is ... [insert_cursing_here] Nov 03 11:29:35 stintel https://paste.mgst.eu/?e65ac4b0af3ba4f3#D3Y1cktdCv9fHTDQK2HZ8mbqggiDcjxx1H9Yf746jDAK Nov 03 11:29:42 even a mother could not love that child Nov 03 11:29:59 m0f0: yeah the problem keeps being unable to install inbound and outbound IPsec SA (SAD) in kernel Nov 03 11:30:13 this is either something missing in the kernel, or something that is broken Nov 03 11:30:18 stintel but that was before changing to aes128gcm128 Nov 03 11:30:24 debugging exactly what is missing is a major pita Nov 03 11:30:32 I had a similar issue with strongswan, the culprit in my case was the module for AMD hw accelerations (hw-ccp). However, SA was installed, etc., but no traffic went through the tunnel Nov 03 11:30:55 strongswan/ipsec is not that bad, you just need to ensure the stars are aligned, tarot cards good, etc. before starting to use it Nov 03 11:31:09 i was just looking at the drivers/net/ethernet/mediatek/gsw_mt7621.c code, is there a reason that hasn't been submitted (or, maybe, accepted) upstream? Nov 03 11:31:11 m0f0: try aes in gcm mode, if that doesn't work, remove that geode aes kernel module Nov 03 11:31:16 ok, so it's worth removing kmod-crypto-hw-geode? Nov 03 11:31:19 kk Nov 03 11:31:24 one moment please Nov 03 11:31:58 it gets more fun with different implementations :) Nov 03 11:32:06 Ah, yes, ipsec interop Nov 03 11:32:26 at some point we were doing strongswan@openwrt, watchguard, azure, and libreswan@centos Nov 03 11:32:36 I love how ever vendor uses different terms to describe the same features Nov 03 11:33:10 and how azure supports aesgcm for esp but not for ike Nov 03 11:33:13 Yeah, I do a lot of strongswan to Palo Alto/Fortinet Nov 03 11:33:20 this web irc client hates me Nov 03 11:33:27 and how people insist ike lifetime and bytes should be exactly the same ;) Nov 03 11:33:33 hehe Nov 03 11:33:33 stintel: so it'll be stintel: so that would be "aes128gcm128-sha256-modp2048!" right? Nov 03 11:33:40 m0f0: for example Nov 03 11:33:44 ok Nov 03 11:34:10 I hope that Fortinet or Palo Alto never changes anything, I ended up making a guide for our users showing exactly what to click/set Nov 03 11:34:39 I made a mobileconfig for our apple users (95% of the company) Nov 03 11:34:51 And yeah, I remember Azure being a pita. When I tried to set up ipsec tunnels to Azure, they insisted on using the IP address as the identifier ... Nov 03 11:35:02 strongswan ipsec.conf and swanctl config for our linux users Nov 03 11:35:08 OK FSCK YEAH Nov 03 11:35:09 and even powershell for windows ;) Nov 03 11:35:14 ipsec up nyx without errors Nov 03 11:35:22 m0f0: ok so it's probably safe to say the problem is the geode-aes stuff Nov 03 11:35:27 Which works great when the routers that were going to be connected used LTE connection with dynamic addresses ... Nov 03 11:35:36 well... nope Nov 03 11:35:41 I dont know why so many vendors do that, use the IP address as default Nov 03 11:35:43 no errors but it doesn't work either Nov 03 11:35:50 is the SA installed? Nov 03 11:35:59 SA? Nov 03 11:35:59 i.e. ipsec statusall shows ... something Nov 03 11:36:05 security assosication Nov 03 11:36:25 ...and ipsec statusall is mute too at the moment Nov 03 11:36:28 if the SA is installed now you're probably hitting another problem :) and the ipsec up .. without errors suggest you have SA installed, but indeed pastebin ipsec statusall ;) Nov 03 11:36:29 let's restart ipsec on client Nov 03 11:36:44 statusall mute sounds like a DNS problem Nov 03 11:36:54 ah, yes, another great thing with ipsec. When something goes wrong with either end of the tunnel ... Nov 03 11:36:59 are you using hostnames in your configs? Nov 03 11:37:23 could be establishing your tunnel messed up connectivity to the DNS server and now it's hanging trying to resolve those Nov 03 11:38:07 https://paste.mgst.eu/?fd14ce23f2bd3826#5u2Z5pongY9vgxyWjB8EB7ApvA37tXT1i47MyU2FnXfi here's ipsec.conf from client Nov 03 11:38:16 (I mean, fragment for that connection) Nov 03 11:39:08 too bad vti sucks hard to configure in openwrt Nov 03 11:39:12 I think I need to add something because ipsec forces (adds as first lines of resolv.conf) DNS proposed by server (which is 1.1.1.1) Nov 03 11:39:46 and those new xfrm interface support requires me (or someone else) to implement swanctl support in strongswan for openwrt Nov 03 11:40:33 it's bloody ugly imo, it requires you to run swanctl --load-all after starting strongswan-swanctl Nov 03 11:40:51 have an init script that does that but it's a race condition waiting to happen Nov 03 11:40:54 (this might have been answered earlier) m0f0, is right reachable over the default interface of the router? Nov 03 11:41:12 and how does ike/esp of responder/server look? Nov 03 11:41:22 wait... Nov 03 11:41:36 shouldn't ipsec create new interface visible in ifconfig? Nov 03 11:41:40 it doesn't lol Nov 03 11:41:50 no, not by default Nov 03 11:41:58 nope, that's what VTI is for, but that's ... ugly Nov 03 11:42:08 it adds to the complexity Nov 03 11:42:20 lol Nov 03 11:42:20 you need to use vti/xfrm (as stintel mentioned). Instead, policies are installed into the kernel that highjacks the traffic Nov 03 11:42:35 this is nonsense lol :D Nov 03 11:42:54 I'll be afk for a while Nov 03 11:43:01 stintel ok Nov 03 11:43:08 thanks for help :) Nov 03 11:44:03 at least when I have to work with it, the ike/esp combinations required for hardware encryption are not part of the default proposal Nov 03 11:44:07 at least for aes-ni Nov 03 11:46:44 haha, headline in Norway right now is that someone has crashed into two road barriers with their Tesla Nov 03 11:46:47 using autopilot Nov 03 11:47:02 nothing beats human stupidity I guess (even though that system is pretty bad) Nov 03 11:49:35 https://audiobacon.net/2019/11/02/the-jcat-signature-lan-a-1000-ethernet-cable/amp/ <- I think I should start a new business Nov 03 11:49:38 * f00b4r0 hides Nov 03 11:56:50 trololololololo Nov 03 11:57:04 not the first absurdly expensive ethernet cable :D Nov 03 11:57:12 Denon did it first if memory serves me well Nov 03 12:40:11 russell--: That file contains two part: one is switch initialization which should be part of mt7530 dsa driver, the other is gmac mux which I believe is already available in upstream driver? Nov 03 12:47:07 gch981213: yeah, in poking around more, it appears there is a dtsi in staging "which provides support for the integrated switch through DSA" Nov 03 12:49:09 i've been trying to understand why i see 2-3 second link drops/recoveries on switch ports on mt7621 Nov 03 12:50:28 russell--: And that's a broken dtsi :) Switch PHY mode is set to trgmii and gmac mode is rgmii. Nov 03 12:51:04 It took me a while to figure out why my ethernet doesn't work at all. Nov 03 12:55:34 [12647.280749] mtk_soc_eth 1e100000.ethernet eth0: port 2 link down Nov 03 12:55:34 [12648.893666] mtk_soc_eth 1e100000.ethernet eth0: port 2 link up Nov 03 12:58:50 it looks like that's coming from the gsw interrupt (gsw_interrupt_mt7621) and a port link change Nov 03 14:45:26 Is there any reason why we check for console in cmdline instead of using /dev/console directly? Current procd implementation doesn't support device tree stdout-path for specifying console. Nov 03 15:33:01 gch981213: maybe because procd also creates the devices? Nov 03 16:24:03 jow: is luci still possible to build on x86 for development? there is no Makefile, just a luci.mk, the old 'make runhttpd' no longer works, what's the proper workflow nowadays? I'm using sshfs for now Nov 03 16:29:07 Bootlin's Best Techniques For A Smaller Kernel + Faster Boot Times https://www.phoronix.com/scan.php?page=news_item&px=Bootlin-Lyon-Linux-Boot-Time Nov 03 16:35:56 very impressive, bootlin is great by the way, except its android slides has not been updated for years, which I don't care that much anymore :) Nov 03 16:38:01 * rr123 just spent one year working on android internals, those proprietary HAL code is not fun to deal with at all, they make the system way more complicated to protect closed source Nov 03 16:38:35 android HAL + vendor's HAL ==> hell Nov 03 16:38:57 I don't rite code my self but I have hird lots of people say that android code is a pane in the ass Nov 03 17:15:34 rr123: I usually use sshfs as well Nov 03 17:15:54 rr123: for rapid prototyping I usually use an x86 based qemu instance and connect to it via sshfs for development Nov 03 17:16:41 there are too many openwrt userland dependencies nowadays (ubus etc.) which makes running luci directly on another distro too complicated Nov 03 17:21:28 thanks. that's exactly what i'm doing, a x86 in virtualbox + sshfs :) Nov 03 17:26:56 Tapper: do you still experience these XHR timeouts? I was able to track down e9hack's issue to an rpcd crash which sohuld be fixed now Nov 03 17:27:49 jow hi just running a build now I will let you know after flashing Nov 03 17:27:59 thanks for your work Nov 03 17:31:44 jow I just flashed r11396 and am still getting it. Nov 03 17:32:14 I updated ./scripts/feeds Nov 03 17:33:04 Tapper: hm, can you login? Nov 03 17:33:23 yeah I can log in and all the other pages work Nov 03 17:33:37 so its only the start status page? Nov 03 17:34:17 O know they don't Nov 03 17:34:28 I spoke to soon Nov 03 17:35:02 cgi-bin/luci/admin/network/wireless does not load Nov 03 17:35:41 hmm, could you arrange remote access for me? Nov 03 17:36:00 an SSH port forward would suffice Nov 03 17:41:09 jow: howdy. Speaking of ssh, do you still need those DNAT on the slashdirt VMs? I'm always happy to close holes in my firewall if you don't ;) Nov 03 17:45:46 f00b4r0: I think there's an autossh on there now but please let me reconfirm on tuesday Nov 03 17:46:18 jow: sure, no worries, no problem if you still need it Nov 03 17:47:07 jow: I have yet to go on site and try and upgrade the VMs to a more powerful hardware too. Hopefully this will happen before year end Nov 03 18:54:43 jow: have you looked into refreshing the package tarballs on sources.openwrt.org ? Some of the tarballs have wrong hashes. Nov 03 19:10:02 mangix: which ones? Nov 03 20:26:18 updated openwrt/upstream, https://sdwalker.github.io/uscan/index.html Nov 03 21:45:47 Hauke: the generated tarballs. For example, the hash for ecdsautils is totally wrong. Nov 03 21:46:57 mangix: for me it worked with the openwrt components which wrere just recently generated Nov 03 21:48:49 right, I think it's an issue with older tarballs Nov 03 21:49:04 since there have been changes to the tarball generation. Nov 03 21:49:29 Hauke: I'll push my luci stuff in a minute, afterwards you just need to merge the ustream-ssl fix and we're good to go for rc0 I'd say Nov 03 21:49:51 Hauke: I'll be on the road the entire day tomorrow, but maybe we can get things started on tuesday Nov 03 21:50:13 with the aim to have the bianries done on wednesday/thursday Nov 03 21:52:39 faillogs-19.07 show nothing surprising. Nov 03 22:04:26 huh, interesting Nov 03 22:04:40 pfring is not failing on the buildbots, when it should be Nov 03 23:26:17 jow: you are talking about: ustream-ssl: skip writing pending data if .eof is true after connect ? Nov 03 23:26:48 jow: you can also merge it if you want Nov 04 00:39:01 build #146 of sunxi/cortexa7 is complete: Success [build successful] Build details are at http://buildbot.openwrt.org/master/images/builders/sunxi%2Fcortexa7/builds/146 **** ENDING LOGGING AT Mon Nov 04 02:59:58 2019