**** BEGIN LOGGING AT Wed Oct 03 03:00:01 2018 Oct 03 10:01:27 Package keepalived is missing dependencies for the following libraries: Oct 03 10:01:28 libnl-genl-3.so.200 Oct 03 10:01:41 another one that I cannot explain Oct 03 10:01:57 DEPENDS has +PACKAGE_libnl-genl:libnl-genl \ Oct 03 10:07:27 stintel: are you able to reproduce this problem locally? Oct 03 10:08:48 Hauke: this is locally, already did make package/keepalived/clean package/liblnl/clean and persists Oct 03 10:09:48 ok Oct 03 10:10:17 stintel: is libnl-genl-3.so.200 getting packaged or only some other *.so file? Oct 03 10:10:45 $(CP) $(PKG_INSTALL_DIR)/usr/lib/libnl-genl-3.so* $(1)/usr/lib/ Oct 03 10:11:05 wait Oct 03 10:11:06 ok Oct 03 10:11:24 # CONFIG_PACKAGE_libnl-genl is not set Oct 03 10:11:28 that explains it of course Oct 03 10:11:55 so it is because InstallDev installs all so's Oct 03 10:12:10 yes that explains the problem Oct 03 10:12:11 and the build of keepalived picks it up Oct 03 10:12:17 how do we solve this :/ Oct 03 10:12:33 can we have IntallDev per package? Oct 03 10:13:35 I do not think so Oct 03 10:13:54 can we force keepalived to use or not use libnl-genl? Oct 03 10:15:08 let me check Oct 03 10:15:33 --disable-libnl compile without libnl Oct 03 10:16:20 not sure if that's a good idea though Oct 03 10:18:12 https://github.com/acassen/keepalived/commit/53cbcee795c9c0cc316c19deb96a33b58bd208f0 Oct 03 10:18:56 let's try :) Oct 03 10:19:40 another possibility could be +PACKAGE_libnl:libnl-genl ? Oct 03 10:20:03 but force disable or enable libnl is probably cleaner Oct 03 10:20:34 I would prefer if we force enable or disable it Oct 03 10:20:48 I do not like this +PACKAGE_foo:foo Oct 03 10:21:43 ok Oct 03 10:21:49 I will test if it works with libnl disabled Oct 03 10:59:17 heh now I'm having the same problem with libpfring Oct 03 11:12:48 I'm trying to add support for a sercomm device (mt7621s) and i'm having some troubles at bootup Oct 03 11:12:52 SQUASHFS error: xz decompression failed, data probably corrupt Oct 03 11:13:30 squashfs fails for some reason and it's not the NAND flash, i tried on a fresh device and it works Oct 03 11:13:35 what could it be? Oct 03 11:14:10 i heard squashfs on NAND is a bad idea because of the inner workings of the fs, however i see no other options for now Oct 03 11:15:02 the system is not loaded at all btw, it just hangs after a couple of squashfs errors Oct 03 11:20:12 it is not xz compressed? Oct 03 11:20:15 but something else? Oct 03 11:21:58 i've only added a DTS file and a target profile without messing with the image generation stuff Oct 03 12:29:29 crywrt: squashfs raw on nand is a recipe for desaster - nand flash can have "gaps" (defect blocks), produce bit errors on read (hence ecc to correct them), and require moving of data to ensure it stays "fresh" - which isn't possible with squashfs, as it requires a linear, gapless space Oct 03 12:29:55 crywrt: the standard solution is to use ubi as a translation layer, and then squashfs on that Oct 03 12:49:57 KanjiMonster: UBIFS? Oct 03 12:50:20 luminoso: no, ubi Oct 03 12:50:42 well, technically ubifs on top of the same ubi for the rootfs_data Oct 03 12:51:06 KanjiMonster: never heard of it before. can you link me to the UBI concept/fs/layer ? Oct 03 12:51:46 just search for those terms, Oct 03 12:51:47 luminoso: http://www.linux-mtd.infradead.org/doc/ubi.html Oct 03 12:51:50 there's plenty of presentations on it. Oct 03 12:51:52 ty Oct 03 12:52:03 :-D Oct 03 12:53:53 heh, I like how they still refer to the ubi-2.6.git as the "current" tree Oct 03 13:46:27 KanjiMonster: how can i achieve this in OpenWrt? i believe i have to hack the image generation in the makefile associated with my target? Oct 03 13:46:44 yikes Oct 03 13:47:11 ldir: cake is seriously slow in my current image on APU2 (2cc821e7edeafecc57ed9f3be46af4c322d58560) Oct 03 13:47:33 not even getting 100Mbps throughput on a 200Mbps uplink Oct 03 13:48:01 https://pastebin.com/HsynTAbe Oct 03 13:51:15 stintel: I noticed the same on my Armada 385, seems to have seriously degraded performance. The CAKE (cerowrt) ppl know about it anyway. Oct 03 13:52:48 maybe we need to revert something for now Oct 03 13:53:46 stintel: https://github.com/tohojo/sqm-scripts/issues/71 Oct 03 13:53:52 https://www.mail-archive.com/cerowrt-devel@lists.bufferbloat.net/msg05736.html Oct 03 13:55:04 hm, that wasn't cake, that was fq_codel Oct 03 13:57:18 wonder if the cake talk was on a different mailing list Oct 03 13:57:33 crywrt: look at other nand based devices; e.g. flash partitions named "ubi" will be auto mounted as ubi (which is part of the "magic"), some details depend on wether uboot can handle ubi itself (and load the kernel from it, or depends on it residing raw on the flash) etc Oct 03 14:00:35 SwedeMike: well, here they speak about 17.01 vs 18.06, I am running master and the problem appeared between ~65d ago and now Oct 03 14:00:48 sucks, I was unable to build an image due to libftdi1 being broken Oct 03 14:11:13 I've noticed ALSA associated with a few router SDKs, why would a router need a sound library? Oct 03 14:12:30 Neighbor11111117: if the device is doubling as a media player of some kind? Oct 03 14:13:04 Neighbor11111117: connect USB sound card and use it to play mp3:s on your stereo for instance Oct 03 14:13:44 people do that? Wow, really cool Oct 03 14:23:25 stintel: ok, that's curious. The last 3 commits are effectively a no-op, especially with kernel 4.14.73 having the gso_split bug fixed. Oct 03 14:24:24 ldir: well, I have been running for 65 with the same image Oct 03 14:24:29 so it might have been an earlier commit Oct 03 14:24:54 At the moment I'm on holiday at the end of a 1M/512Kbit adsl link on a router that I can't run cake on to make it all fair. It's agony. Oct 03 14:25:16 damn Oct 03 14:25:37 and here I am complaining that due to SQM/cake I get only ~75Mbps of 200 Oct 03 14:25:53 ahh, so you're running a bugged kernel and probably without the cake built in workaround for the kernel bug. Oct 03 14:26:50 I am running current git HEAD Oct 03 14:27:08 previous image was fine (maxed out 200Mbps with that sqm config I pastebin'd) Oct 03 14:27:48 4.14.73 and all your latest commits related to cae Oct 03 14:28:20 I'd try to bisect but currently working for day job Oct 03 14:30:08 oh I misunderstood. thought you had 65days uptime. That's a bit ugly then. https://forum.openwrt.org/t/low-speeds-on-ipv6-with-sqm-cake/22403/8 has a recent discussion where the latest commits fixed a performance issue. I'd be tempted to try ethtool -k eth2 to disable gso/gro Oct 03 14:31:15 What's cpu usage like? Oct 03 14:31:20 100% Oct 03 14:31:29 iirc Oct 03 14:31:42 maybe time for some perf/flamegraph runs? Oct 03 14:32:38 ah, so you're running out of cpu?! Oct 03 14:32:50 ldir: apparently not 100%, that was with suricata enabled Oct 03 14:33:11 but roughly 100% on 3 of 4 cores Oct 03 14:33:19 without suricata Oct 03 14:36:09 as a rough benchmark for me, 20mbit egress consumes 25% cpu on the archer c7 - the ingress side is a bit lighter so I can *just* squeeze 80/20 out of it. You should be much faster than that I would have thought. Oct 03 14:45:32 ha! the local router/modem just rebooted. Oct 03 14:48:18 something is definitely wrong, start a 2nd speed test and the first one slows to a crawl Oct 03 14:51:14 do you actually have cake instances running? tc -s qdisc show dev eth2 & for ifb4eth2 ? Oct 03 14:52:06 would be useful to pastebin the output Oct 03 14:52:27 nbd: I don't think this is helpful: https://www.linux-ipv6.be/OpenWrt/perf.svg Oct 03 14:53:38 that looks rather ... incomplete Oct 03 14:53:41 how did you do that measurement? Oct 03 14:53:45 ldir: https://gist.github.com/stintel/c43044d340d20a3e6b49dc4140b049c8 Oct 03 14:54:02 nbd: perf record -F 99 -g -- sleep 60 Oct 03 14:55:42 try adding -a Oct 03 15:00:40 nbd: https://www.linux-ipv6.be/OpenWrt/perf.svg Oct 03 15:03:22 well nothing screams at me, and I can tell you're running a recent version of cake & tc because of the 'nonat nowash no-ack-filter' etc display (a recent change to tc) I've a personal dislike of 'triple-isolate' and would prefer to see 'dual-srchost' on the egress interface (eth2) and 'dual-dsthost' on the ingress. There's a small cpu saving to be made there. the thing I really don't understand from your output is 'qdisc cake 8019: root Oct 03 15:03:22 refcnt 9' specifically the refcnt value of 9. Oct 03 15:04:59 ldir: this used to work fine at max speed so I prefer to keep my settings ;) Oct 03 15:07:11 well the router here has just crashed again, so I'm giving up too. Oct 03 15:08:31 :( Oct 03 15:08:40 fix it with a hammer ;) Oct 03 15:16:56 wow, that svg is not fun for my browser :) Oct 03 15:22:45 karlp: ;) Oct 03 15:47:26 hey all, can anyone recommend any good books on IPv6? It's time to bite the bullet Oct 03 15:47:32 or online resources Oct 03 15:54:01 In a brief moment when the local internet was working.... I did a couple of speedtests on the home router/link & also did a couple simultaneously...50% bandwidth sharing and an A+ rating on bufferbloat. cake appears to be working correctly for me. Oct 03 15:55:53 nice, my router crashed too Oct 03 15:59:33 cake works just fine for me as well Oct 03 16:00:09 well, better actually since 6rd packets are treated properly now Oct 03 16:00:54 Cake is working fine for me too, but this is not OpenWRT but 4.19-rc6 on Ubuntu. Oct 03 16:03:14 mamarley: ooooh in tree cakey goodness! I can't wait for openwrt to bump to whatever LTS kernel 4.19> and we can get rid of the out of tree module :-) Oct 03 16:03:44 19:03:30 up 0 min, load average: 1.41, 0.32, 0.10 Oct 03 16:03:46 goddamnit Oct 03 16:04:19 Your router again crashed? Oct 03 16:05:19 ldir: Yep, and I'm also looking forward to iproute2 4.19 coming out so I can package that and use only in-tree stuff. Oct 03 16:09:31 anyknow what PKGARCH is really for? Oct 03 16:09:44 it's listed as "(optional) Set this to “all” to produce a package with “Architecture: all”" in the wiki, but that doesn't really help. Oct 03 16:10:10 there's a tiny handful of packages that explicitly set it to "all" and none that do anything else. Oct 03 16:10:56 I'd like to clear up some guidance for package atuhors so this doesnt' just get copypasta'd for no reason Oct 03 16:13:02 * ldir suspect karlp has seen his cooking Oct 03 16:13:37 ldir: although, when I think about it the upload latency has been quite high ever since I got my 500/500 connection. worth noting is that I limit cake to 100/100 mbps in order to not have the cpu as a bottleneck Oct 03 16:13:42 first I saw was in the safe search pr I think, Oct 03 16:13:51 ldir: is this normal: http://www.dslreports.com/speedtest/39825676 Oct 03 16:16:26 huaracheguarache: no I would not regard that as normal. is wifi involved anywhere? Oct 03 16:16:56 yes, but I'm fairly certain that the same thing happens over wire Oct 03 16:17:19 hold on, let me check just to make sure Oct 03 16:18:09 the latency graph suggests a buffer is slowly filling up in the egress direction. Oct 03 16:19:42 So, does the same thing happen with sqm scripts if using fq_codel? Also what happens if the egress link bandwidth is reduced even further? Oct 03 16:20:38 brb, I'm going to run the wired test on my laptop Oct 03 16:25:44 karlp: I suspect it gets overwritten by include/package-defaults.mk Oct 03 16:27:31 ldir: so with the same 100/100 limit and cake the results are the same over wire: http://www.dslreports.com/speedtest/39826381 Oct 03 16:30:18 ok, so what happens if you drop the egress even further to say 50 ? I've found egress cpu usage to be a bit higher so want to prove not cpu issue. Oct 03 16:32:43 ldir: it may very well be a cpu issue: http://www.dslreports.com/speedtest/39826783 Oct 03 16:32:54 ldir: I set egress to 50 mpbs Oct 03 16:33:41 the isolated latency spikes are probably due to some background browsing traffic from other people on the network Oct 03 16:34:05 looks suspiciously like it Oct 03 16:34:50 the r7800 is a bit disappointing =( Oct 03 16:35:49 I only know the Archer C7 v2 and that is really at the limit with 80/20. Or basically 100mbps total up/down. Oct 03 16:36:36 I think I have set cake to 100/100 on my brother's archer c7 Oct 03 16:37:35 for me personally, low latency beats absolute bandwidth. Oct 03 16:38:09 * mamarley throws an x86_64 at the problem. Why not both? Oct 03 16:39:54 mamarley: yeah, sure, if you've the cpu then no problem. But if you don't, for me, latency control is better than jiggabits per second mega whizzy super ISP salesman bandwidth numbers. Oct 03 16:40:35 probably going to get myself an x86 box when I'm going to replace the r7800 Oct 03 16:48:06 For 500/500 with cake, x86_64 probably is your best bet. Oct 03 16:48:53 Based on the numbers I have seen with mvebu devices, I figure even those would probably choke in that scenario. Oct 03 16:50:00 dum de dum Oct 03 16:50:57 I really should update to something a bit more gutsy & modern. But I like the fact it sits in a cupboard with wifi etc all in a single box. I have no idea whatsoever on what/how to replace it. Oct 03 16:50:58 I didn't expect to be able to use the full 500/500 capacity with the r7800, but what surprised me was that the r7800 didn't really seem to outperform the archer c7 by much Oct 03 16:51:16 mmm, so 4.9 support has been removed on a bunch of targets, are they 4.14 only? Oct 03 18:38:02 hmm, [ 404.512572] genirq: Flags mismatch irq 13. 00014c80 (mips_perf_pmu) vs. 00000080 (ehci_hcd:usb1) Oct 03 18:38:02 [ 404.521558] Unable to request IRQ13 for MIPS performance counters! Oct 03 18:38:14 hi any idea on how to modify a net phy register just before rebooting the device? Oct 03 18:40:15 I can use phy_register_fixup_for_uid, but it seems it needs to restart the net driver or something similar to get it applied Oct 03 18:48:13 and no idea how to do this inside a kernel module Oct 03 18:48:16 :P Oct 03 20:06:42 I'm thinking of writing a tool to parse a pcap to try to figure out whether transmitter is failing to transmit, or receiver fails to receive...as well as look for block-ack issues and so forth. Anyone know of a tool that already does this sort of thing? Oct 03 20:08:34 you mean parsing pcap? or processing monitor frames? Oct 03 20:10:57 parsing a pcap with lots and lots of frames in it: for instance, we do a 20 station UDP upload test, and performance sucks on some APs...why does it suck? Oct 03 20:11:42 pcap would have been gathered from a monitor device, so it has wifi frame info in it, like seq-no, block-acks, etc Oct 03 20:11:51 ah I see Oct 03 20:12:18 if your want to run some large scale analysis you probably want to do it in python or something Oct 03 20:12:27 there should be readymade pcap bindings Oct 03 20:12:52 pcapy Oct 03 20:13:04 I was thinking pcap -> tshark-decode-to-big-text-file -> perl processor script Oct 03 20:13:05 or was it scapy? Oct 03 20:13:39 the pcap format is quite simple, you should be able to process it with perl directly, possibly using some unpack magic Oct 03 20:14:23 https://git.openwrt.org/?p=openwrt/openwrt.git;a=blob;f=package/network/utils/iwcap/src/iwcap.c;h=bc69550ef38dc675bdfc5e8a7790223c8c42e327;hb=2cc821e7edeafecc57ed9f3be46af4c322d58560#l79 Oct 03 20:14:25 either way, that is a detail..the hard and/or interesting part will be to track seq-no vs blockacks, mcs vs retries, and so forth Oct 03 20:14:29 its just a series of u32 Oct 03 20:15:17 I see, so you anticipate the ieee80211 dissecting to be the hard part? Oct 03 20:16:09 huaracheguarache: that just means you'd be in the perfect position to work on supporting the NSS cores ;) Oct 03 20:16:37 tshark will dissect the frame, but knowing if frame is out-of sequence or not acked by a block-ack shortly afterwards is the tricky part Oct 03 20:16:38 greearb_: you should look into scapy, it should support all bits you need Oct 03 20:17:56 huaracheguarache: QCA seems to heavily bet on this kind of hardware acceleration, instead of optimizing the software path Oct 03 20:18:26 I somehow feel uneasy about opaque, specialized hardware ICs altering network packets in flight Oct 03 20:18:44 best driven by some proprietary firmware Oct 03 20:19:10 yep, it's pretty nasty Oct 03 20:19:40 scapy looks interesting, but probably not geared towards what I think I want Oct 03 20:34:27 jow: is it possible to enable per-device manifest files with TARGET_MULTI_PROFILE/ TARGET_PER_DEVICE_ROOTFS? Oct 03 21:01:31 pkgadd: I would be if I had extensive programming knowledge and experience Oct 03 21:02:44 it's interesting to see that ipq40xx apparently works without NSS cores, while ipq8074 has bumped them to 1.5 GHz Oct 03 21:05:27 ok, master is extremely unstable for me Oct 03 21:05:29 https://pastebin.com/vT5wTejR Oct 03 21:07:52 jow: so, like all networking ICs? :) Oct 03 21:08:54 blogic: I'm suspecting this might be your RPS patch ^ Oct 03 21:09:10 karlp: the NSS/ NPU cores are a bit worse in that regard, as they really take over most of the connection handling with a pretty overengineered firmware Oct 03 21:09:27 going to revert it and build a new image **** ENDING LOGGING AT Thu Oct 04 03:00:02 2018