**** BEGIN LOGGING AT Thu Jun 27 02:59:57 2019 Jun 27 03:46:34 jow: I see you cherry-picked them as well. Thank you so much for it! Jun 27 04:15:15 DonkeyHotei: actually i already did that, lol Jun 27 04:16:57 https://github.com/angelsl/openwrt/blob/ea8100/target/linux/ramips/patches-4.14/0101-save-rootfs-from-bootloader-hack.patch Jun 27 04:18:10 ugly ugly ugly Jun 27 04:18:36 i'll refactor it a bit, that was 4am code Jun 27 04:22:15 i was thinking it would be more like target/linux/kirkwood/patches-4.14/202-linksys-find-active-root.patch Jun 27 06:00:27 Hello! I've got a problem with an AR9331 board - 8Devices Carambola2 SoM. If I stop the watchdog or run 'tail /dev/zero', when the watchdog resets the board, it sometimes hangs and only a power cycle will start it up again. I've found some other reports: Openwrt issue #12121 (https://dev.archive.openwrt.org/ticket/12121.html) and Openwrt Github pu Jun 27 06:00:27 ll request 747 (https://github.com/openwrt/openwrt/pull/747) but none provide a solution. The guy in the Github 747 issue has tried "fixing" it with a patch, which does indeed move my "hanged after N watchdog resets" from N of around 3-5 to about 90, but this still worries me. Can anyone please give me some pointers on how to solve/better diagnose Jun 27 06:00:28 this? Jun 27 09:56:47 nbd: Would you agree with this one? I've tested it thoroughly and it indeed improves uqmi a lot. Currently in testform patch but I would add it to uqmi tree https://git.openwrt.org/?p=openwrt/staging/xback.git;a=commit;h=09e328dabfd221a5a5176b4685a80908d3192dad Jun 27 10:02:14 xback: looks good **** BEGIN LOGGING AT Thu Jun 27 11:54:12 2019 Jun 27 12:02:50 ynezz: you can add my tested-by on macos to "build: image: make image padding OS agnostic" if you like Jun 27 12:20:35 kernel bumps pushed containing the TCP performance fix Jun 27 12:31:28 * nmrh sighes with relief that there is no 4.19 bump... Jun 27 12:32:05 lol - yes it's been a bit relentless of late Jun 27 12:42:33 * stintel blames intel Jun 27 12:45:43 * ldir blames stintel Jun 27 12:51:48 :( Jun 27 12:53:22 does the TCP SACK CVEs affect linux kernel based routing as well? Jun 27 12:53:34 No Jun 27 12:53:41 Only established connections are affected Jun 27 12:54:00 So as long as you dont export any TCP ports to the outside, you are safe Jun 27 12:54:11 AFAIK Jun 27 12:54:16 ldir Hi mate what build is your c7 on? Jun 27 12:54:25 (should always add disclaimer ;) Jun 27 12:54:35 I have a c7 and the ath10 radio will not come up Jun 27 12:54:37 kristrev_: "export TCP ports"? Jun 27 12:54:59 kristrev_: so as long as the TCP SACK packets aren't processed by the host kernel as part of socket handling, it's fine? Jun 27 13:10:33 Tapper: a recent-ish master but with the latest ath10ct bump reverted 'cos I saw some firmware crashes - koen has just pushed a new ct firmware update so currently building that. Jun 27 13:11:40 ldir ok thanks mate I will wate til that hits master. I was going to go back to 1.8.xx but I cant be arst to redo configs and all that as i would have to switch back to ar71xx Jun 27 13:11:57 *18.xx Jun 27 13:12:41 I was getting ): cat: can't open '/var/run/wifi-phy0.pid': No such file or directory Jun 27 13:12:42 Jun 27 13:12:42 Thu Jun 27 06:30:54 2019 daemon.notice netifd: radio0 (1434): WARNING (wireless_add_process): executable path /usr/sbin/wpad does not match process path (/proc/exe) Jun 27 13:13:05 ): Device setup failed: HOSTAPD_START_FAILED Jun 27 13:18:22 *yawn* Jun 27 13:18:47 * blogic manages to get ath11k flying on MCS11 Jun 27 13:18:51 using qam1024 Jun 27 13:20:23 https://patchwork.kernel.org/patch/11019277/ Jun 27 13:20:31 nugget took me several hours to track down Jun 27 13:21:27 SwedeMike: Yes, that is my understanding Jun 27 13:22:07 The error has do with how the TCP stack processes SACK-info Jun 27 13:23:58 kristrev_: ok, good, I was worried this would also affect the kind of re-assembly or handling that the firewall code would do, but I guess the firewall code doesn't aggregate TCP messages with GRO and handle them the way that is vulnerable. Jun 27 13:25:58 x4c4a: sad to hear it only made it harder for you to see, it's been 100% successful for my carambola2 based boards so far (at least from reports....) Jun 27 13:26:16 SwedeMike: From what I have read, we are safe on the routers :) Jun 27 13:26:17 ifyou're still seeing problems with my patch, then you should be also seeing hangs from just "reboot" enough times too. Jun 27 13:26:41 I just toook the approach that reboot had been "fixed" to not hang, so I'd call reboot instead of relying on the wdt to do the right thing. Jun 27 13:27:06 karlp: Maybe it has something to do with the bootloader version? Which are you using? Jun 27 13:27:54 i recall the carambola2 loader being bad and you had to use pep2ks loader Jun 27 13:28:52 blogic:as I mentioned in the ticket, that doesn't help if there's no rational way for the user to replace the bootloader, and when it can be worked around to a farrr higher percentge from the kernel. Jun 27 13:29:32 I assume you mean issue and not ticket Jun 27 13:29:52 blogic: You did mention a bootloader fault in the github issue, but I couldn't find any other reference Jun 27 13:30:04 pepe2k told me at the time Jun 27 13:30:22 but if karlp has a fix then all the better Jun 27 13:31:35 pepe claimed that it was a bootloader issue that must be fixed in the bootloader. Jun 27 13:31:38 I'd be fine with a bootloader change fix, since the deployment isn't that huge and the bug is incredibly rare in actual use, however karlp's fix, if working, would be even better. Jun 27 13:32:03 karlp: Which bootloader are you using on your boards which don't hang with your patch? Jun 27 13:32:06 I've never been against that, but I also need a way of fixing things that doesn't require a user to do bootloader replacements. Jun 27 13:32:33 my "fix" is gross, I understand that, and it applies to all ath79 devices too, because of how I implemented it. Jun 27 13:32:53 x4c4a: the 2.3 that they normally ship with, but I've also tried all the released versions of caraboot. Jun 27 13:33:34 Alright, I'll try again with 2.3 and see what happens. I tried only with 2.10-dev and pepe's versions until now. Jun 27 13:33:50 if pepes version also hangs, you've got a much better chance of getting it fixed there then :) Jun 27 13:34:13 he was't interested in it when it was in caraboot only (understandably) Jun 27 13:34:30 What's the longest test you've run regarding the watchdog resets? Jun 27 13:35:04 Tapper: both C7 v2s now running master - no obvious problem, have some stations associated on 5GHz, but I'm not actually at home to check it *really* works. Jun 27 13:35:07 doing the tail /dev/zero thing, maybe an hour or so, going from "within a 1-3 minutes" to "never" was more than enough for me. Jun 27 13:35:29 I ran for more, that's when it manifested. Jun 27 13:35:35 It did go from 1-3 to 90. Jun 27 13:35:47 N = 1 for now Jun 27 13:36:10 Currently looks like the best way to go is - your fix + hardware watchdog. Jun 27 13:36:56 Incredibly frustrating glitch. Thank you very much for replies karlp, blogic! Jun 27 13:47:18 I've not (yet) ported my hack to ath79 btw, but it's on my list of things to do. Jun 27 13:47:44 x4c4a: feel super welcome to chase pepe2k over it if you can reproduce it with their bootloader though :) Jun 27 14:23:27 Hi all, bit of a potentially stupid question. But I'm certainly not an EE, got into OpenWRT from CS side perspective. SoCs will be used by many different manufacturers. The manfacturers develop boards. And there are families of SoCs if my current understnading is ok Jun 27 14:24:28 In OpenWRT parlance target system is a family of SoCs. Subtarget is a specifc SoC, and target profile will be a board. Jun 27 14:24:31 Is that correct? Jun 27 14:24:50 that is correct Jun 27 14:24:53 well not quite Jun 27 14:25:04 Presuambly families of SoCs are similir but with minor improvements over time. Until there is a major change to the SoC to introduce a new family. If that correct? Jun 27 14:25:05 subtarget is often different generation of SoCs Jun 27 14:25:09 Thanks KanjiMonster Jun 27 14:25:44 * gch981213 made a target-wide PR and is unsure about whether he should ask people to test every subtargets for him: https://github.com/openwrt/openwrt/pull/2170 Jun 27 14:26:05 Ok makes sense KanjiMonster Jun 27 14:26:20 different bit-ness (32 vs 64), or support for newer instruction sets are often reasons for subtarget splits Jun 27 14:26:31 ahh ok gotcha Jun 27 14:26:37 This is just an upstream feature and theoretically it shouldn't be broken, right? Jun 27 14:27:32 gch981213: hah, I don't need to test that one for brcm63xx, I wrote that support myself to avoid using the hack ;D Jun 27 14:29:26 KanjiMonster: Cool. Jun 27 14:29:28 gch981213: 4 targets are already using it (ath79, brcm63xx, lantiq, octeon), so presumably it's working fine Jun 27 14:29:30 OK ldir thanks mate. I mite have to redo configs I think. Jun 27 14:29:51 So KanjiMonster, a board is basically the specfic SoC picked in subtarget and the manufacturer/OEM then picks the RAM, flash, switch, etc. Someone there has to get the memory map correct then OpenWRT needs to get kernel that is aware of flash init, flash layout, switch init and the SoC components should be the same. And finally somonone has to write a bootloader to init the RAM component before passing it onto the kernel Jun 27 14:29:54 is that the jist of it? Jun 27 14:30:59 gist* Jun 27 14:35:27 Neighbor11111111: usually the chip vendor also provides reference boards with working bootloader and linux SDKs and reference code. OEMs then adapt/modify these to match their product (sometimes more, sometimes less) Jun 27 14:36:56 Ok great thanks KanjiMonster Jun 27 14:37:10 KanjiMonster: Alright. I'll drop the RFT tag then. There doesn't seem to be any differences on dealing with dtbs across subtargets anyway. Jun 27 14:52:11 gch981213: Are going to do more mt7621 stuff? A new i2c driver is in i2c-next and NAND driver can use an update for >4.14 kernel Jun 27 14:53:38 mt7621 i2c driver https://git.kernel.org/pub/scm/linux/kernel/git/next/linux-next.git/commit/?h=next-20190627&id=d04913ec5f89f13760f8e8411c93cee542560d68 Jun 27 14:55:35 Rene__: I'm going to do a massive dts renaming the week after next: https://github.com/openwrt/openwrt/pull/2167#discussion_r298142978 Jun 27 14:56:53 As for backporting... I may try to backport GPIO driver. I don't have a plan for other drivers yet. Jun 27 14:59:50 gch981213: Ok, i2c should be easy. Jun 27 15:00:26 i was thinking it would be more like target/linux/kirkwood/patches-4.14/202-linksys-find-active-root.patch Jun 27 15:00:37 that depends on MANGLE_BOOTARGS or something being on Jun 27 15:01:23 https://github.com/openwrt/openwrt/blob/master/target/linux/generic/pending-4.14/920-mangle_bootargs.patch Jun 27 15:01:37 which.. isn't on and i decided i would rather not turn it on Jun 27 15:02:12 Rene__: Our out-of-tree driver works fine now and we could use upstream driver when ramips is bumped to a future kernel version with it. Jun 27 15:02:17 because on ramips the bootloader command line is overwritten before this patch Jun 27 15:02:24 so.. if i'm gonna have to hack things anyway.. Jun 27 15:03:43 angelsl: I dropped that prom-fixes patch in https://github.com/openwrt/openwrt/pull/2170 Jun 27 15:04:48 ah.. sigh Jun 27 15:04:59 angelsl: I'm not sure if it makes any differences though. Jun 27 15:05:08 gch981213: I know that NAND is an issue that prevents ramips to bump to a higher version, do you know other parts that have issues? Jun 27 15:06:12 gch981213: my hack piggybacks off the dtb scanning added by 0100-prom-fixes Jun 27 15:06:26 guess i'll figure something out when #2170 is merged Jun 27 15:10:39 ldir: thanks for testing Jun 27 15:10:48 I can't remember who made the backport, but when the mt7621 pci driver was backported some time ago then pci broke on a lot of devices Jun 27 15:10:52 wouldn't it make more sense to bump ramips to 4.19? Jun 27 15:11:03 Might be fixed upstream now though Jun 27 15:12:40 The email thread with the subject "ramips: Fix pcie interrupt mapping for ZBT WE1326" contains a description of the problem as well as what went/goes wrong Jun 27 15:12:46 There was also a forum thread somewhere Jun 27 15:12:53 Though, I agree with ynezz. We should bump and fix Jun 27 15:14:57 karlp: that ath79 wdt driver is upstream, so you can try it there :) Jun 27 15:15:45 got a lnk? Jun 27 15:16:10 Here is a link to the forum thread: https://forum.openwrt.org/t/mt7621-pcie-dts-issues-mt76-stopped-working-with-latest-updates/14623 Jun 27 15:17:06 kristrev_: When I try to run a linux-next kernel on my mt7621 board 2 months ago, it hung after some kernel log about PCIE without any obvious error message. Jun 27 15:17:50 Interesting. I haven't had a chance to dig more into the PCIe issue or test 4.19 (or more recent) on my boards :( Jun 27 15:18:35 The board worked fine with the PCIe patch that was later reverted, it was just PCIe that was broken Jun 27 15:19:42 kristrev_: And there isn't a backport of upstream PCIE driver into OpenWrt applied I think? Jun 27 15:20:27 gch981213: Here is the patch-series that was later reverted (or at least parts of it): https://patchwork.ozlabs.org/patch/912506/ Jun 27 15:22:48 kristrev_: Found it. I was using git log on a wrong patch. Jun 27 15:24:15 great :) Jun 27 15:25:06 ynezz: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/drivers/watchdog/ath79_wdt.c?h=v5.1.15#n104 just shows it being hardcoded to doing the chip reset still, is there something else you were thinking of? Jun 27 15:25:26 that appears to be just the same implementation that we have already? Jun 27 15:25:39 we're using that driver Jun 27 15:26:42 you want me to try and convince upstream to change it? when I can't convince us here? is that what you meant? Jun 27 15:26:50 yeah, exactly Jun 27 15:29:38 I'm much more interested in using https://elinux.org/Tests:Watchdog-Pretimeout so that I don't hve to bother trying to convince people there's a problem. Jun 27 15:29:58 but that requires patching the driver to support those governors Jun 27 15:30:09 so I'll probably just carry the patch a bit longer for the time being. Jun 27 15:30:26 or make that patch less invasive Jun 27 15:31:21 like adding some property which would apply those changes only for certain boards, where it's confirmed, that they contain broken SoC/bootloader Jun 27 15:32:17 I got no followup to https://github.com/openwrt/openwrt/pull/747#issuecomment-406581179 so I've not continued any further work. Jun 27 15:32:32 pepe seemed largely to just stick on "it's abootloader issue, must be fixed there" Jun 27 15:36:25 yeah, I agree with that conclusion as well Jun 27 15:37:31 but I think, that it's impossible to fix all bootloaders out there Jun 27 15:38:12 I can imagine, that we can convince carambola2 users to flash fixed bootloader, but I'm not sure we can do that for other devices Jun 27 15:39:03 I would simply try to come with something less invasive, and try to upstream it first Jun 27 15:41:20 BTW it can't be that hard to add few dozens of printfs into that bootloader to find out where it hangs Jun 27 15:50:12 from my point of view, it's irrelevant, I still have devices that I have to support that I cna't change the bootloader on. Jun 27 15:51:55 you'd at least agree to my last comment on the ticket, that it is in principle legitimate to workaround bootloader bugs via kernel code? Jun 27 15:53:00 feel free to update the ticket to that effect for the record if you do. Jun 27 17:53:39 whoop, i have wifi on mt7615 Jun 27 17:59:00 has anyone ever expierenced fast LAN to WAN on any wired and wireless Jun 27 17:59:11 however crippiling slow wireless LAN to wired LAN ( < 1mbit) Jun 27 17:59:27 I've never expierenced anything before like this. Does it ring a bell, any typical gotchas that manifiest like this? Jun 27 17:59:43 also CPU usage is very low < 3% while this is happening Jun 27 18:04:33 gch981213: no idea how the kernel bump will work. The upstream ethernet driver or DSA driver has no support for the other ramips SoCs Jun 27 18:05:25 Same situation with the upstream MMC driver Jun 27 19:26:02 gosh, this device sets wlan MACs from nvram too Jun 27 19:29:12 blogic: what should I install atm? 18.04.02? or head of tree? It's going onto an AP on a pole, and therefore a bit more trouble to swap out than usual... Jun 27 19:36:46 jg: can you wait for 19.07? Jun 27 19:37:24 if not, 18.06.3 should be uploaded this week Jun 27 19:45:12 how do i handle a device whose wifi mac is set via mtd_get_mac_ascii? Jun 27 19:46:45 DonkeyHotei: no, can't wait. Family and I are experiencing flaky service and I need to replace the router ASAP. Jun 27 19:47:11 jg: can you build your own image? Jun 27 19:47:34 DonkeyHotei: not set up for it at the moment, unfortunately. Jun 27 19:47:49 angelsl: take a look at target/linux/ath79/base-files/etc/hotplug.d/ieee80211/10_fix_wifi_mac Jun 27 19:49:46 jg: 18.06.3 should be uploaded any time now Jun 27 19:50:07 anytime, as in the next few hours? or few days? Jun 27 19:50:17 thanks DonkeyHotei Jun 27 19:50:35 i actually saw that earlier but.. for some reason i thought it wasn't the right thing Jun 27 19:51:13 jg: i'd guesstimate a day or two. if you really really need it sooner, 18.06.2 has been recently reuploaded Jun 27 19:51:55 angelsl: it would be "more correct" to do it via the dts file, but i'm not sure there's a binding Jun 27 19:53:25 DonkeyHotei: ok. I'll see how it goes. Problem is, I leave town for a week on Sunday, and my daughter will be needing service while I'm gone. So I can't procrastinate for very long. Thanks in any case for the advice! Jun 27 19:53:55 DonkeyHotei: heh Jun 27 19:54:26 angelsl: how do other linksys devices do it? Jun 27 20:02:59 DonkeyHotei: i can't find another linksys device that does this Jun 27 20:03:27 not in kirkwood and not in mvebu? Jun 27 20:05:39 nope.. maybe i'm not looking hard enough Jun 27 20:05:51 although, there are other devices that are doing what i'm doing Jun 27 20:05:55 just not linksys ones Jun 27 20:06:19 do they use dts? Jun 27 20:11:18 hmm.. Jun 27 20:13:49 odd.. i'm looking at the eeprom dump and there *is* a MAC specified here Jun 27 20:13:59 not sure why it comes up as zeroes Jun 27 20:14:51 but the factory firmware just overwrites whatever mac with hw_mac_address + 1/+ 2 for phy1 Jun 27 20:48:29 ldir: just noticed your dropped patches try-out. Could you let me know if this changes anything? Jun 27 22:26:37 Got the Reviewed-by, now just one more Linux-MTD sign-off for ath79 SPI-NAND Jun 27 23:18:54 jeffsf: regarding classifying https://bugs.openwrt.org/index.php?do=details&task_id=1746 as a duplicate, the VGV7510KW22 uses plain spi-nor - I think that might rather fail/ fall over dot files being forgotten by file removal from a mounted overlay (or something different); at least there's no ubi and plain jffs2 Jun 27 23:19:36 pkgadd: thanks, I'll look at that more carefully -- yes sh glob is a different problem Jun 27 23:20:15 I have two VGV7510KW22, if you're looking for more info (but no problems ;) Jun 27 23:21:10 trying hard not to respond with https://www.youtube.com/watch?v=IA2EBWFCULg ;) Jun 27 23:21:44 I've got enough "problems" with strangeness in stage2 sysupgrade with sh functions not returning to their caller -- had to give up on some helpful flashing for ath79 SPI-NAND, at least for the incoming PR Jun 27 23:22:41 LOL, yeah Jun 27 23:22:51 Spot on Jun 27 23:33:26 3 devices for 5 OpenWrt firmwares and OEM for each, too damn many sysupgrade combinations to check! Jun 27 23:34:03 Had forgotten how slow it is to re-iniitalize a JFFS2 overlay on NOR Jun 28 00:05:37 that's a luxury problem, it just means that your spi-nor flash is too big ;) Jun 28 00:30:35 outside of ctags, anyone know of a c function callback tracing tool for emacs? Jun 28 00:30:58 or perhaps a linux kernel mode? Jun 28 00:31:27 s/know/recomend/ Jun 28 00:43:22 i've never tried cscope but perhaps i'll give it a go Jun 28 01:50:48 mangix: for eth driver, rename our local driver to something else and change compatible string for mt7621 a bit? Jun 28 01:52:29 mangix: As for MMC driver, we may have to use our local driver for a while until a proper solution for mt7628 pinctrl is found, or patch the hacky pinctrl settings into upstream driver. **** ENDING LOGGING AT Fri Jun 28 02:59:57 2019