**** BEGIN LOGGING AT Tue Oct 15 02:59:58 2013 Oct 15 06:24:46 build #366 of avr32 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/avr32/builds/366 Oct 15 07:54:39 build #356 of xburst is complete: Failure [failed compile_8] Build details are at http://buildbot.openwrt.org:8010/builders/xburst/builds/356 Oct 15 11:21:26 nbd r38409 trunk/target/linux/mpc85xx/patches-3.10/200-gianfar_napi_poll_revert.patch Oct 15 11:21:26 mpc85xx: revert the napi polling code to the one from an older kernel version to fix stability issues (#14020) Oct 15 11:33:47 luka r38410 trunk/package/system/uci/Makefile * [package] uci: upgrade to latest git version Oct 15 12:24:40 jow r38411 trunk/package/network/ services/openvpn-easy-rsa/Makefile services/openvpn-easy-rsa/patches/100-run-ootb.patch services/openvpn-easy-rsa/patches Oct 15 12:24:41 openvpn-easy-rsa: restore essential patch to make it actually work on the target (#14324) Oct 15 12:44:17 luka r38412 trunk/package/network/ services/openvpn/files/openvpn.init services/openvpn/Makefile services/openvpn/files/openvpn.config * [package] openvpn: make comp_lzo a parameter Oct 15 13:11:52 jow r38413 trunk/package/network/services/dropbear/Makefile * dropbear: add dropbear.nl mirror, provided by dropbear maintainer Oct 15 13:29:28 jow r38414 packages/libs/libnfc/Makefile * libnfc: Update to version 1.7.0 Oct 15 13:38:52 jow r38415 packages/libs/libgphoto2/Makefile * libgphoto2 depends on libusb-compat, not libusb-1.0 Oct 15 13:38:53 jow r38416 packages/net/ ddns-scripts/Makefile ddns-scripts/files/etc/init.d ddns-scripts/files/etc/init.d/ddns * ddns-scripts: add init script Oct 15 13:47:17 jow r38417 packages/net/pptpd/files/options.pptpd * pptpd: remove IP in options.pptp Oct 15 14:58:09 blogic r38418 trunk/target/linux/ramips/dts/MT7620a_MT7610e.dts * ralink: fix default vlan mapping for MT7620a_MT7610e Oct 15 14:59:37 build #412 of orion is complete: Failure [failed compile_8] Build details are at http://buildbot.openwrt.org:8010/builders/orion/builds/412 Oct 15 15:30:03 nbd ping Oct 15 15:30:08 groz: pong Oct 15 15:30:26 I've just been reading the list, regarding the dma stuff etc on ar7xx Oct 15 15:30:38 what dma stuff exactly? Oct 15 15:30:39 I've found a very very interesting tidbit here this weekend Oct 15 15:31:05 folks having thruput issues, and seeing dma error messages in the mailing list Oct 15 15:31:13 you mean ath9k? Oct 15 15:31:13 catching up here from being gone all weekend Oct 15 15:31:20 yes, ath9k in particular Oct 15 15:31:31 and I have an observation on my own stuff, that may be very useful Oct 15 15:31:42 which I just connected the dots on over the last few days Oct 15 15:32:07 I have a setup with my all-sky camera in the observatory, which uses a wds link to get data back to the house, until we get ethernet in the ground Oct 15 15:32:25 then another ap pair one in the house, one as wds repeater outside to cover the back yard Oct 15 15:32:47 all sky camera generates a 3 meg data file every 15 seconds, and dumps it back to the house over the wds link Oct 15 15:32:56 on the house end, it's a unifi access point Oct 15 15:33:04 on the observatory end, it's wrt160nl Oct 15 15:33:17 the other pair is unfi inside, and picostation outside Oct 15 15:33:34 then another wrt160 on my desk, which I use for test etc Oct 15 15:33:52 the one on the desk, seems to have dma errors triggered, when all sky is transferring files Oct 15 15:34:03 it's physically separated from the unifi by about 3 feet Oct 15 15:34:12 but, unplug the unifi, and, those errors vanish Oct 15 15:34:30 seems to be correlated to phyically close spacing between 'busy' access points Oct 15 15:34:34 which are on different channels Oct 15 15:34:40 1 and 11 in this case Oct 15 15:34:48 interesting Oct 15 15:34:54 what version of openwrt are you running on the wrt160nl? Oct 15 15:35:07 it would be useful to re-test with current trunk Oct 15 15:35:15 the 160 had trunk from about 2 months ago, and, today I'm going to check out the changes mentioned in that list Oct 15 15:35:15 because of all those fixes that i made in the mean time Oct 15 15:35:27 but here is the other interesting test I tried Oct 15 15:35:42 so, I placed a 160 by itself, somewhat spaced from the rest Oct 15 15:35:49 and let it run for an hour, no errors Oct 15 15:35:52 channel 6 Oct 15 15:36:06 then I put a wrt54g beside it, about 2 foot spacing Oct 15 15:36:12 on channel 11 Oct 15 15:36:25 instantly, dma errors showing up on the wrt160nl Oct 15 15:36:53 and yes, I will repeat all these tests, after I get latest trunk built Oct 15 15:37:10 the all-sky camera runs from sundown, tuill sunup, and really stresses that link Oct 15 15:37:21 3 meg file transfer every 15 seconds, non stop Oct 15 15:37:59 this is an example of the end result Oct 15 15:38:08 http://youtu.be/X6NAcNfrs1U Oct 15 15:38:08 some of those issues are definitely triggered by wifi activity, busy channel, etc. Oct 15 15:38:15 each frame is from a 3 meg data file Oct 15 15:38:35 i have to leave now, let me know when you have test results with trunk Oct 15 15:38:38 thanks Oct 15 15:38:39 but, the other interesting bit, the link sending them Oct 15 15:38:43 has no issues Oct 15 15:38:48 it's the others that have problems Oct 15 15:39:01 I will let you know, I'm still workingo n the 'wake up coffee' here this morning Oct 15 15:40:32 interesting :) Oct 15 15:41:06 that AP where I have the DMA issues is at work, where there are a whole lot of other APs Oct 15 15:41:21 we just moved here a couple months ago, so I haven't had a chance to run an ethernet underground yet Oct 15 15:41:50 thinking, maybe leave this on wifi, it'll force me to get looking into this problem, if I can begin to figure my way around inside the code Oct 15 15:42:09 what chipset stintel ? Oct 15 15:42:49 ehr, not sure, it's a buffalo something, let me check irc logs Oct 15 15:42:56 don't have access to the device right now Oct 15 15:43:32 Buffalo WZR-HP-G300NH Oct 15 15:51:38 build #379 of rb532 is complete: Failure [failed shell_12] Build details are at http://buildbot.openwrt.org:8010/builders/rb532/builds/379 Oct 15 15:53:33 build #379 of ppc44x is complete: Failure [failed shell_12] Build details are at http://buildbot.openwrt.org:8010/builders/ppc44x/builds/379 Oct 15 15:54:39 btw. if you run further tests, and you want to use AA instead of trunk, please use this backport of mac80211: http://nbd.name/gitweb.cgi?p=aa-mac80211.git;a=summary Oct 15 15:58:22 Gonna do them both Oct 15 15:58:53 just setting up to do a clean trunk build, from scratch Oct 15 15:58:59 will do the same with aa, and compare them Oct 15 15:59:27 * nbd heads out to play some badminton Oct 15 18:04:19 build #310 of iop32x is complete: Failure [failed shell_12] Build details are at http://buildbot.openwrt.org:8010/builders/iop32x/builds/310 Oct 15 18:06:11 build #60 of imx23 is complete: Failure [failed shell_12] Build details are at http://buildbot.openwrt.org:8010/builders/imx23/builds/60 Oct 15 21:06:35 why is there not a generic image for the ramips chip Oct 15 21:08:46 froek: because ramips chips do not provide means of autodetecting the board configuration Oct 15 21:10:55 KanjiMonster: ahh.. ok Oct 15 21:32:33 build #392 of uml is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/uml/builds/392 Oct 15 21:38:51 luka r38419 trunk/include/image.mk * ubifs: remove breaking commas from ubi calls Oct 15 22:16:28 how do you get bootup messages on a device that has no display? Oct 15 22:16:46 serial console generally Oct 15 22:16:54 ooooh... Oct 15 22:16:58 Oct 15 22:17:02 thx zinx Oct 15 22:17:19 if it's actually booting and you can connect, you can use dmesg for the kernel messages Oct 15 22:18:06 yeah, I know the dmesg trick. completely forgot about serial console Oct 15 22:18:17 can most devices flash via serial? Oct 15 22:18:25 idk, many can though Oct 15 22:19:24 the bootloaders are all different, mine lets you flash via serial, or you can choose to flash via tftp via a serial console menu, or you can flash via tftp if you held the reset button down while it booted :) Oct 15 22:19:51 cool Oct 15 22:20:25 when doing a full clean build from source, make clean is the only thing I need to do Oct 15 22:20:29 ? Oct 15 22:20:51 generally Oct 15 22:21:04 make defconfig does the defaulting in the subsequent Oct 15 22:21:07 make menuconfig Oct 15 22:21:14 clean doesn't erase your config Oct 15 22:21:20 ah Oct 15 22:21:32 rm .config is probalby best fore that Oct 15 22:21:46 there's probably a make rule to erase the local stuff too Oct 15 22:21:53 never had a need to use it myself Oct 15 22:22:05 i am hopping between devices, so i don't want to have any lingering issues.. Oct 15 22:22:25 all the device-specific files are in target/ i think Oct 15 22:22:34 ok Oct 15 22:22:35 which should be removed by a clean Oct 15 22:22:40 well, apart from the .config of course Oct 15 22:22:58 maybe it was build/ i don't know Oct 15 22:23:01 too many projects :) Oct 15 22:23:11 hmm.. so i just tried to flash trunk (built by me) onto a hame-a2.. and it's got port 22, 23, 53 in filtered state.. but i can't connect Oct 15 22:23:17 lol Oct 15 22:23:21 oh, no, staging probably has device-specific patches Oct 15 22:23:48 i just did a init ram flash from the web console of the device brand new Oct 15 22:24:00 openwrt must have booted, as it reset it's ip Oct 15 22:24:06 but for some reason, no connection.. Oct 15 22:41:45 what happens if you just flash the initramfs image Oct 15 22:54:11 you need to remove .config.old too or it'll be used, iirc Oct 15 22:55:21 nbd, i think i might have found what is triggering the kernel memory leak ... need more time to confirm Oct 15 22:57:34 we have a horrid shell script to keep user statistics, that digs through /sys/kernel/debug/ieee80211/phy0/netdev:wlan*/stations/ and then runs a bunch of wget's to an https server. when i killed off the script on our device i've been watching, the kernel memory leak stopped Oct 15 22:58:12 i'm running the script faster now to see if the leak returns Oct 15 23:00:25 mostly it's testing to see if particular stations exist, a [ -d /sys/kernel/debug/ieee80211/phy0/netdev:wlan*/stations/$foomacaddr ] Oct 15 23:00:29 mostly the don't Oct 15 23:00:42 not sure if it's that, or the wget's Oct 15 23:01:10 russell--: interesting Oct 15 23:01:58 nbd, I don't have definitive / quantitative results yet, but, got an aa build with backports you posted earlier on my systems Oct 15 23:02:17 MUCH improved, but, still seeing the dma errors when multiple systems on separate channels under heavy load Oct 15 23:02:40 when the heavy load is taken off of the other channel, this wrt160nl has sat for hours with no dma errors Oct 15 23:02:55 but, under some load (a 12 meg burst every 2 minutes or so) Oct 15 23:03:21 as soon as I start the all-sky on the other system, channel 11 doing it's heavy data transfer Oct 15 23:03:28 how frequent, and which dma errors exactly? Oct 15 23:03:30 I start seeing rx and sometimes tx dma errors on this one Oct 15 23:03:39 hang on, let me pull a log and get details, Oct 15 23:03:49 russell--: when you find out which exact call is causing the leak, i'll look into the code Oct 15 23:04:51 nbd: right Oct 15 23:05:09 will let you know what i find Oct 15 23:07:29 http://pastebin.ca/2467144 Oct 15 23:07:36 That's a snip to relavent messages Oct 15 23:09:31 after stopping the allsky on channel 11, this one on channel 1, hasn't had an error for almost 2 hours Oct 15 23:10:30 I'm still puzzling how I can narrow anything down farther here Oct 15 23:10:49 been a busy day, I haven't had much time to start looking Oct 15 23:11:04 hi Oct 15 23:11:26 i get this when i try to do a sysupgrade on a unifi ap pro Oct 15 23:11:26 Sysupgrade is not yet supported on generic. Oct 15 23:11:27 Image check 'platform_check_image' failed. Oct 15 23:11:49 what do i have to do to make it work? Oct 15 23:12:43 add support for that board into /lib/upgrade/platform.sh Oct 15 23:13:33 sounds like the board is not being detected correctly Oct 15 23:13:43 what is in /tmp/sysinfo/board_name and model Oct 15 23:15:44 cat /tmp/sysinfo/board_name Oct 15 23:15:44 cat: can't open '/tmp/sysinfo/board_name': No such file or directory Oct 15 23:17:50 groz: this is /proc/cpuinfo Oct 15 23:17:51 system type : Atheros AR9344 rev 2 Oct 15 23:17:51 machine : Ubiquiti UniFi AP Pro Oct 15 23:18:24 is this trunk, or aa, or ?? Oct 15 23:18:28 trunk Oct 15 23:19:01 rev 38407 Oct 15 23:19:03 I'd have to look at latest trunk, don't have it flashed on anywhere right now, but in earlier stuff Oct 15 23:19:19 somewhere in startup scripts, board detection code writes the board_name file Oct 15 23:19:31 same routines are used in upgrade, to determine how to do the upgrade Oct 15 23:20:54 it seems the board is missing in /lib/ar71xx.sh Oct 15 23:21:10 that would do it all right Oct 15 23:24:39 groz: what aa version are you using at the moment? Oct 15 23:24:59 I checked it out from svn just before I built Oct 15 23:25:02 ok Oct 15 23:25:05 hang on, I'll get v number Oct 15 23:25:11 no need Oct 15 23:25:18 38418 Oct 15 23:25:29 last change, 38401 Oct 15 23:25:30 i was just making sure it's not something old Oct 15 23:25:43 nope, checked it out, then grabbed the git stuff you posted earlier Oct 15 23:25:46 because there are other people using old AA versions with recent mac80211 packages Oct 15 23:25:53 subbed in the hostap and mac80211 directories Oct 15 23:26:13 well, this I will say, a long time ago had lots of similer problems with ar7240 Oct 15 23:26:27 but, now it's just on the wrt160, the ubiquity stuff, doesn't seem to have this anymore Oct 15 23:26:35 ar7240 in picostation and unifi Oct 15 23:27:01 but if you want specific tests, I'll build and test at any rev you request Oct 15 23:27:06 ar7240 ist just the cpu, not the wifi Oct 15 23:27:33 wifi is a pci express chip Oct 15 23:27:45 and if there is anything you want out of the units in terms of info, I'll get it if I know how Oct 15 23:28:01 and, I could set up remote access (including serial console) if you want it, and it'll help you Oct 15 23:28:11 no, probably won't help me at the moment Oct 15 23:28:22 what i'm interested in is how well the devices are working (ignoring all those messages) Oct 15 23:28:23 I have about a dozen wrt160nl kicking around Oct 15 23:28:33 ok, well, it's easy to quantify that Oct 15 23:28:44 I see little issue when the occaisional rx dma error shows up Oct 15 23:28:53 when the tx dma shows up, unit stops responding Oct 15 23:29:07 my monitor process will note shortly, it's not associated on the wds anymore Oct 15 23:29:12 ok Oct 15 23:29:12 tear down the wds and ap, rebuild them Oct 15 23:29:15 then it works fine again Oct 15 23:29:28 not sure if the wrt160nl is ever going to be 100% stable though Oct 15 23:29:40 the wifi chip it has is the oldest atheros 802.11n chip generation that they ever made Oct 15 23:29:42 now, one possible issue, my monitor process does do a scan every 15 minutes or so Oct 15 23:29:49 oh, right Oct 15 23:29:53 scans can easily mess up the state Oct 15 23:29:55 ya, I'm not super worried about 160, if the ubiquity stuff works ok Oct 15 23:30:09 I only have them for historical reasons, and plug them into spots where I want the usb connector Oct 15 23:30:21 actually, the scanning thing might be a useful hint for further code review Oct 15 23:30:28 this one in particular, is scheduled to be replaced by an optical fiber Oct 15 23:30:41 ok, I can also tell you one more important maybe detail Oct 15 23:31:00 in past, when I did a scan using the wds adapter as the scan adapter Oct 15 23:31:16 sometimes, after the scan, iwinfo would report tx power had been dialed down to 3 Oct 15 23:31:22 and I would reset it back up Oct 15 23:31:24 i fixed that Oct 15 23:31:28 it doesn't do that with this build Oct 15 23:31:37 i remember that bug Oct 15 23:31:41 groz solved the problem was a problem with my image Oct 15 23:31:42 iwinfo was actually triggering it Oct 15 23:31:42 thanks Oct 15 23:31:54 hehe, I kinda thought that may be the case Oct 15 23:32:00 iwinfo causes other issues at times too Oct 15 23:32:10 try not call it unless I want to get something I can't find another way to get Oct 15 23:32:24 the main problem with iwinfo was that it used a quirky way to test for multi-bssid support Oct 15 23:32:25 but, again, for the 160nl, I'm not super concerned, only reason I still use them Oct 15 23:32:31 is because I have them, and they have a usb plug Oct 15 23:32:32 it created an interface and tore it down again right after Oct 15 23:32:40 that might have caused all kinds of crazy stuff Oct 15 23:32:44 but that code has been replaced Oct 15 23:33:25 the 160 is the only device I have here, other than my old eeepc that has a wifi and usb adapter Oct 15 23:33:43 fyi, same code running on the eeepc 701 with the 2xx aetheros wifi stuff in it Oct 15 23:33:47 runs, no issues Oct 15 23:33:57 but, it's slow, and can't keep up to my data rates Oct 15 23:34:24 that's why I put the wrt160nl between the eeepc and the house with a wds link Oct 15 23:34:35 it's bridging for the eeepc Oct 15 23:34:58 it'll probably be next spring before we get an electrician to run a conduit with optical fiber in it Oct 15 23:35:13 have to wait for our old house to sell, before we have any $$$ to spend on upgrades here Oct 15 23:35:30 so I'm just cobbling together 'make do' with what I have on hand already Oct 15 23:37:46 nbd will a owner ship of the ubus socket be implemented to have some acl with ubus? Oct 15 23:39:01 yes Oct 15 23:39:05 but i don't know when Oct 15 23:44:15 oh, and one more point of interest, you may or may not find interesting nbd Oct 15 23:44:31 when we moved in here, I had to set up an electric fence to protect our beehives from bears Oct 15 23:44:53 so, I've got 110v out to the back of the lot in the form of an extension cord, into a waterproof box where the fence charger sits Oct 15 23:45:03 I put another 160nl in there, and have it wds in too Oct 15 23:45:06 just for giggles Oct 15 23:45:24 it's sitting on the fence charger, which produces a 5000V pulse every couple of seconds Oct 15 23:45:29 so, LOTS of electrical noise Oct 15 23:45:38 that one is NOT suffering from these issues Oct 15 23:45:47 kind of narrows it to wifi, not random electrical noise Oct 15 23:46:05 only units in close proximity to other access points do this Oct 15 23:46:13 at least in my setup here Oct 16 00:07:51 nbd groz the latest openvpn change brakes openvpn for the lzo compression Oct 16 00:08:01 its 38412 Oct 16 00:08:17 reverting it fixes openvpn Oct 16 00:09:05 just the changes in the init script need to be reverted Oct 16 00:09:08 package/network/services/openvpn/files/openvpn.init Oct 16 00:09:55 no both - b/package/network/services/openvpn/files/openvpn.config and package/network/services/openvpn/files/openvpn.init Oct 16 00:10:38 wait a second could be that my old config breaks it Oct 16 00:17:28 okay my fault i have to use "yes" instead of "1" Oct 16 01:02:32 tripolar: yeah, i merged a patch with that Oct 16 01:03:25 groz: nbd i veryfied the config its not right it now writes Oct 16 01:03:29 comp-lzo no Oct 16 01:03:33 and yes Oct 16 01:03:51 it it should just be comp-lzo when its used and no line at all when its not used Oct 16 01:03:58 luka: see http://openvpn.net/index.php/open-source/documentation/howto.html Oct 16 01:04:21 @luka Oct 16 01:04:42 # Enable compression on the VPN link. Oct 16 01:04:42 # Don't enable this unless it is also Oct 16 01:04:42 # enabled in the server config file. Oct 16 01:04:43 comp-lzo Oct 16 01:05:36 the connection gets established but doesnt work as expected afterwards Oct 16 01:06:42 reverting the commit fixes the problem Oct 16 01:09:43 00:17 < tripolar> okay my fault i have to use "yes" instead of "1" Oct 16 01:11:48 luka i allready changed this and afterwards it establishes a connection but traffic that gets sent over a brigdes doesnt reach the other end Oct 16 01:12:09 brigdes/bridge Oct 16 01:12:31 changing it to just comp-lzo - makes it work Oct 16 01:13:40 ok, i will run some tests tomorrow Oct 16 01:19:58 luka okay, could also be that the problem is my server which is running openvpn 2.2 Oct 16 01:24:15 luka it seems the problem here is not the "yes" its the "no", when just remove the line without setting it to "no" in /etc/config/openvpn fixes the problem here Oct 16 01:27:10 luka: found the problem - it's the server - it sets "comp-lzo" - so "comp-lzo no" creates a "dead" connection Oct 16 01:37:39 great **** ENDING LOGGING AT Wed Oct 16 02:59:58 2013