**** BEGIN LOGGING AT Fri Feb 14 02:59:59 2020 Feb 14 06:34:40 Morning Feb 14 06:35:29 JaMa: memtest ran for 8,5 hrs with 0 errors and was at 90-92 GB when I stopped it Feb 14 06:36:08 I'll run 90-128 GB some other time to make sure the last 2 modules are fine too Feb 14 08:57:52 JaMa: I'll rerun the full test to see how it behaves Feb 14 08:58:15 You reckon there will be much difference between 32GB and 64GB swap? Feb 14 09:36:11 JaMa: Hmmz again getting ICE on the all-cores one :S Feb 14 09:36:33 I guess I'll run memtest86 again tonight, will just put it to run earlier, so it's going to be done in the morning ;) Feb 14 10:55:57 looks like our jenkins doesn't like that benchmark much Feb 14 10:56:02 jenkins@bonaire:~/test-oe-build-time$ dmesg Feb 14 10:56:04 bash: /bin/dmesg: Input/output error Feb 14 10:56:07 jenkins@bonaire:~/test-oe-build-time$ htop Feb 14 10:56:09 Segmentation fault Feb 14 11:00:03 JaMa: Hmmz Feb 14 11:01:04 FYI for buying hardware, stock etc, I usually use https://geizhals.eu/ (whole EU, lots of German, NL, UK and Polish stores) or https://idealo.de (mainly German stores). Feb 14 11:01:36 Price wise Germany used to be a lot cheaper, but in recent years prices in NL and DE are similar mostly. So when I buy in NL I check https://tweakers.net/pricewatch/ as well Feb 14 11:03:10 Seems there's Epyc boards in stock ;) Feb 14 11:03:30 Seems my X399 can do 3* M.2 NVMe as well, so can always add more ;) Feb 14 11:38:07 https://www.alza.cz/amd-ryzen-threadripper-3990x-d5776458.htm there is one in stock :) Feb 14 11:38:46 this is biggest HW eshop in Czech Republic and it's quite well stocked for more consumer parts than Epycs :) Feb 14 11:40:12 JaMa: They're expensive though v.s. German shops ;) Feb 14 11:40:44 EUR 4350 vs EUR 4100 Feb 14 11:41:10 Most German shops deliver to NL without problems, not sure how in CZ, but should be similar hopefully ;) Feb 14 11:47:36 it might be more complicated with taxes and possible rma later Feb 14 11:50:59 Well it's all EU, so I don't care about it ;) I had very little RMA cases in general ;) Feb 14 11:51:21 Taxes should be no issue as well, unless you buy it as a corporate, then it's a different story v.s. private person of course ;) Feb 14 11:52:21 yes, I will buy it as a corporate Feb 14 11:53:13 Ah ok ;) Well then it's slightly more complicated, but still doable, at least here in NL ;) Feb 14 11:53:23 I can imagine things to be more tricky in CZ Feb 14 11:54:11 yeah and I might get a discount from alza, so in the end if you count that and shipping delay it's almost the same Feb 14 11:54:40 from alza I can have same day delivery when I finally decide what to buy :) Feb 14 11:54:47 Well Germany to NL is usually 1-3 days Feb 14 11:55:05 Seeing you're in PRague it shouldn't differ too much Feb 14 11:55:11 My memory came from Amazon Spain LOL Feb 14 11:56:26 https://geizhals.eu/?cat=ramddr3&xf=1454_32768%7E15903_DDR4%7E254_3200%7E256_4x%7E256_8x looks like memory kit might be the most difficult part to get the most from the system (without waiting for G.Skill kit in Q2 2020) Feb 14 11:57:21 there are few more options in https://pcpartpicker.com/ but most people don't need 256G, so there aren't many kits :) Feb 14 12:01:35 bonaire isn't happy Feb 14 12:01:37 Bitbake still alive (5000s) Feb 14 12:01:37 tee: 4-build-all-cores.log: Read-only file system Feb 14 12:03:56 will try to reboot it Feb 14 12:04:44 free -g Feb 14 12:04:44 Segmentation fault Feb 14 12:07:25 $ sudo reboot Feb 14 12:07:26 bash: /usr/bin/sudo: Input/output error Feb 14 12:07:41 ka6sox: we might need some help rebooting bonaire :/ Feb 14 12:13:02 novaldex|away: ^ Feb 14 13:04:31 JaMa: What are you planning to buy? 3990x ? Feb 14 13:05:18 will see after the benchmarks Feb 14 13:05:44 if it's only 10% faster in the benchmark, then will go with 3970x Feb 14 13:05:52 But you must have some ideas ;) Feb 14 13:06:03 So either 3970 or 3990? Feb 14 13:06:21 I have ordered Epyc 7502 back in November Feb 14 13:06:43 then canceled it when they postponed MZ32-AR0 MB delivery to at least january Feb 14 13:06:52 and 3970x was released Feb 14 13:07:32 so I've ordered 3970x instead 7502P and canceled it as well when AMD confirmed that there will be 3990 announcement on CES in January Feb 14 13:09:13 so it will be either 3970x, 3990x, 7502P or 7702P, depending on how beneficial 8 channel memory in Epyc will be Feb 14 13:10:30 JaMa: Ah OK Feb 14 13:11:12 3990x is the most tricky one from this group, I would probably go with ASUS ROG ZENITH II EXTREME ALPHA jus tto have higher chance to be able to run 3200 or 3600 256GB memory kit (other motherboards should be compatible with 3990x as well in theory, but the VRM might not be as good to keep 256GB kit stable with 128threads) Feb 14 13:11:45 and with 128threads I don't want to go with just 128GB kit Feb 14 13:12:17 with 3970x I don't need to be so picky about the MB and 128GB kit would be fine Feb 14 13:15:10 7502p will be slightly more expensive than 3970x while significantly cheaper than 3990x without any compromises on memory, but with those difficult to cool server motherboards Feb 14 13:16:02 and 7702p is again slightly more expensive than 3990x but without any limitation on memory Feb 14 13:17:59 in the end I might try to get IT to just consolidate all servers I have access to with just a few epyc and then I don't need any builder at home :) Feb 14 13:19:08 and honestly 1600AF (which I bought as temporary replacement when my old motherboard died before christmas) isn't so terrible as well if I buy some memory for it as well :) Feb 14 13:19:30 and it's only 80$ :) Feb 14 13:21:20 JaMa: Yeah if you have a good remote server you don't need so much at home ;) Feb 14 13:21:33 8 * 32GB kits seem hard indeed Feb 14 13:22:41 Corsair seems to have some kits too, but they don't seem to appear on Asus QVL for now: https://www.corsair.com/eu/en/Categories/Products/Memory/VENGEANCE-LPX/p/CMK256GX4M8E3200C16#tab-tech-specs Feb 14 13:23:26 there are some 3200 CL16, but some are 16-20-20-38 garbage (by someones standards) and 16-18-18-38 are even more rare (e.g. https://pcpartpicker.com/product/rJkgXL/gskill-ripjaws-v-128-gb-4-x-32-gb-ddr4-3200-memory-f4-3200c16q-128gvk is current favourite if g.skill doesn't deliver the promissed 3600 CL16 kit soon enough) Feb 14 13:24:05 Yeah Corsair is 16-20-20-38 as well Feb 14 13:24:15 maybe I'm just listening too much to https://www.youtube.com/channel/UCrwObTfqv8u1KO7Fgk-FXHQ Actually Hardcore Overclocking :) Feb 14 13:24:38 for him everything which doesn't OC over 4000 or with really low latency is complete garbage :) Feb 14 13:25:24 for me I just want the tmpfs to be significantly faster than disk, which is harder and harder with NVMe drives getting so fast :) Feb 14 13:26:52 that's probably main reason why I'm looking at 8 channel Epyc so much, so that it has enough memory bandwidth for build and tmpfs Feb 14 13:27:38 CAS shouldn't matter too much if not overclocking Feb 14 13:27:53 With the heat of the TR you probably don't want to OC too much anyway Feb 14 13:31:19 ASrock seems to support the following 2: 8*32GB: Samsung M378A4G43MB1-CTD and Crucial CT32G4DFD8266.16FB1 Feb 14 13:32:58 there aren't many benchmarks concentrating on memory for 3rd gen trx, but from what I've read the latency is important for TRX and 3600 is best for infinity fabric (which is easy to do with smaller kits, but problematic with 32GB modules) Feb 14 13:33:26 MSI doesn't even have anything on their QVL in terms of 32GB modules Feb 14 13:34:12 https://www.youtube.com/watch?v=MDKQGLFqrS0 Feb 14 13:34:57 yes, that's the issue with TRX40, for EPYC you can buy cheaper 256GB kit even with ECC Feb 14 13:35:12 with slower frequency, but well compensated by double channels Feb 14 13:45:11 Gigabyte seems to have some on their list but also not a whole lot. Same Corsair as Asus and a few other ones Feb 14 13:45:40 there are the g.skill 3200 kits in QVLs I was looking at Feb 14 13:47:17 well it's still just few %, but on this kind of budget I don't want to save few USD to get suboptimal memory bandwidth Feb 14 13:47:31 and considering that I'm looking for a new builder for over a year.. :) Feb 14 13:48:20 but now I have 20+ servers available for myself and my CI jobs, so it's easier to just schedule the builds there Feb 14 13:49:15 and LuneOS builds are small enough even for 1600AF with 64G ram Feb 14 14:06:38 JaMa: They are in QVL, but are they available? That's the big question usually Feb 14 14:07:24 I need to have a look why mine run @ 2666 and not at 2933 which thye should be able to do Feb 14 14:09:56 2666 is the 2nd gen default for 8 sticks Feb 14 14:10:19 the modules might be able to do 2933, but you need to explicitly enable their XMP Feb 14 14:10:47 and then the controller might not cope with that as they are also 2R Feb 14 14:12:20 https://en.wikichip.org/wiki/amd/ryzen_threadripper/2990wx#Memory_controller even says 1866 Feb 14 14:15:11 the ripjaws 3200 kits are available here Feb 14 14:15:30 p Feb 14 14:15:34 https://www.alza.cz/gaming/g-skill-64gb-kit-ddr4-3200mhz-cl16-ripjawsv-d5720773.htm Feb 14 14:16:40 but not as kit of 8, because nobody is buying those, on pcpartpicker there are kits of 8 available Feb 14 14:17:08 https://pcpartpicker.com/product/2KhmP6/gskill-ripjaws-v-256-gb-8-x-32-gb-ddr4-3200-memory-f4-3200c16q2-256gvk Feb 14 14:19:02 well not in stock anymore today Feb 14 14:20:04 or rather not available in europe, but in stock for US Feb 14 14:24:57 Seems there is only 1 store selling 128GB kits of those in Europe :S https://geizhals.at/g-skill-ripjaws-v-schwarz-dimm-kit-128gb-f4-3200c16q-128gvk-a2177027.html?hloc=at#offerlist Feb 14 14:25:07 And they would deliver them in about 1 week Feb 14 14:34:26 yeah and I'm even considering waiting for 3600 kits to be available :) Feb 14 14:34:52 ryzen 4000 will be a thing before I finally build this 3rd gen :) Feb 14 14:35:10 but at least I already have a case, AIO, NVMe drive.. :) Feb 14 14:36:39 Hehe ;) Feb 14 14:36:47 Got a proper PSU as well? Feb 14 14:37:47 Used to have Antec, but switched to Seasonic Prime Ultra now Feb 14 14:41:08 Got a simple but elegant Factal Design Define R5 case at my end, nothing fancy but works :) Feb 14 14:41:08 yes, corsair HX1000 Feb 14 14:41:16 And doesn't break the bank ;) Feb 14 14:41:54 I have http://www.phanteks.com/Enthoo-719.html because I was looking at some SSI-EEB boards which doesn't fit in R5 IIRC Feb 14 14:42:48 but with the PSU shroud there isn't too much space above the MB for the radiator and fans Feb 14 14:44:25 http://www.lian-li.com/pc-o11d-rog/ would be a bit better option if I should decide again today Feb 14 14:48:04 LianLi makes nice cases Feb 14 14:48:10 My previous builder was in LianLi ;) Feb 14 14:49:45 But that was a MiniITX board with AM3 ;) Feb 14 14:50:07 yeah, but needs extension board for EEB boards and I haven't found it in stock here (when I was looking for case for Asus KRPA-U16 Feb 14 14:50:52 http://www.lian-li.com/o11dxl-2/ Feb 14 15:02:09 Yup extension board will be tricky to get probably :S Feb 14 15:02:18 Some of these things are really obscure and hard to get :S Feb 14 15:02:34 Like the elusive 256GB memory kits, I guess there's not so much demand ;) Feb 14 15:03:11 yes, while for Epyc you can easily buy 16x32G without any issues (other than price) :) Feb 14 15:04:28 even AMD says that 3990X is kind of edge case good only for specific workloads and 3970x or 3960x is for general HEDT Feb 14 15:05:22 the benchmark script is just my way to figure out if OE builds fall in this kind of specific workloads :) Feb 14 15:07:10 my order was for 8x32G with an option to upgrade to 16x32G later after benchmarking the tmpfs benefit on 8 channel memory Feb 14 15:09:03 unfortunately those 2 dual-xeons I was testing weren't really good for memory bandwidth.. one was using just 4x64 instead of 8 sticks, and the other one had unbalanced 12sticks on 8 channels Feb 14 15:10:26 and the 4x64 one with newer CPU had 5 times _slower_ memcpy https://github.com/shr-project/test-oe-build-time/blob/master/table.mbw.csv Feb 14 15:11:50 an my 1600AF with 3200 kit is about twice faster :) Feb 14 15:18:50 Yeah TR seems limited to 8 slots, but Epyc you can go 16 without issues ;) But of course you'll need a bigger board for that as well ;) Feb 14 15:21:05 and also TR is limited to UDIMMs which currently doesn't exist bigger than 32G Feb 14 15:21:25 while 64G LRDIMMs are common Feb 14 15:22:25 Yes true ;) But I simply don't have Epyc (7702) money LOL ;) Feb 14 15:23:11 and it wouldn't make any sense to build such monster if OE builds aren't your job :) Feb 14 15:23:25 2990WX is already more than enough overkill :) Feb 14 15:23:30 True ;) Feb 14 15:23:56 Still curious what causes my ICE's though... Will investigate a bit more Feb 14 15:25:42 yeah on 72 threads it looks like in some cases 192G ram+swap isn't enough, so it might be the OOMK killing something which resulted in broken input for gcc or something like that Feb 14 15:25:58 but I would definitely run that memtest and bios upgrade Feb 14 15:27:02 tonight my build also ended with OOMK even when it should have had enough swap Feb 14 15:27:18 it depends on how badly the linking phase of big do_compile tasks align I guess Feb 14 15:28:56 JaMa: Bios was updated last night before I kicked off the memtest Feb 14 15:29:05 Memtest did 0-90-ish GB without any issues Feb 14 15:29:18 Will start to run it when home tonight right away so it will finish 0-128GB overnight ;) Feb 14 15:30:02 yeah it takes a while with so many memory Feb 14 15:30:41 I remember testing 16G systems to couple passes overnight or many passes over weekend, but with 128+ GB it's getting tricky :) Feb 14 15:31:40 but once had to RMA some ADATA kit 3 times before I got kit which passed couple passes :) Feb 14 15:32:01 or 2 RMAs, 3rd kit was ok Feb 14 15:34:20 Well that's Adata ;) Feb 14 15:34:56 I would expect better from Corsair ;) Feb 14 15:35:26 JaMa: Optane v.s. NVMe: No real benefit I would say going by various non-OE benchmarks? Feb 14 15:36:36 I don't know, but the price difference is huge Feb 14 15:37:34 and the performance in numbers doesn't looks so impressive when comparing with PCIE4 NVMe drives (IOPS as well as seq read/write) so I have no idea how it would compare in e.g. OE build Feb 14 15:38:17 Yeah I don't think it's worth the money ;) Feb 14 15:38:22 it's still cheaper to have huge 8TB raid 0 with 4 NVMe drives than 2TB optane (at least last time I've checked) Feb 14 15:42:16 JaMa: Yeah it seems so ;) Hence I have 10TB in RAID1 ;) It's mainly for some storage ;) And I might add some more NVMe later, have the possibility ;) Feb 14 15:42:27 And 1TB is still cheaper vs. 2TB it seems Feb 14 15:43:04 JaMa, Herrie working on bonaire Feb 14 15:43:13 (saw your request) Feb 14 15:43:17 ka6sox: Thnx Feb 14 15:45:51 Herrie: 10TB in NVMe? Feb 14 15:46:08 ka6sox: thanks Feb 14 15:46:36 ka6sox: it was running the OE benchmark since last night, so maybe something didn't like the load Feb 14 15:47:05 but strange that it would cause IO errors and result in RO / and /home Feb 14 15:47:28 saw that...I'm doing an analysis now and then will reboot it Feb 14 15:47:32 (looking at logs) Feb 14 15:55:24 btw there was some progress on qtwayland recently at lge, hopefully it will be merged and public in couple weeks Feb 14 16:03:27 JaMa: No 10TB regular HD's at my end an 1TB in NVMe ;) Feb 14 17:25:14 ka6sox: I see bonaire is back online and working, can I run the benchmark there or rather not or later? Feb 14 17:26:56 later please...we are doing an analysis of why this failure occured. Feb 14 17:28:41 ok, thanks Feb 14 17:38:52 JaMa: memtest running again **** ENDING LOGGING AT Sat Feb 15 02:59:57 2020