**** BEGIN LOGGING AT Tue Jun 18 02:59:57 2019 Jun 18 04:59:24 pratimugale: yes. Make the C apis use daemon. Also, the ideas, I think most of them would require a kernel driver. And I was thinking to not go into the kernel thing. Jun 18 05:26:33 zeekhuge: Okay, will discuss with abhishek_ too and think of more ideas Jun 18 05:54:15 pratimugale : In terms of hardware requirements of your ideas a good thumb rule to go by is that you shouldn't have to spend more than a day debugging it if it doesn't work, and should be buildable without requiring a custom board, some breadboarding is OK Jun 18 05:58:14 abhishek_: okay Jun 18 08:42:45 embden[m]: Are you online? Jun 18 08:43:09 yes Jun 18 08:43:16 julieng: Jun 18 08:46:07 embden[m]: On your weekly e-mail you had a couple of blockers (I leave the debugger aside for now). Jun 18 08:46:47 embden[m]: I imagine your problem with booting is because the secondary CPU does not boot? Jun 18 08:49:35 julieng: I think so. I think because it doesn't boot in nonsec mode. Jun 18 08:50:08 Right now, I am trying to configure u-boot to enable nonsec mode Jun 18 08:56:36 julieng: I have a strange error: though in my .config file I have CONFIG_ARMV7_NONSEC=y, linker can't find armv7_boot_nonsec function from arch/arm/lib/bootm.c Jun 18 08:57:00 embden[m]: I am not entirely convince U-boot is part of the equation here. From my understanding, on TI, the secondary CPU will be turned off and directly turned on by Xen/Linux. Jun 18 09:00:07 embden[m]: If you were going the U-boot way, then you would need to implement a few platform specific option (potentially the SMP bring-up) and also PSCI. This is not a totally trivial task. Jun 18 09:03:24 embden[m]: The reason I mention PSCI is U-boot will take care of bring-up all the CPUs. The secondary CPUs will need to be hold in U-boot until Xen is ready to handle them. Jun 18 09:03:52 PSCI would be the way for Xen to ask U-boot to release CPU X. Jun 18 09:03:56 julieng: so, how should I proceed? Jun 18 09:04:26 embden[m]: Trying to boot Linux on baremetal and see if CPUs are brought up in Hypervisor mode. Jun 18 09:05:17 embden[m]: If they are brought up in hypervisor mode (You should see KVM initializing), then the issue is lying in Xen. If not, that's going to the ROM (whoever is reading AUXCORE1). Jun 18 09:05:34 embden[m]: Could you give a try a Linux baremetal and provide the logs? Jun 18 09:06:02 julieng: ok Jun 18 09:06:12 embden[m]: making sure that your kernel of choice has LPAE, virtualisation and KVM enabled Jun 18 09:11:31 julieng: https://pastebin.com/XrvL0gTx Jun 18 09:12:53 julieng: [ 0.020045] CPU: All CPU(s) started in HYP mode. Jun 18 09:13:59 julieng: maybe the actual secondary pen is in ROM? Jun 18 09:14:18 embden[m]: Ok, so the problem is in Xen. Jun 18 09:14:51 apritzel: I think it is in the ROM. We probably are not configuring it correctly. I noticed that the Linux SMP for OMAP grew up quite a bit. Jun 18 09:15:25 julieng: should there be LPAE support? bb-x15 has 2 GB of RAM Jun 18 09:16:42 embden[m]: LPAE is not about the amount a RAM. It is a mandatory feature for virtualization. Actually the hypervisor page-table can only be LPAE (not short-page-tables) Jun 18 09:18:00 embden[m]: Now that we know the issues in the SMP code, it would be interesting to see the difference in configuration between Xen and Linux. The SMP code for Xen can be found in xen/arch/arm/platforms/omap5.c. For Linux it would be arch/arm/mach-omap2/omap-smp.c. Jun 18 09:19:48 One of the key difference I can spot so far is Linux supports SMP bring-up using secure API (i.e SMC). I am wondering whether this path is going to be used on the X15. Jun 18 09:22:05 julieng: doesn't look like it Jun 18 09:22:41 omap_secure_apis is only set when cpu_is_omap44xx() returns true Jun 18 09:23:01 so you can ignore that path for now Jun 18 09:25:08 there seems to be some secure API in place, though Jun 18 09:25:22 julieng: check set_cntfreq() Jun 18 09:26:39 julieng: which brings up the question whether U-Boot is actually started in secure mode Jun 18 09:26:53 if not, we won't have the chance of installing PSCI handlers there Jun 18 10:04:19 apritzel: As Linux is able to bring-up secondary CPUs without PSCI, I would prefer not to spend time try to get PSCI working. It would be best if Xen manage to use the TI SMP protocol. Jun 18 10:04:32 julieng: I agree Jun 18 10:30:51 embden[m]: I think we only covered one of the blocker. How about the second blocker? Jun 18 10:32:02 julieng: debug via serial port? covered in your reply in ML Jun 18 10:38:36 embden[m]: Ok. Let me know if you have any other issues. Jun 18 10:40:58 julieng: I am wondering whether I should use astartup function from linux .../omap-headsmp.S omap5_secondary_hyp_startup in xen head.S for arm32 Jun 18 10:49:25 embden[m]: I think we already have something similar in Xen. See init_secondary. Secondary CPUs will spin until smp_up_cpu match the MIDR. Jun 18 10:51:31 embden[m]: Actually, I read omap5_secondary_startup instead of omap5_secondary_hyp_startup... Sorry for that. Jun 18 10:52:25 The last 3 instructions of the function looks necessary for Omap4. You can try to add them in head.S and see whether it works. Jun 18 11:43:10 zeekhuge abhishek_ I modified the C apis to use the daemon today: https://github.com/pratimugale/PRUSS-Bindings/blob/master/c-bindings/ Jun 18 11:43:11 Also added userspace programs and tested it. Jun 18 11:43:12 Now currently I'm working on adding the pru memory read write functions to the Daemon Jun 18 12:03:18 julieng: Jun 18 12:42:28 embden[m]: There no content in your message. Is that a mistake? Jun 18 13:15:15 julieng: yes. I wanted to ask what this function does: https://github.com/torvalds/linux/blob/master/arch/arm/mach-omap2/omap-headsmp.S#L61 line by line. I don't especially understand smc, badr,mrc and what we read in the first instruction Jun 18 13:17:07 embden_[m]: mrc is used to read a co-processor register (they are registers to configure your processors). The counterpart is mcr to write in a co-processor registers. Jun 18 13:17:23 In this case, the code is reading MIDR which contains the Processor ID. Jun 18 13:18:01 smc is a call to the secure monitor (you can relate this to a syscall but between the kernel/hypervisor and secure monitor). Jun 18 13:18:42 is it CP15 coprocessor? Jun 18 13:19:30 badr is a macro implemented in arch/arm/include/asm/assembler.h. The macro will load the address of a symbol in a register. Jun 18 13:21:28 embden_[m]: CP15 is one of the existing co-processor. The rest of the instruction (0, c0, c0, 5) will tell which exact register you want to access. You can usually grep the value in the Arm Arm to find the name of the register. Jun 18 13:31:32 julieng: but what is the argument for smc? is it a placeholder? Jun 18 13:34:23 embden_[m]: Do you mean #0? Jun 18 13:48:46 julieng: Jun 18 13:48:48 yes Jun 18 13:48:48 embden_[m]: somewhat, in ARMv8 you can easily access this number from the handler, but not in ARMv7 Jun 18 13:49:25 so this number is rarely used, instead people put a function number is some register, typically r0 Jun 18 13:49:33 but in this case it's actually r12 Jun 18 13:49:42 and r0 contains an argument Jun 18 13:50:30 apritzel: where are the hadnlders are located? Jun 18 13:50:41 embden_[m]: that is a good question! Jun 18 13:51:01 Either in BootROM, but I doubt this Jun 18 13:51:14 or in some code loaded before U-Boot Jun 18 13:51:47 did you build the whole "firmware" for the board? Is there anything needed beyond U-Boot? Jun 18 13:53:13 no, I have only u-boot Jun 18 13:53:50 embden_[m]: is it mainline U-Boot? Or some special branch? Jun 18 13:56:56 seems like U-Boot calls in there as well, and a comment calls it "ROM Code" Jun 18 13:58:59 apritzel: mainline Jun 18 14:01:06 embden_[m]: did you come across something called "ML0" when creating the SD card? Jun 18 14:01:53 apritzel: yes, as a part of u-boot Jun 18 14:03:07 ah, it's MLO, not ML ... Jun 18 14:03:30 I mean U-Bott being built produces MLO Jun 18 14:08:02 yeah, it looks like this is the SPL, wrapped in some special image format Jun 18 14:20:41 abhishek_: How should I proceed with the examples? zeekhuge said that kernel drivers should be less Jun 18 15:54:10 pratimugale: is it the same repo ? Or have you shifted to a different (forked) repo ? Jun 18 17:03:35 apritzel: MLO is SPL + a header Jun 18 17:04:09 ds2: thanks, that was my impression as well Jun 18 17:04:40 ds2: do you know where the magic SMC handlers live? in the Boot ROM? Jun 18 17:27:34 apritzel; I think in the rom on the SoC that normal people can't get access to source wise Jun 18 17:28:16 ds2: I see, thanks Jun 18 17:28:38 ds2: so does this mean we can't run any own secure code on this SoC? Jun 18 17:29:04 apritzel: yes... as far as I know this P/N does not let you use secure mode at all Jun 18 17:29:40 that magic MLO header would also have some kind of signing (vs a dumb has in our case) Jun 18 17:30:57 yeah, this seems to be pretty common, though. This actually makes some sense, since you don't want to run random bytes you picked up from an SD card, I guess Jun 18 17:31:27 yep Jun 18 17:32:06 side note MLO or a variant is used in all indirect access boots (AFAICT) - so that also includes NAND Jun 18 17:32:22 it is also know as x-loader in case you come across it in older docs in the context of NAND boot Jun 18 17:33:06 ds2: thanks Jun 18 17:33:10 on older parts that is a suffix to indictate if secure mode is available or not and if so, the MLO/x-loader/SPL needs to be signed with the right key Jun 18 17:33:36 ah, I see Jun 18 17:34:48 well, good news is we don't need to waste our time with a U-Boot PSCI implementation then ... Jun 18 17:36:28 what is PSCI? Jun 18 17:39:33 ds2: It is a set of standard call to manage the state of the machine (system off, suspend...) and also the CPU (on, off, suspend). Jun 18 17:40:02 standard as in ARM standard? like the ARM version of ACPI? Jun 18 17:40:24 ds2: AFAIK this is only used by Arm so far. Jun 18 17:41:02 ds2: Well, the other solution is each vendor to implement there custom SMP protocol and all the softwares have to replicate it (which can be a massive pain). Jun 18 17:41:11 got it. should have asked it better... was trying to figure out ARM std vs I std :D Jun 18 17:41:18 ds2: http://infocenter.arm.com/help/topic/com.arm.doc.den0022d/Power_State_Coordination_Interface_PDD_v1_1_DEN0022D.pdf Jun 18 17:41:52 thanks... just expanding the name helps... sounds a lot like ACPI on x86 Jun 18 17:42:00 ds2: I think every arch like to re-invent there own protocol (or sligthly alter standard to make it more difficult) ;). Jun 18 17:42:56 ds2: ACPI is much more complex than PSCI. The former is also doing a lot more things. Jun 18 17:44:00 ds2: On Arm, you still need PSCI to bring-up CPU even with ACPI. Jun 18 17:44:59 julieng: one more dumb question - for practical purposes, how does one go from this document to applicable cores? I assume this won't apply to A5, A7, A8, A9 Jun 18 17:49:13 ds2: PSCI is part of the firmware, ideally provided by the board Jun 18 17:49:33 U-Boot has an implementation of it, you just need to wire in the core/SoC/platform specific bits Jun 18 17:50:05 that is what we were after, but this approach can't work on OMAP5, since we need to provide an SMC handler Jun 18 17:53:41 PSCI just provide a well defined interface for SMC calls, so the kernel just puts 0x84000002(CPU_OFF) in r0 and calls smc #0, and the firmware will do whatever is needed to turn that core off Jun 18 17:54:40 ds2: it's more the equivalent of the x86 mptable, if you like Jun 18 18:03:44 'k Jun 18 18:59:16 julieng: so, I don't understand what the next step is Jun 18 19:04:07 Embden sorry for the confusion. It sparked a long discussion. The next step would be to do the smc call that switch to hyp in the boot code. Jun 18 19:04:29 You can hack it for now so we have an idea whether it works. Jun 18 19:05:33 jkridner: will CONFIG_DYNAMIC_DEBUG work?https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html, I was following this instructions, I have posted the current logs here : https://github.com/vaishnav98/gbsim/issues/6 Jun 18 19:05:47 julieng: I don't understand where and what arguments to put? I am not very successful in finding smc description. Jun 18 19:07:53 ds2 : could you provide some more details on how to do this? The transfers happen through greybus where through the simulator hex values of each transfer can be logged , also had a look at the mmc_spi driver could not find any additional information Jun 18 19:09:21 julieng: I've tried to add this: ldr r12, =API_HYP_ENTRY Jun 18 19:09:22 smc #0 Jun 18 19:09:23 but with no success Jun 18 19:16:13 vaishnav98_[m]: the easiest hack to be to find out where the printk that complains about invalid cSD is and just add some small code there... I think CSD process is in the common code... mmc_spi appears just as another transport Jun 18 19:16:46 vaishnav98_[m]: do a search for that error string, i.e. use grep, ctags, or whatever your favorite tool is Jun 18 19:22:00 ds2 Sure, will do that , Thanks for the quick reply :) Jun 18 19:46:53 embden[m]: I am not aware of any documentation for the TI SMC unfortunately. The best way is to look at the Linux and try to replicate it. Jun 18 19:47:13 embden[m]: I think you also need to set r0 to the address you want to jump after the call. Jun 18 19:47:33 Also where did you call it in Xen? Jun 18 20:00:02 zeekhuge: Sorry for the multiple messages, the Riot app wasn't sending them for some reason and I clicked on resend message Jun 18 20:13:42 hey, you around? Jun 18 20:35:43 julieng: right after the message: RINT("- CPU doesn't support the virtualization extensions -\r\n") Jun 18 20:37:06 oh, I feel myself stupid Jun 18 20:41:57 embden you need to do it before the check. Otherwise you will still fail to bool as the next bit is a panic. Jun 18 20:43:51 please give feedback on this progress video : https://youtu.be/yTIs10-VrbM Jun 18 20:44:46 julieng: yes, that's the reason for feeling this way Jun 18 20:45:41 Don't worry :). Going through the boot code is not that simple. Jun 18 20:46:56 julieng: how can I substitute badr? "adr rd, SYM"? Jun 18 20:50:13 abhishek_: @hendersa ds2 can you suggest how to progress further regarding the kernel module development. Jun 18 20:50:21 embden[m]: adr r0, = should work Jun 18 20:51:44 embden[m]: Also, bear in mind that the position I suggest will hopefully allow you to go past the hyp mode check but it may break afterwards. We can cross that once you confirmed the secondary CPU is in hyp mode. Jun 18 20:52:33 why "=" is needed? Jun 18 20:53:12 embden[m]: Because I confused adr and ldr. Sorry for that. The = is not necessary Jun 18 20:54:06 embden[m]: I am about to switch off for the rest of the evening. You can probably ask sstabellini if you still running into issue. Jun 18 20:56:20 julieng: thanks Jun 19 02:55:52 wait, is there a TI aarch64 device somewhere in the mix this year? **** ENDING LOGGING AT Wed Jun 19 02:59:57 2019