**** BEGIN LOGGING AT Fri Aug 16 03:05:29 2019 Aug 16 09:45:25 For some example code to run DM timers see these links for some code to create a kernel module. The original AM335x one is https://e2e.ti.com/support/processors/f/791/t/415079 and a followup for AM572x was http://e2e.ti.com/support/processors/f/791/p/821652/3044672. Aug 16 09:51:49 drhunter: ok, but I want first to set up the crossbar because it definetely doesn't do mapping right Aug 16 10:29:49 drhunter: also, am I right that I need then an array that maps MPU_IRQ number of crossbar to gic ID? Aug 16 10:51:28 drhunter: I can't find how in Linux it is done (the mapping between MPU_IRQ and GIC number) Aug 16 10:56:59 embden_new[m]: See allocate_gic_irq() in irq-crossbar.c Aug 16 10:57:18 This is called when the IRQ is allocated by Linux. Aug 16 11:03:07 julieng: oh, now I can see that gic line number = crossbar line + 32 Aug 16 13:13:13 embden_new[m]: I don't think that's guaranteed. Aug 16 13:29:35 If you look at the function I pointed out. We we iterate through an array to find a free index. Aug 16 13:29:48 Once a free index is found, then we write the crossbar line. Aug 16 13:30:23 The index is an SPIs number. In order to find the GIC interrupts, you need to add 32. Aug 16 13:33:23 julieng: >embden_new: I don't think that's guaranteed. Aug 16 13:33:24 I found it in the TRM for the crossbar Aug 16 13:34:12 I think we have a different definition of the crossbar line :). Aug 16 13:34:30 https://github.com/embeddedden/xen/commit/fee7a2c0eb32d5c219c10ad0ed0064f48b70b625 Aug 16 13:35:04 From your perspective, it looks like the crossbar line is the output of the crossbar (i.e the index) that is wired to the GIC. Aug 16 13:35:05 I added something and you can't imaging it (at least I haven't expected it) interrupts started to work Aug 16 13:35:28 julieng: yes Aug 16 13:35:55 julieng: also, ping started to work Aug 16 13:36:12 https://pastebin.com/syvTUDHH Aug 16 13:36:16 embden_new[m]: Do you see numbers incrementing in /proc/interrupts? Aug 16 13:37:02 embden_new[m]: Looks like you knew what I was about to ask :). Aug 16 13:37:26 So yes you are receiving interrupts! That's quite an achievement! Aug 16 13:37:52 julieng: they are incrementing though I thought they wouldn't Aug 16 13:38:39 embden_new[m]: Why not? The crossbar is just about routing. All the rest is in the realm of the GIC and should already work. Aug 16 13:39:07 So if you able to receive one interrupt, you should be able to receive all of them. Aug 16 13:39:45 I can see you are receiving them on several lines :) Aug 16 13:40:10 julieng: I think they started to work due to your workaround Aug 16 13:40:22 (allowing all of them) Aug 16 13:42:11 julieng: and also am I right that they work till vgic==gic? Aug 16 13:46:56 embden_new[m]: My workaround is only routing all interrupts not used by Xen in 1:1 way (i.e vIRQ == pIRQ). Aug 16 13:48:12 I am a bit confused by your code. Aug 16 13:48:48 I can see you are writing to the host crossbar the exact value written by dom0 Aug 16 13:48:59 julieng: yes Aug 16 13:49:15 But then you are walking through crossbar_offsets. What is it for? Aug 16 13:50:12 julieng: I was in a process of making crossbar make right routes to the vgic Aug 16 13:50:44 for that purpose I need to find out the gic irq number Aug 16 13:52:16 as I said. right now it works only due to your workaround - I haven't finished work with routing Aug 16 13:55:47 julieng: and by index i in crossbar offsets I am finding to what vgic line an irq installed by Linux should be routed Aug 16 13:56:27 For future testing the /cat/proc file gives a list of interrupts that are triggered by the kernel in normal operation without having to write test code. Aug 16 13:56:39 A good candidate to work with would be Aug 16 13:56:39 335 0 CBAR 51 Level 48070000.i2c Aug 16 13:57:51 drhunter: /cat/proc? Aug 16 13:58:30 i2c is it an i2c channel to some kind of memory? Aug 16 13:58:31 cat /proc/interrupts Aug 16 14:00:50 It is an I2C bus that has a couple of devices hanging on it - TPS659038 (power management), a TMP102 temp sensor, control channel for an TLV320AIC3104 audio codec and an EEPROM. Aug 16 14:01:56 Its not possible to tell which ones of there are generating the interrupt for I2C0 but the fact they are is all we need to test routing through the xBAR Aug 16 14:02:28 ok Aug 16 14:12:20 julieng: so, if I remove your workaround there will be no guarantee that cirq == pirq, right? Aug 16 14:24:21 embden_new[m]: If you remove my workaround, then the interrupt will not be routed at all to Dom0. Aug 16 14:27:05 embden_new[m]: Also routing the interrupt while the guest is alive may bring some interesting problems. Aug 16 14:27:59 julieng: so, should I rework it? Because I also was thinking about rerouting to online guest (I have seen also some checks that abandon this kind of behavior) Aug 16 14:28:06 For instance, you have to be careful the state of the physical GIC interrupt when routing/release IRQ to/from a guest. Aug 16 14:28:39 rerouting interrupts is not trivial if you want to be safe. Aug 16 14:29:14 What happen if the interrupt were active when rerouting? Aug 16 14:29:48 As dom0 is the only users of the crossbar, it feels to me that all the GIC interrupts related to the crossbar should be routed to Dom0. Aug 16 14:30:11 This should be done during building. Aug 16 14:30:50 This will impose a 1:1 mapping, but I think this is fine as the bindings would allow you to mark some of the interrupts as reserved. Aug 16 14:31:52 julieng: ok, but wouldn't dom0 expose the crossbar to domU? or will domU use a special virtualization interface in dom0's kernel? Aug 16 14:33:02 embden_new[m]: Why would the crossbar need to be exposed to domU? Aug 16 14:35:13 Without device passthrough, all interrupts used by the guest would be GIC interrupts. Aug 16 14:36:13 With device passthrough, some of the interrupts may go through the crossbar. But then, AFAIK, the platform does not have an IOMMU... Aug 16 14:36:50 so DMA-capable device could not be safely assigned to the guest. Aug 16 14:38:06 for non-DMA capable device, it might be possible. However, I would like first to see a use case that warrant to virtualize the crossbar for the guest. There might be other way to do it, such as Xen dealing the crossbar allocation (guest device driver should not depend on crossbar). Aug 16 14:38:59 In any case, I think we should first focus on Dom0. This would already be a big step. Aug 16 15:00:11 ok, so, what should I do next? Aug 16 15:00:42 julieng: because I can see only tidying up the code... Aug 16 15:06:59 embden_new[m]: Tidying up the code would be good. Aug 16 15:08:43 So the mentors can then start to review/evaluate it. Aug 16 15:10:02 The current state as I understand it is that you are now able to boot dom0. That is a good status. Are you mounting the filesystem on NFS or SD card. If it is just NFS it would be good to also check if MMC driver works well enough for an SD mount. Aug 16 15:12:48 So I would focus rest of the time on tidying up code and documentation. There have been many insights from Julien and Stefano on how Xen works. It would be good if your documentation pulls in as much of that as possible. I would like to see documentation that not only explains what your code does but why. Often the why is missing which makes the documentation only half useful. Aug 16 15:20:59 drhunter: no, I haven't used an fs, I boot via tftpboot Aug 16 15:21:08 ok, I understand Aug 16 15:21:26 with an initramfs? Aug 16 15:22:44 Julien, is there a prefered Xen filesystem? initramfs seems common (in Xilinx build for example). Is that by design? Aug 16 15:24:28 hm, I don't know. I built an busybox image and pass it as xen,linux-initrd device tree node Aug 16 15:26:10 I don't really understand the difference between initrd and initramfs drhunter Aug 16 15:28:53 For this discussion there isn't a relevant difference. Key point is that both are read only in the sense that they always boot from an unchanged filesystem. There is no persistence. (https://www.slax.org/blog/18432-initrd-versus-initramfs.html) Aug 16 15:30:40 drhunter[m]: There are no preferred filesystem. I tend use a mix of busybox/buildroot (in initramfs), centos/debian (NFS, SDCard). Aug 16 15:31:01 If there are any existing rootfs targeting x15 downloadable, then I would recommend to use it. Aug 16 15:31:30 The only difference for Xen should just be the kernel image that can be easily repleceable. Aug 16 15:32:34 there are several. In the past I only used initramfs/initrd for Xen on X15 as I couldn't get crossbar up to give me SD card interrupts. Aug 16 15:33:28 embden_new[m]: drhunter[m]: https://elinux.org/BeagleBoardDebian#BeagleBoard-X15 seems to provide some image for the board. Aug 16 15:35:29 I would follow the instructions at https://www.digikey.com/eewiki/display/linuxonarm/BeagleBoard-X15#BeagleBoard-X15-RootFileSystem which explains the few customisations that are needed to go from a standard debian to an x15 one. You may want to skip copying of kernel to FS Aug 16 15:42:45 julieng: drhunter ok, then tomorrow I will tidy up the code (make it at least a bit more appropriate in the first iteration) and after that I will start to document everything and load a real os with a patched kernel? Aug 16 17:59:25 oh, there is no need to patch the kernel, we work with xen **** ENDING LOGGING AT Sat Aug 17 03:00:04 2019