**** BEGIN LOGGING AT Sun Aug 20 02:59:56 2006 Aug 20 05:19:39 ~seen rwhitby Aug 20 05:19:54 rwhitby is currently on #oe (1h 11m 21s) #nslu2-linux (1h 11m 21s) #openjtag (1h 11m 21s) #openslug (1h 11m 21s), last said: 'ok, we'll get it sorted out sooner or later. I expect it's not time critical.'. Aug 20 05:20:11 beewoolie-afk: pong Aug 20 05:20:16 :-) Aug 20 05:20:44 The fellow with the fat slug really doesn't seem to get it, does he? Aug 20 05:21:27 rwhitby: do you know martin's handle? Aug 20 05:21:40 I'm continually amazed by the number of people doing the hardware side of fatslugs without being able to do the software side. Aug 20 05:21:48 beewoolie-afk: just pinged him Aug 20 05:21:52 k. Aug 20 05:22:02 tbm is his nick Aug 20 05:22:20 I wish one of them would do the hardware side for me. I've been looking for SDRAM chips without finding anything to make a 256MiB slug. Aug 20 05:22:35 I figure, if I'm going to go through the trouble, I'd like something we can use for building. Aug 20 05:22:47 tbm is over on OFTC in the debian channels, but no response to ping Aug 20 05:23:07 no worries. Aug 20 05:23:34 I hope you recognize that I'm not interested in being heavy about the tools angle. Whatever you want is fine with me. Aug 20 05:24:07 I'm very eager to get a debian slug that has an up-to-date kernel *and* NFS drivers. Aug 20 05:24:17 ...and an NPE driver. Aug 20 05:25:30 beewoolie-afk: yep, I think you, I and Martin are all pretty cool about most things, and happy to come to an agreement on whatever makes sense. Aug 20 05:26:57 I wish my real life job was not so hectic at the moment. I'm part of the management team of a new 30 person startup, and I'm doing both technical work and also being the IT department. Aug 20 05:27:20 So I haven't had a lot of time in NSLU2 world to do much more than reply to emails :-( Aug 20 05:27:51 Hated that. I hired a pal to handle the IT stuff at a startup where I was the CTO. He's now (5 years later) the big shot in the IT department of another company. Aug 20 05:29:05 Yeah, we're gonna hire someone else soon. Aug 20 05:29:49 So let's just summarize the current understanding, and we can email that to Martin if he can't be reached. Aug 20 05:30:02 How come you're dragging TBM through the muck, forcing him to edit slugimage? Aug 20 05:30:08 k. Aug 20 05:30:24 So we have the "stock" kernel and ramdisk partitions, each of which has a SerComm header, and are in fixed positions according to the Linksys RedBoot Aug 20 05:31:00 We plan to put APEX in the stock kernel partition (renaming that partition to "Loader" or something). Aug 20 05:31:05 check Aug 20 05:31:13 slugimage is muck? Aug 20 05:31:20 ;-) Aug 20 05:31:53 jacques: really, all I mean is that tbm has to make the patches and figure it all out. It isn't his program after all. Aug 20 05:32:14 beewoolie-afk: If I had to the time to be able to test changes, I'd be making them. Aug 20 05:32:23 rwhitby: That's only partially accurate. The partition sizes are going to change when we put APEX into flash. Aug 20 05:32:25 I wasn't aware changes were needed Aug 20 05:32:50 jacques: see the discussion on debian-arm mailing list Aug 20 05:32:53 jacques: IMHO, there shouldn't be a need for changes, but slugimage wasn't designed to be flexible enough for what we are doing. Aug 20 05:33:10 jacques: the changes are adding a second-stage bootloader function Aug 20 05:33:26 beewoolie-afk: nod - partition sizes change Aug 20 05:33:44 And coping with RedBoot limitations about the placement of data. Aug 20 05:34:10 So the new "Loader" partition (which is location in flash at the same place as the stock Kernel partition, but has a different size) also needs to have a SerComm header, cause Redboot looks for that to load it. Aug 20 05:34:36 Exactly. Aug 20 05:34:45 The Loader partition contains the SerComm header, and at +16 it contains APEX. Aug 20 05:34:58 RedBoot is being told that the kernel is very small and it's actually APEX instead of Linux. Aug 20 05:34:59 The data_length is the size of APEX+16. Aug 20 05:35:07 Nod Aug 20 05:35:11 Right. Aug 20 05:35:23 BTW, we *can* support fat slugs by doing a little magic there. Aug 20 05:35:27 I've thought a little about this. Aug 20 05:35:40 But Redboot also wants to load up a ramdisk, and it does that from a fixed location in flash, expecting to see a SerComm header at that location. Aug 20 05:36:04 If the APEX partition started with a jump to flash, it would allow APEX to re-init the SDRAM controller and even detect the amount of memory installed. Aug 20 05:36:14 If only I had a FAT slug to test this with.... Aug 20 05:36:37 If the kernel is larger than 1MB-(size of apex), then that Ramdisk sercomm header ends up in the middle of the new kernel partition that starts immediately after apex. Aug 20 05:36:59 yep. Aug 20 05:37:13 So we've taught Apex how to skip sections of data. Aug 20 05:37:36 The skip descriptors. Aug 20 05:37:42 And we put the skip descriptors in the FIS directory entry corresponding to the partition. Aug 20 05:38:21 As many as four per partition. Aug 20 05:38:41 we can have zero or more skip descriptors (up to an arbitrary current limit of 4), that do not need to be ordered (cause Apex will do that), but we're going to order them anyway as a tribute to Jon Postel Aug 20 05:39:01 :-) We live Jon. Aug 20 05:39:10 s/live/love/ Aug 20 05:39:10 beewoolie-afk meant: :-) We love Jon. Aug 20 05:39:18 To make Apex load commands nice and tidy, we make sure that Apex is asked to skip the SerComm header at the start of any partition that has one. Aug 20 05:40:00 And for a kernel partition that is larger than 1MB-(size of APEX), we also need to ask APEX to skip the "stock ramdisk partition sercomm header" which ends up smack in the middle of our kernel. Aug 20 05:40:34 So the "Kernel" partition FIS directory entry ends up with two skip descriptors, one for the first 16 bytes, and one for the 16 bytes at the location of the stock ramdisk partition in flash. Aug 20 05:41:38 Now, the Linux userland sees the kernel partition in it's entirety, including the sercomm header at the start and the one in the middle. Aug 20 05:41:46 Right. And these will tend not to every need to be updated unless someone decides to install a smaller kernel. Aug 20 05:41:57 the data_length for the new Kernel partition includes both those headers Aug 20 05:42:22 data_length doesn't reflect skips, ever. Aug 20 05:45:26 beewoolie-afk: one moment ... (phone call) Aug 20 05:45:31 np Aug 20 05:46:54 ok, bacj Aug 20 05:46:56 back Aug 20 05:47:49 if a partition has 16 bytes header, then 32 bytes data, then 16 bytes skip, then 32 bytes data, then data_length will be 96 bytes Aug 20 05:48:28 6*16 = 96, yeah Aug 20 05:48:45 data_length reflects the number of valid bytes that can be read from a /dev/mtdblock partition (which shows the raw bytes, and does no processing of headers or skips) Aug 20 05:50:01 So I think we need to say that if someone wants to change partition layout (i.e location or size of partitions), then they need to reflash a whole new 8MB image, including a new FIS directory which should include the appropriate skip descriptors for that new flash layout. Aug 20 05:51:36 So Apex, and any userland tools that process flash contents, see the FIS directory as read-only. the only execption might be the data_length field - I'm not sure whether that can be updated from userland or not. Aug 20 05:51:43 rwhitby: I agree. There may be situations where it is possible to reflash just a partition. The SW that handles this can verify that the FIS table is the same, and only reflash the pieces that need updating. Aug 20 05:51:53 agreed Aug 20 05:52:10 the normal kernel upgrade procedure will be such a situation. Aug 20 05:52:13 rwhitby: Not without rewriting the whole FIS partition table. Aug 20 05:52:18 Right. Aug 20 05:52:28 We *can* rewrite the partition table via the MTD driver. Aug 20 05:52:54 Worst case will be that the user needs to upslug2 the whole thing. Aug 20 05:52:54 In SlugOS, and Debian, yes. In Unslung, no. Aug 20 05:53:05 (although we could change that for Unslung if we needed to) Aug 20 05:53:14 Does unslung not have an MTD driver? Aug 20 05:53:28 it does, but it marks the last block as read only Aug 20 05:53:50 (there is a kernel config option to do that) Aug 20 05:54:24 Hmm. Not my favorite way to handle things. R/O should really mean R/O, not just "don't write this unless you know what you are doing". Aug 20 05:54:46 So a kernel upgrade to a new kernel which is larger than the partition size that we choose, will need a complete reflash. Aug 20 05:54:55 (not just the kernel partition) Aug 20 05:55:16 Yeah, but it cannot be done from unslung. I doubt that that will be a problem, tho. Aug 20 05:55:43 Right, we can exclude Unslung from this equation, cause the kernel is quite static. Aug 20 05:56:21 And as long as we make the default partitions of an appropriate size, we should be able to support kernel upgrades without partitions changes most of the time. Aug 20 05:56:26 and the size of Unslung is already to large to allow for even a 128KB apex. Aug 20 05:56:47 In fact, we can be comfortable making the APEX partition 128KiB. Aug 20 05:57:02 The next major version of Unslung would need to forget about flash and go straight to external disk. Aug 20 05:57:03 If we really had to grow it, the tools should do the right thing. Aug 20 05:57:10 agreed. Aug 20 05:57:19 and it would require a complete image reflash Aug 20 05:57:30 I read that uboot now has an OHCI driver. Perhaps I could add one as well. Aug 20 05:58:15 ok, so we now have covered the Loader partiton and the Kernel partition. Aug 20 05:58:33 what was the reason why we wanted Kernel1 and Kernel2 as well? Aug 20 05:58:47 That, I suppose, is up to you. Aug 20 05:59:01 I thought that it would make it easier to locate the sercomm headers. Aug 20 05:59:19 Hmm - yes. Aug 20 05:59:22 There isn't really a need for it if the tools know about the positions of the secromm headers. Aug 20 05:59:37 Also, I am not super excited about partitions having more than one view. Aug 20 06:00:11 I suppose we could just have kernel and kernel2 where kernel2 is the existing kernel2 and kernel is the existing kernel and we drop kernel1. Aug 20 06:00:19 I guess the tools can easily locate the SerComm header in the middle if the kernel is larger than 1MB-size of Loader partition Aug 20 06:00:59 In my mind, that could lead to confusion for folks who are not intimately familiar with the flash layout. Aug 20 06:01:13 The partitions are a cheap way to make our intentions clear. Aug 20 06:02:38 This isn't as necessary for the ramdisk since there is only the header at the start. Aug 20 06:03:49 yeah, it's not necessary at all for the ramdisk - it's format hasn't changed from stock, just it's location (which is irrelevant when reading/writing it via mtd) Aug 20 06:04:06 I'm just looking at slugimage, and wondering if we really need the kernel1 and kernel2. Aug 20 06:05:22 So I think for slugimage, it doesn't matter. It has enough info at it's disposal to put the mid-header in place easily. Aug 20 06:05:26 Originally, I proposed this design so that APEX could see both the kernel data, and the underlying structure of the kernel pieces. Aug 20 06:05:54 I'm now looking at our reflash script to see what that would entail ... Aug 20 06:05:59 I think that I can add a feature so that an unskipped presentation of the partition is available as well. Aug 20 06:08:01 ok, reflash can find the size of the loader partition easily. Aug 20 06:09:02 So, are you going to remove kernel1 and kernel2? Aug 20 06:10:42 thinking about it. what's your opinion? Aug 20 06:11:38 the reflash script would need to do some split calculations anyway (cause it can't assume a certain apex size, cause it might be a different bootloader in there) Aug 20 06:11:50 I don't really need them. What was motivating my original thoughts is that I'd like to be able to reflash partitions in a way that makes sense. Aug 20 06:12:29 I think I've figured out how to make this work without having to understand the sercomm headers. Aug 20 06:12:29 if I tell apex to write to a partition with skip descriptors, will it "Do The Right Thing" when writing the kernel into there? Aug 20 06:12:48 rwhitby: That's the hard part. It need to erase the block and then rewrite the skipped data. Aug 20 06:12:53 I think I can make it work. Aug 20 06:13:06 In other words, it isn't the writing that's the hard part, but the erasing. Aug 20 06:13:24 It turns out that this has been on my mind for some time. Aug 20 06:13:33 (i.e. will it begin writing the first byte of the input file to location +16, then keeping pulling bytes from the input file until it hits the mid-skip, then write the following byte from the input at mid-skip+16 ? Aug 20 06:13:42 I'd like to be able to erase the environment block even if there is data in that block that doesn't belong to APEX. Aug 20 06:14:02 and that could use the same skip mechanism ... Aug 20 06:14:16 rwhitby: That part is easy. While I haven't written the write function, I only need to copy the read function to make it work correctly. Aug 20 06:14:21 right. Aug 20 06:14:36 As I wrote, it's the erase that's the problem. Aug 20 06:16:05 right, cause you need to save the skipped parts and rewrite them back after the erase Aug 20 06:16:15 And, before I worry about that, I need to update the nor_flash driver to handle the funkiness in the SLUG where we have BE flash, but an LE program. Aug 20 06:16:41 I implemented reading some time ago, but I didn't vet the write code. Aug 20 06:17:05 So with regard to the kernel1,kernel2 question, I think that the use cases are as follows: Aug 20 06:17:37 1) User flashes a new kernel, using our provided script, which needs to add the header anyway, so it can "easily" put in a mid-skip too. Aug 20 06:17:50 2) User flashes a new complete image - slugimage does all the hard work. Aug 20 06:18:07 3) User disects a full upgrade image to pull out the kernel - slugimage does all the hard work. Aug 20 06:18:39 Does slugimage do flashing? Aug 20 06:18:54 no, it just prepares a 8MB file for flashing. Aug 20 06:19:31 The user may wish to update the ramdisk separately as well. Or even just APEX. Aug 20 06:19:33 4) User tries to read a kernel directly from mtd, and they get a sercomm header at the start which tells them that some foo is happening and they should use a provided script to do that. Aug 20 06:19:44 (yep, getting to those) Aug 20 06:19:45 :-) Aug 20 06:20:25 5) User flashes a new ramdisk, using our provided script, cause they would have had to add a sercomm header anyway in the stock situation Aug 20 06:20:42 6) User reads a ramdisk from mtd - same deal as the kernel. Aug 20 06:21:18 7) User flashes a new apex. They need to add the SerComm header - if they don't know this, then they shouldn't be messing with new bootloaders without our provided flashing script. Aug 20 06:21:42 8) User reads apex from mtd - same deal as kernel and ramdisk - they see the header first. Aug 20 06:22:31 9) User flashes a complete image with Upslug2, or SerComm upgrade utility, or even the stock web interface. Slugimage has prepared the 8MB image correctly for them. Aug 20 06:22:43 (oh, that's #2 again) Aug 20 06:23:15 10) User wants to do stuff at the apex command line. Aug 20 06:23:26 beewoolie-afk: (this is where you continue the use cases ....) Aug 20 06:23:36 :-) Aug 20 06:24:13 We presently have only one way to get data into the SLUG when APEX is running, xmodem. Aug 20 06:24:44 nod - in the future they will be able to tftp or something, over the network, right? Aug 20 06:25:00 So, 11) the user transfers in a new kernel and want to flash it. This is OK because the sercomm headers don't need to reflect the full size of the kernel. Aug 20 06:25:41 12) the user transfers a new ramdisk and wants to flash it. This is OK for the same reason. In both of these cases, the FIS driver will handle the skips. Aug 20 06:25:45 ah - cause Apex ignores the SerComm header, and only looks at the data_length Aug 20 06:25:55 Not yet. Aug 20 06:26:49 If APEX is going to honor the data_length, it will also have to rewrite the FIS partition table to accommodate the larger file. Aug 20 06:27:00 ok, apex knows the size of the kernel it's writing, so it can store that info whereveer we need it to go, but it won't be messing with the SerComm header area? Aug 20 06:27:09 Presently, APEX copies the whole partition which should always be OK. Aug 20 06:27:19 rwhitby: That's right. Aug 20 06:28:01 13) user downloads a new version of APEX and wants to write it to flash. This one is more complex because it requires that APEX also write a sercomm header with the appropriate length. Aug 20 06:28:18 In other words, I'll have to do something to make this work. Aug 20 06:29:00 how are we going to ensure that the stock ramdisk sercomm header contents are "reasonable" (i.e. they are a small value which will not cause redboot to overwrite the whole of memory)? Aug 20 06:29:02 I'm not terribly concerned about the APEX use cases since very few users will prefer to use APEX to write flash when the upslug2 and family are so much easier and don't require a console. Aug 20 06:29:22 as long as I don't need to change the sercomm headers, all should be fine. Aug 20 06:29:43 I can write a 5MiB ramdisk even though the sercomm header says that it is only 4MiB. Aug 20 06:30:07 As long as you don't care that the sercomm header could be wrong... Aug 20 06:30:44 If you insist on that, I'll have to add the capability to deal with writing sercomm headers. Aug 20 06:30:44 ok, so someone needs to flash a *full* 8MB image before they can start messing about on the apex command line, cause only the full 8MB image will have the correct FIS directory contents, skip descriptors, and "safe" kernel, mid-skip, and ramdisk sercomm headers Aug 20 06:31:41 An 8MB image which includes a second-stage loader will have a mid-skip sercomm header which has a length field of 1. Aug 20 06:32:26 Oh, I neglected to mention that. Aug 20 06:32:33 APEX has not idea how to write FIS partitions. Aug 20 06:32:35 as long as that is preserved by apex due to the skip descriptor covering it, then all is fine for that mid-skip sercomm header (by "mid-skip" I mean the one at the location of the stock redboot ramdisk partition, which is now ending up in the middle of the new larger kernel partition) Aug 20 06:33:18 oh, so Apex cannot update data_length when writing partitions from the apex command line? Aug 20 06:33:23 Yeah, as long as you don't care that the size reported in the mid kernel sercomm header or the ramdisk sercomm header is incorrect. Aug 20 06:33:33 Well, it could if I added that feature. Aug 20 06:33:50 well, I'm fine with the mid-skip header. still thinking about the kernel header and ramdisk header. Aug 20 06:33:52 Remember, that is an erase/rewrite operation. Aug 20 06:33:58 nod Aug 20 06:34:28 There isn't anything I don't know at that point, so it could make everything precisely correct. Aug 20 06:36:08 so for slugimage, it doesn't care if the flash header contents are incorrect, cause it only deals with full 8MB images. Aug 20 06:36:44 The only folks who would care would be the ones reading from flash and breaking apart the image, or pulling out the kernel. Aug 20 06:36:47 for example. Aug 20 06:36:59 also those reading from mtd Aug 20 06:37:50 That's what I mean. Aug 20 06:38:50 looking at upslug2 and reflash now ... Aug 20 06:40:19 ok, upslug2 only writes, so it will always overwrite the headers with correct and safe values Aug 20 06:43:50 So? Aug 20 06:44:07 reflash also rewrites the headers without reading the contents/ Aug 20 06:44:44 Well, that isn't the use-case I'm talking about. In fact, I'm thinking of something more unusual. Aug 20 06:45:06 so as long as our 8MB image that slugimage produces has the correct headers, then we don't care if someone messing on the apex command line writes a new kernel or ramdisk and that doesn't update the headers to the new correct value. Aug 20 06:45:07 Let's say that I wanted to generate a backup flash image. Aug 20 06:45:32 If I want to pull what currently exists in flash, I need to pull each piece individually. Aug 20 06:45:52 If the tools that do this care about the sercomm header being correct, then I'd better make sure it is. Aug 20 06:46:43 right - if someone pulls kernel or ramdisk from mtd, after writing them from apex command line, then the resulting file they get out of mtd will be inconsistent and will cause any tools to barf (cause the only way the tools know the correct length to read from mtd is by those sercomm headers) Aug 20 06:47:24 the problem will only occur if the header value is smaller than the real value. then the file written by the extraction tools will be truncated. Aug 20 06:47:26 rwhitby: Or, you could change the tools to check the FIS data_length first and use that if it isn't zero or 0xffffffff. Aug 20 06:50:08 ok, looks like the tools currently only use the partition size value, not the data_length. Aug 20 06:50:37 So they ignore the sercomm header altogether? Aug 20 06:51:16 but that's only cause we don't currently have a tool to reconstruct kernel or ramdisk from mtd. the current tools only write to mtd from files, or read files to create an 8MB image. Aug 20 06:51:41 BBIAM Aug 20 06:53:36 k. Aug 20 06:53:49 I think that the reconstruction case may be important. Aug 20 06:54:10 looking at kernel/drivers/mtd/redboot.c it seems that nothing uses the data_length field. Aug 20 06:54:23 If a user can write a portion of their boot strategy without writing a complete image, then it is possible that they have something they'd like to preserve. Aug 20 06:54:37 (i.e. you can't extract that information from /proc/mtd, or from an individual mtd partition) Aug 20 06:54:55 I don't doubt it, but remember that the MTD driver wouldn't ever have a need to recognize the data_length field. Aug 20 06:55:08 You can read the partition table. Aug 20 06:55:24 true Aug 20 07:01:01 ok, a stock slug has not FIS dir, so has not data_lenght. An Unslung slug has no data_length entries either. A SlugOS slug has no data_length entries. Aug 20 07:01:27 So why do we want data_lenght again? No current tools read or write it. Aug 20 07:02:55 data_length RedBoot is the standard way to describe the amount of data in a partition. Aug 20 07:03:19 IMHO, it is ill conceived, but they didn't consult me. Aug 20 07:03:40 If we were to support a method for specifying the amount of data in a partition, that would be it. Aug 20 07:04:12 I don't think this is crucial as the *only* optimization it will permit is limiting the amount of data that APEX copies from flash to SDRAM. Aug 20 07:04:15 Ho hum. Aug 20 07:04:43 The different between 6MiB and 4MiB isn't such so that you'd notice. Aug 20 07:05:02 right, and it causes extra work cause then tools need to write the FIS dir area when otherwise they wouldn't. Aug 20 07:05:15 I suggested this because slugimage was failing to properly extract data from an image. Aug 20 07:05:23 Indeed. Aug 20 07:06:25 What this suggests is that APEX is going to have to write the sercomm header if we want to have a chance of reading out the partition data after APEX writes it. Aug 20 07:06:31 slugimage writes out the whole partition, except for the cases where slugimage knows that a partition has a sercomm header, in which case it only writes out the amount identified by the sercomm header Aug 20 07:06:32 The only tricky one is writing the kernel. Aug 20 07:06:59 so are you happier to support sercomm headers instead of data_lenght? Aug 20 07:07:22 data_length is easier to support than sercomm headers. Aug 20 07:07:25 * rwhitby needs to slow down when typing "length" Aug 20 07:07:44 Think about the case where I write the kernel. Aug 20 07:07:45 but for data_length you need to rewrite fis dir, right? Aug 20 07:08:04 I need to cope with the fact that the sercomm header in the middle of the kernel needs to reflect the amount of data that follows. Aug 20 07:08:06 Blech. Aug 20 07:08:14 That's going to be a hack right there. Aug 20 07:08:31 Sure, I need to rewrite the FIS data, but at least that makes sense. Aug 20 07:08:32 ah, no. the header in the middle always contains a non-valid safe value. Aug 20 07:08:42 it does *not* need to reflect the amount following Aug 20 07:09:06 (it can always be a static safe value, irrespective of kernel length) Aug 20 07:09:07 Oh really? So is the sercomm header at the top of the kernel the size of the whole kernel? Aug 20 07:09:20 Well, it should be zero. Aug 20 07:09:38 The middle sercomm header that is. Aug 20 07:09:58 yes, it should logically be zero - I just need to check that "zero" doesn't mean "overwrite the whole of memory". Aug 20 07:10:25 Now, for the sercomm header at the top of the kernel ... Aug 20 07:10:47 yes, I think that one should contain the whole length of the kernel in that partition. Aug 20 07:11:04 Remember, the way this started, these two sercomm headers were the correct lengths of the contents of each smaller partition. Aug 20 07:11:22 yes, but I think we've determined that we no longer need two partitions. Aug 20 07:11:22 And it excludes the skip in the middle? Aug 20 07:11:34 agreed. Aug 20 07:12:06 when someone reads from mtd, and uses the sercomm header at the start to identify the length of data, they will write out the complete kernel, including writing out the 16 bytes in the middle. Aug 20 07:12:18 the sercomm header lenght value is inclusive of the middle skip Aug 20 07:12:44 That means that the kernel they write out won't be valid. Aug 20 07:12:59 the tool which does that, needs to also know (from the existence and size of the loader partition) when to remove the middle skip Aug 20 07:13:08 I thought that you'd want to make sure the kernel was valid at all times. Aug 20 07:13:32 Oh, I see what you mean. Aug 20 07:14:15 That also makes it hard for APEX to understand. Aug 20 07:14:24 I think in the course of this discussion, I've been convinced that use cases for extracting a kernel from mtd mean that someone should be using a tool, and that tool should know about nslu2 kernel formats for second stage loader situations. Aug 20 07:14:58 what makes what hard for apex to understand? Aug 20 07:15:09 If I tell it to write to the partition and ignore the skips, it will have to write both sercomm headers. If I tell it to write and use the skips, it will have to go back and rewrite the first sercomm header. Aug 20 07:15:59 right - you either need to pre-massage the data, or pre-calculate the sercomm header length Aug 20 07:16:09 you can't do it all on-the-fly Aug 20 07:16:15 That isn't what I mean. Aug 20 07:16:44 ok, I'm not following then ... Aug 20 07:17:27 It either needs to know that there is a special sercomm header inserted into the middle of the kernel, or it needs to rewrite the first block with the correct first sercomm header. I cannot write the first header at the time I write the kernel unless I know to insert the sercomm header in the middle as well. Aug 20 07:17:58 (apex only ever has to write a calculated value into the first sercomm header - the mid-skip header always has static contents) Aug 20 07:17:59 Note that writing the ramdisk doesn't have this problem, nor does writing APEX. Aug 20 07:18:22 rwhitby: right, but apex doesn't know about the details of booting on the slug. Aug 20 07:18:27 It's a general purpose program. Aug 20 07:18:50 right, for the kernel, you need to know the start address and length of the input file, then determine if that is going to cross the magical mid-skip, and add 16 to the value you were going to write to the sercom header if it does Aug 20 07:18:55 So, if I download a clean kernel that boots, I need a way to write the to flash in a way that preserves how it works. Aug 20 07:19:10 Right. I don't want any 'magical' code in APEX. Aug 20 07:19:31 understood. let's discuss how magical it needs to be Aug 20 07:19:46 apex will know there is a skip at the start and a skip in the middle. that's general purpose. Aug 20 07:19:55 so the magical thing is writing the header. Aug 20 07:20:01 Not really. Aug 20 07:20:26 I could add a switch to the copy command that says 'add a sercomm header to the start of this write sequence'. Aug 20 07:20:33 That's not too bad. Aug 20 07:20:47 but the calculation of the size does not depend on nslu2 specifics, only on the existence and processing of skip descriptors. Aug 20 07:21:15 so it's a "add a length header" switch, not a "add nslu2 specific stuff" switch. Aug 20 07:21:23 I'm ignoring the fact that the length in the sercomm header includes the middle skip. Aug 20 07:21:30 Right. Aug 20 07:21:50 yes, you don't need to know about the middle skip, cause it has a descriptor and you just preserve the contents. Aug 20 07:21:54 The problem is really that I need to let it know that it is going to write to a partition and ignore the first skip, but not others. Aug 20 07:22:11 why doesn't it just ignore all skips? Aug 20 07:22:29 because if it ignores the middle skip, it won't preserve the contents of that area. Aug 20 07:22:50 ok, I think I see what you're getting at Aug 20 07:22:51 In other words, the middle skip is special. Aug 20 07:23:04 for the general case, you always ignore all skips and preserve their contents. Aug 20 07:23:13 See, I was planning to say something like this: Aug 20 07:23:46 then if you are told to write a length header, then that overrides the preservation of the first 16 bytes, even though the skip descriptor told you to preserve them Aug 20 07:23:47 copy -S $kernel fis://noskip@kernel Aug 20 07:23:56 Right. Aug 20 07:24:39 I don't exactly know what I'd add to tell the FIS driver to ignore skips, but it'll be something in the region string. Aug 20 07:24:39 the input file has no headers anywhere (neither at the start or in the middle), right? Aug 20 07:24:45 Right. Aug 20 07:24:47 It's a real kernel. Aug 20 07:24:50 right Aug 20 07:25:02 User compiles kernel, downloads it, write it to flash. Aug 20 07:25:04 so you write it out according to the skip descriptors Aug 20 07:25:21 and if asked to, you write a length header. Aug 20 07:25:26 Also, keep in mind that this isn't likely now, but it will be when APEX has a telnet server and NPE support. Aug 20 07:25:36 That's the -S switch. Aug 20 07:25:47 (which you can pre-calculate based on the skip descriptors and length of input file) Aug 20 07:26:15 it seems doable to me ... (but I don't know the internals of apex) Aug 20 07:26:25 I haven't figured out how, but that would be reasonable. Aug 20 07:26:30 and it's a general purpose 16 byte length header :-) Aug 20 07:26:34 This shouldn't be about APEX internals. Aug 20 07:26:43 -S means write a sercomm header. Aug 20 07:26:50 (i.e. -L, rather than -S) Aug 20 07:27:04 Sure -L. I don't really care about the mnemonic. Aug 20 07:27:12 or even -L 16, .... Aug 20 07:27:21 Hmm. Unlikely. Aug 20 07:27:28 then you could do -L 4 for another use. Aug 20 07:27:32 what exactly does that mean? Aug 20 07:27:48 write a length header, padding to N bytes Aug 20 07:27:53 I'd rather just make it write the sercomm header as that is likely to be the only kind. Aug 20 07:28:04 hence, -S. Aug 20 07:28:16 ah - the SysConf partition has a length header too, which is only 4 bytes I think. Aug 20 07:28:19 But, that isn't the problem. Aug 20 07:28:58 The problem is explaining to the FIS driver that it is going to write while ignoring one skip and not another. Aug 20 07:29:00 yes, SysConf length header is 4 bytes. Aug 20 07:29:06 I believe you. Aug 20 07:29:14 So a -L switch would have multiple uses ... Aug 20 07:29:52 I can imagine lots of variations, but that doesn't mean I want to support them all. Aug 20 07:30:01 can you do a single erase, then an FIS write for the header, then an FIS write for the kernel. The kernel write will skip the header, so all should be sweet. Aug 20 07:30:18 (i.e. you don't need another erase in-between) Aug 20 07:30:47 ah, but you need to tell the FIS driver to ignore skips for the first write. Aug 20 07:30:50 You could, but the user wouldn't be able to do Aug 20 07:30:55 copy $kernel fis://kernel Aug 20 07:31:51 I suppose I could do something with the fis: driver's parsing of the region to tell it that we are going to ignore any skip at the stat of the partition, and let all of the rest remain. Aug 20 07:31:56 It's kinda hokey. Aug 20 07:31:56 So "copy -L 16 $kernel fis::/kernel" would do an erase, then write the pre-calculated header (based on $kernel size and skip descriptors) using a skip-ignore write, then write $kernel using a skip-understanding write Aug 20 07:32:23 no, you only need a switch on the fis driver to ignore all skips or comprehend all skips Aug 20 07:32:29 That's a little too much hidden functionality. Aug 20 07:32:44 but the -L switch exposes it Aug 20 07:33:06 No, it doesn't. The FIS driver doesn't see the -L switch. That's a copy switch. Aug 20 07:33:31 ah, ok. I see what you mean. Aug 20 07:33:55 so a new command is required. Aug 20 07:34:08 That won't work either. Aug 20 07:34:14 write_header -L 16 $kernel fis::/kernel Aug 20 07:34:38 Oh, I see what you mean. Aug 20 07:34:53 does the copy do an erase, or does the user erase ? Aug 20 07:35:32 Well, remember that we need to be able to ask the FIS driver "how much do I add to the length if I include skips". Aug 20 07:35:34 the most efficient steps would be "erase fis::/kernel, write_header ...., copy ..." Aug 20 07:36:11 yes - that's a new interrogation function to the FIS driver which the write_header calls Aug 20 07:36:13 That's how it would work. erase, write header, copy. Aug 20 07:36:27 But you've told it to ignore skips. Aug 20 07:36:39 so, the amount of data to add to the length would be zero. Aug 20 07:37:00 I'm not sure there is a way to do this without adding some magic in the skips. Aug 20 07:37:07 which we could do. Aug 20 07:37:21 write_header would ask the fis driver what the total skip length is (with skips enabled), then do the write (with skips disabled)( Aug 20 07:37:31 We could flag a skip as permanent. Aug 20 07:37:57 But the length with skips enabled includes the first skip which makes the length wrong. Aug 20 07:38:40 * rwhitby has forgotten whether sercomm header lengths include the 16 bytes of the header Aug 20 07:38:43 Or we could flag a skip as a length header skip. Aug 20 07:38:52 They don't. Aug 20 07:39:01 I know this from usage. Aug 20 07:39:44 right - slugimage agrees. Aug 20 07:40:07 I'd like to table this for the time being. Aug 20 07:40:22 It isn't going to impact our usage in the near term. Aug 20 07:40:35 And it won't really come into play until we have telnet in APEX. Aug 20 07:41:18 I think I like the write_header command, but it may be unnecessary if I find a way to properly calculate the data length for the header. Aug 20 07:41:21 so for the moment, someone messing with the apex command line will cause sercomm headers to be inconsistent, and they should expect that inconsistency if they use mtd to read the partition. Aug 20 07:41:32 Indeed. Aug 20 07:41:47 We will document how the sercomm headers must conform. Aug 20 07:41:57 and we will drop implementation of data_length. Aug 20 07:42:09 and have only a single kernel partition. Aug 20 07:42:25 It will be my worry to make sure that APEX manages the sercomm headers properly. Aug 20 07:42:33 Yes, and make only one kernel partition Aug 20 07:42:51 does the apex command line user have a way to write a manually-calculated value into the header location in the mean time? Aug 20 07:43:28 (i.e. using the NOR driver instead of the FIS driver?) Aug 20 07:43:42 Yeah, but it isn't complete right now. Aug 20 07:43:55 There is a FILL command. it doesn't allow for setting of the word size. Aug 20 07:44:07 I think it only writes bytes at the moment. Aug 20 07:44:21 ok, so they need to write four bytes manually. Aug 20 07:44:28 I believe so. Aug 20 07:44:41 That isn't a well used command, so it suffers from incomplete implementation. Aug 20 07:44:54 ok, that's good enough as a power user apex command line workaround Aug 20 07:45:05 In fact, I need a store command so that I can write arbitrary strings of bytes to memory. Aug 20 07:45:20 Sure, but I really doubt anyone would use it. Aug 20 07:45:39 The utilities are going to be much higher functioning. Aug 20 07:45:42 Only those who are anal about Sercomm headers :-) Aug 20 07:45:59 (and use the apex command line to write partitions) Aug 20 07:46:20 ok, time for another summary of the story so far ... Aug 20 07:46:39 We have a Loader partition, then a Kernel partition, then a Ramdisk partion Aug 20 07:46:40 ..and after I fix the APEX nor flash write code. Aug 20 07:47:28 All three have a SerComm header at the start, which is kept correct by all userland utilities, including slugimage, upslug2, reflash, and any new ones we write. Aug 20 07:48:11 if the kernel is larger than (1MB - size of Loader partition), then it also has a mid-skip header, whose contents is a fixed safe value (either 0 or 1, we need to check) Aug 20 07:48:29 The ramdisk always only has a single sercomm header at the start. Aug 20 07:48:39 Probably should be 4 instead of 1. Aug 20 07:49:01 All sercomm headers (both start and mid) are documented in skip descriptors in the corresponding FIS directory entry Aug 20 07:49:07 nod on the 4 Aug 20 07:49:48 Sounds right. Aug 20 07:49:50 FIS directory entries are read-only as far as Apex and any userland utilities are concerned. Aug 20 07:50:04 The only way to update an FIS directory entry is to write a completely new 8MB flash image. Aug 20 07:50:12 As long as the partitions don't need to be resized. Aug 20 07:50:22