**** BEGIN LOGGING AT Fri Aug 05 02:59:58 2016 Aug 05 07:02:11 Hello everyone Aug 05 07:05:31 I would like to know if someone knows something about binary diff algorithms :). I say this because I am struggling with bsdiff ( suffix sorting ) and xdelta ( VCDIFF ) in order to get small deltas to update a firmware for a ARM CORTEX M0+ Aug 05 07:05:37 ^^ Aug 05 07:07:08 I just want some advise, because I am a bit lost after trying this, concretely bsdiff has totally beaten me down, because this should be the best regarding to generate deltas from two binaries but... :S Aug 05 07:10:49 so, why bsdiff without using compression is generating my big deltas ( the same size than the original file ) when the delta should only contain the changes from the old to the new file? Aug 05 07:47:02 I am trying to use bsdiff ( a suffix sorting algorithm) for generating a patch wiith the diferences between two files: an old file and the new file with new features. The problem is that I am getting patches with the same size than old, and I shouldn't be like this. Another option was to use VCDIFF, another algorithm in order to calculate binary differences. Any advice? Aug 05 08:05:01 fulgor3: don't know, double check how you invoke it perhaps? Aug 05 08:05:48 fulgor3: for what it is worth, bsdiff patches are generally smaller then vcdiff, but take longer to create... Aug 05 08:16:14 hello jeroen__ Aug 05 08:19:22 yes, that is why I decided to use bsdiff, so first I test it from its creator website, and I got good results (they seemed to be, as you will see now), but I felt curious and tried to remove the compression (bzip2 is not suitable for me because the little memory I have for this purpose), and I was really surprised when I saw that the incompressed patch was bigger than the files compared. So I looked for a bsdiff implementation without compres Aug 05 08:20:32 And the problem could be there, so I tried minibsdiff (https://github.com/thoughtpolice/minibsdiff), which had implemented bsdiff without compression and... I got big patches also :( Aug 05 08:21:25 so... this project creator and me have made a mistake or I am missing something important Aug 05 08:21:48 but OTA/FOTA is the embedded present/future!!! I need this jeje Aug 05 08:22:10 fulgor3: right, it needs the compression, there are huge amounts of zeros in it otherwise! Aug 05 08:22:37 so it always complete the patch using zeroes? Aug 05 08:26:15 anyway, there will be a moment in which I should decompress the delta and I will get a huge file, because I guess there is no possible to decompress in blocks, or yes :S. The problem is that I have like 8 or 16 KB for storing the patch in FLASH, so I cannot decompress it and deal with a 200KB patch file Aug 05 08:28:19 But I guess that maybe in the compression tool source code it could be modified Aug 05 08:34:14 yes, if I recall correctly... it will apply similar changes to multiple location (like a function moved) all its callees get patch with the same delte. the uncompressed representation just uses 0 between the locations.. (or whatever value doesn't change them).. Aug 05 08:34:46 in the patch file itself these get compressed.. Aug 05 08:38:09 umh... thanks then, I think I will have to change bzip2 to lwz for example and modify the patcher in order to do it work in blocks and a few memory space Aug 05 08:38:26 lzw* Aug 05 08:40:02 fulgor3: mind it, bspatch itself is memory hungry as well, it has the whole file to patch in memory.. Aug 05 08:41:29 "bspatch uses memory equal to the size of plus the size of , but can tolerate a very small working set without a dramatic loss of performance." Aug 05 08:52:26 yes, jeroen__, but won't be possible to patch while I am decompressing the blocks which forms the modifications of the oldfile to build the newone? Aug 05 08:53:00 I don't understand well what do you mean with the "can tolerate a very small working set" Aug 05 08:53:12 fulgor3: yes and no, by default no, assuming you are embedded it will destroy your nand / emmc Aug 05 08:53:44 fulgor3: me neither it is a quote from the man page Aug 05 08:54:40 fulgor3: the problem is it can rewrite the whole image per "chuck", so you get huge amounts of writes to the storage Aug 05 08:54:54 let me dig up some mail... Aug 05 08:55:06 destroy? :O. If I use a buffer with at least the same size of the amout of bytes of a FLASH page, it should't affect the FLASH I guess Aug 05 08:56:55 no no, but I can buffer some, maybe 25KB, and in ARM CORTEX M0+ I have, the page deletion has a 250KB granularity, but I can write 2 bytes if the FLASH is empty (0xFF), so the FLASH shouldn't be affected if I manage this well Aug 05 08:57:27 I already perform a DFU update using USB, working very well, but the FOTA due the use of those algorithms is quite more complicated Aug 05 08:58:49 fulgor3: and how are you going to insert a new range in a existing file? Aug 05 09:01:02 ok, after reading "Naïve Differences of Excutable Code", that is a paper written by Colin Percival (bsdiff creator), he explains at the last paragraph of the 2nd point the following: Aug 05 09:01:06 The patch file is then constructed of three parts: First, a Aug 05 09:01:06 control file containing ADD and INSERT instructions; Aug 05 09:01:06 second, a ‘difference’ file, containing the bytewise differences Aug 05 09:01:06 of the approximate matches; and third, an ‘extra’ Aug 05 09:01:06 file, containing the bytes which were not part of an Aug 05 09:01:07 approximate match. Each ADD instruction specifies an Aug 05 09:01:07 offset in the old file and a length; the appropriate number Aug 05 09:01:08 of bytes are read from the old file and added to the same Aug 05 09:01:08 number of bytes from the difference file. INSERT instructions Aug 05 09:01:09 merely specify a length; the specified number Aug 05 09:01:09 of bytes is read from the extra file. While these three files Aug 05 09:01:10 together are slightly larger than the original target file, Aug 05 09:01:10 the control and difference files are highly compressible; Aug 05 09:01:11 in particular, bzip2 tends to perform remarkably well Aug 05 09:01:47 so... I should read those HEADERS in order to know what should I move of my current old fw Aug 05 09:02:41 you might like this as well https://www.usenix.org/legacy/event/usenix03/tech/freenix03/full_papers/rasch/rasch.pdf Aug 05 09:02:46 concretelly the INSTERTS, which will require to move the code Aug 05 09:03:27 fulgor3: exactly and on linux that means copying the whole file or hacking around... Aug 05 09:03:39 (well most of the file) Aug 05 09:04:31 so hence you get the massive writes to the storage I mentioned before... Aug 05 09:05:10 fulgor3: http://man7.org/linux/man-pages/man2/fallocate.2.html Aug 05 09:06:02 depending on the size it can be suitable... if it isn't...how other perform FOTA? :( Aug 05 09:06:05 fulgor3: but likely means patching a kernel... Aug 05 09:06:59 because I don't think they read all the file without saving anything only for reading those headers, move all the old fw and later on do again the read but this time doing the modifications... Aug 05 09:07:05 but it is like a mess Aug 05 09:07:24 yes... I cannot compare this with an ARM CORTEX M0+ Aug 05 09:08:04 whenever I google with "embedded", the results are related to embedded linux or mobile phones... which are more powerful than my device :( Aug 05 09:08:12 I look for ultra low power of course Aug 05 09:09:52 fulgor3: I have no idea how people do it in general, we decided to just send a complete copy and put this on the todo list.. ;) Aug 05 09:13:27 fulgor3: another one, http://hssl.cs.jhu.edu/papers/burns_podc98.pdf Aug 05 09:18:54 jaja, yes... it usually is like this. I usuarlly sent all the file using single or dual bank depending on the flash available Aug 05 09:19:02 thanksss, I check it ASAP Aug 05 09:23:20 fulgor3: https://clearlinux.org/features/software-update perhaps? Aug 05 09:34:25 fulgor3: perhaps this might help, https://github.com/clearlinux/bsdiff/blob/master/src/patch.c#L428 .. BSDIFF_ENC_ZEROS it might solve the original problem you faced... Aug 05 09:35:03 jeroen__ I think this is more focused on performing OS update I think, so for sure it will be heavy jeje. They don't care about our little embedded devices Aug 05 09:35:13 umh... let's see Aug 05 09:38:50 but I will need to dig in more in WHAT this definition exactly means Aug 05 09:39:29 clear's software updater is very heavy Aug 05 09:45:37 yes, it seems to be like this Aug 05 09:46:07 minibsdiff was the ligher I found to generate deltas Aug 05 09:46:24 and lzw or other variants for compressing Aug 05 09:46:54 but I should extract just the patcher to the the FOTA in order to get an smaller FOTA application Aug 05 09:47:50 rburton do you know any light binary diff algorithm or something? Aug 05 09:51:50 if I have a package name, how do get the recipe name which declared it in a bbclass? Aug 05 09:52:05 ...I get ... Aug 05 10:21:26 jeroen__ yocto? I don't know about bbclass :( Aug 05 10:21:42 rburton, seen any lions lately? Aug 05 10:27:51 this will work I guess oe.packagedata.read_subpkgdata(rdepend, d) Aug 05 10:39:31 Crofton|work: it was a lynx Aug 05 10:40:15 LION! Aug 05 10:40:18 lol Aug 05 10:45:55 yeah the lion thing is surely nonsense Aug 05 10:46:02 £100 says it was a sheep Aug 05 10:50:04 a sheep in lions clothing Aug 05 11:16:20 jeroen__ regarding to the delta generation for a FOTA, I recommend you this http://michael.dipperstein.com/delta/index.html. Maybe you can find something useful there, and it has examples in ANSI C. Aug 05 11:42:53 anyone a better idea then using oe.packagedata.read_pkgdata to get the recipename of package, since it doesn't seem to work..? Aug 05 12:00:03 found it, oe.packagedata.recipename(dep, d) Aug 05 13:03:50 So "devpyshell" is also taking up 100% cpu, python3 seems to spin on select() with a zero timeout **** ENDING LOGGING AT Sat Aug 06 02:59:58 2016