**** BEGIN LOGGING AT Wed Jun 07 03:00:03 2017
Jun 07 03:12:21 <Wikiwide> How large is your el-v1.db? Mine is 50MB+
Jun 07 03:25:51 <Maxdamantus> Mm .. mine's only 833 K
Jun 07 03:31:45 <Wikiwide> Do you use telepathy-idle for IRC? Or something else?
Jun 07 03:32:14 <Wikiwide> Because that's the primary source for "events" in my el-v1.db
Jun 07 03:44:49 <Maxdamantus> No, I just ssh into my computer at home, which is always running irssi in tmux.
Jun 07 03:51:47 <Wikiwide> Fun :-)  Anyway, telepathy-idle should speak with NickServ more, because the way it handles nicks makes it difficult to work across nick changes, as well as integrate with Contacts.
Jun 07 04:05:34 <Maxdamantus> O_o
Jun 07 04:06:33 <Maxdamantus> so it messages nickserv `info ..` for lots of nicks and tries to parse out the account name?
Jun 07 04:07:03 <Wikiwide> Nay, it doesn't speak with NickServ at all atm.
Jun 07 04:07:14 * Maxdamantus imagines the WHOIS information on freenode would be better to deal with .. in either case, the logic is kind of network-specific.
Jun 07 04:07:58 <Maxdamantus> Oh, you're saying it doesn't spak with nickserv, but should.
Jun 07 04:08:18 <Wikiwide> So I have to keep "svetlana", "svetlana-gry", "gry", "gry_" and "evilgry" in Address Book, and it isn't perfect at all.
Jun 07 04:09:11 <Wikiwide> Because 1) Gry may use yet-another-nick; 2) somebody else can use one of her nicks, especially not protected ones (like gry_ )
Jun 07 04:10:40 <Wikiwide> Also, it doesn't notice servers. It thinks that gry on freenode, moznet and whatever server has to be the same person, regarding integration with Contacts at least.
Jun 07 05:23:59 <Maxdamantus> Presumably another good way of avoiding corruption would be to just append messages to log files.
Jun 07 05:25:45 <Maxdamantus> Hmm .. just wondering, is your filesystem mounted with data=ordered? Mine is .. presumably there isn't a practical way for sqlite to maintain consistency without it.
Jun 07 05:26:10 <Wikiwide> How to check?
Jun 07 05:26:19 <Maxdamantus> cat /proc/mounts
Jun 07 05:26:34 <Maxdamantus> Look for the line mentioning your main ext3 mount
Jun 07 05:26:54 <Maxdamantus> Probably /home
Jun 07 05:27:20 <Maxdamantus> /dev/mmcblk0p2 /home ext4 rw,noatime,nodiratime,commit=20,barrier=1,data=ordered 0 0
Jun 07 05:27:37 <KotCzarny> or just finding a way to minimize that db
Jun 07 05:27:57 <Wikiwide> /dev/mmcblk0p2 /home ext3 rw,noatime,nodiratime,errors=continue,commit=1,data=writeback 0 0
Jun 07 05:28:28 <Maxdamantus> Mm .. I suspect that should really be ordered.
Jun 07 05:28:28 <Wikiwide> How do I change from writeback to ordered? And what's the difference?
Jun 07 05:28:49 * Maxdamantus wonders if that's something he changed at about the time he switched to ext4
Jun 07 05:29:43 <Maxdamantus> writeback refers to a writeback cache, the important part is that there isn't much guarantee about what order stuff within files is written in.
Jun 07 05:29:50 <Wikiwide> 21.2011.38-1Tmaemo11+thumb0
Jun 07 05:30:42 <Maxdamantus> with data=ordered, if a process performs two writes to a file and the system crashes, it won'at be left in a state where nthe second write seemed to happen but the first didn't.
Jun 07 05:31:19 <Wikiwide> /me just wants to check whether it is typical for CSSU-thumb to have data=writeback... Or is it just-me problem?
Jun 07 05:31:54 <Maxdamantus> I think it's probably typical. I probably would have changed it myself if it were the default.
Jun 07 05:32:47 <Maxdamantus> anyway, to change it you need to modify some scriupt in /etc so it doesn't generate fstab on boot, then you can just modify the fstab
Jun 07 05:33:27 <Maxdamantus> or else modify the script so it adds data=ordered
Jun 07 05:33:38 <KotCzarny> mine has writeback too
Jun 07 05:33:41 <KotCzarny> (cssu-thumb)
Jun 07 05:33:57 <Wikiwide> Wouldn't it be better to modify script to generate data=ordered fstab ?
Jun 07 05:34:50 <Maxdamantus> /etc/default/mount-opts
Jun 07 05:35:39 <Maxdamantus> Maybe. It seems kind of silly to me generating the same fstab every boot though anyway.
Jun 07 05:36:31 <Wikiwide> In case it gets damaged somehow?
Jun 07 05:36:39 <KotCzarny> o.O
Jun 07 05:37:23 <KotCzarny> for dynamic mounts it should be written as an udev rule
Jun 07 05:37:30 <KotCzarny> not rewriting fstab O.o
Jun 07 05:38:26 <Wikiwide> Is /home a dynamic mount ?
Jun 07 05:39:40 <KotCzarny> unless that script only runs once
Jun 07 05:41:02 <Maxdamantus> Actually, I'm not even mounting based on that fstab .. I have a script that mounts it  before preinit
Jun 07 05:42:09 <Wikiwide> No idea. /etc/default/mount-opts ext3_opts="data=writeback"
Jun 07 05:47:22 <Maxdamantus> Eventually we should all just be using CoW filesystems, which work perfectly well with writeback caching (as long as there's an underlying barrier mechanism)
Jun 07 05:48:08 <Maxdamantus> otherwise you need to do journalling, but noone uses data journalling on ext (though it is supported)
Jun 07 05:48:35 <Maxdamantus> (journalling = write everything twice)
Jun 07 05:53:39 <Wikiwide> What is CoW?
Jun 07 05:53:56 <KotCzarny> copy-on-write
Jun 07 05:59:14 <kerio> sadly everyone seems to be fucking it up, as of late :<
Jun 07 05:59:29 <kerio> apple's brand new apfs has no data checksums
Jun 07 06:00:24 <kerio> how do you go "yep, this zfs thing is good, i'ma copy it but i'll remove the best feature"
Jun 07 06:01:09 <KotCzarny> maybe its patented :P
Jun 07 06:28:13 <Maxdamantus> Does their apfs still do unicode normalisation in the filesystem?
Jun 07 06:28:37 <Maxdamantus> Oh good, it doesn't.
Jun 07 06:28:51 <Maxdamantus> So I guess it's no longer worse than Microsoft.
Jun 07 06:37:50 <kerio> Maxdamantus: no but they moved unicode normalization in half the APIs
Jun 07 06:38:03 <kerio> so you can have two files with names that normalize to the same thing
Jun 07 06:38:21 <kerio> but you can only ever see and access one in guis
Jun 07 06:40:52 <KotCzarny> whoever thought that using anything else than ascii for file names should sort files in hell
Jun 07 07:02:03 <Wikiwide> Unicode is Unicode. At long as everybody uses the same encoding, it shouldn't a problem. Anyway, file name doesn't matter for tracker-ed files.
Jun 07 07:02:23 <KotCzarny> sure, and then are no-printable characters
Jun 07 07:02:32 <KotCzarny> and whole lots of problems
Jun 07 07:13:18 <Wikiwide> Hmm, I understand that most programs don't treat filenames as "text document" aka "let's change text direction, breakline, and so on", but it should be doable.
Jun 07 07:13:28 <Wikiwide> Granted, non-standard.
Jun 07 07:17:35 <Maxdamantus> Well, text documents aren't stored as some magical normalised unicode object in the filesystem.
Jun 07 07:17:42 <Maxdamantus> Whatever happens, neither should filenames.
Jun 07 07:18:05 <Maxdamantus> They should both be treated the same way by the filesystem: a sequence of bytes.
Jun 07 07:18:13 <KotCzarny> wikiwide: i'm not saying unicode is bad, i'm saying it shouldnt be used for file names
Jun 07 07:18:33 <Maxdamantus> Something outside of the kernel is responsible for interpreting the bytes in some way so as to have various text directions.
Jun 07 07:18:41 <KotCzarny> unless you want confusion, collisions and errors
Jun 07 07:18:50 <Maxdamantus> all the kernel has is byte 0, byte 1, byte 2, .. byte n
Jun 07 07:20:23 <Maxdamantus> I think unicode is perfectly fine for filenames, just make sure the assumptions about unicode stay in userspace, preferably at a high level that only graphical applications interact with.
Jun 07 07:21:45 <Maxdamantus> So if you call `open(2)` on the string ".GIT", you shouldn't be able to read that file back by calling `open(2)` on the string ".git"
Jun 07 07:22:55 <Maxdamantus> Otherwise this happens: https://github.com/blog/1938-git-client-vulnerability-announced
Jun 07 07:23:45 <Maxdamantus> git shouldn't be responsible for handling arbitrary normalisations, since it's not trying to render text.
Jun 07 07:24:03 <Maxdamantus> All it should care about is arrays of bytes, possibly terminated with 0
Jun 07 07:26:58 <Maxdamantus> I've been wondering how OS X behaves regarding unicode normalisation across OS versions.
Jun 07 07:27:12 <Maxdamantus> because the effects of unicode normalisation change across revisions of unicode.
Jun 07 07:28:04 <Maxdamantus> So what happens if you transfer files back and forth between systems that do and don't use Unicode 9.0?
Jun 07 07:29:33 <Maxdamantus> or worse, if you plug a harddrive into various versions of OS X?
Jun 07 07:30:24 <Maxdamantus> since then you don't really get a chance to do the translations at the protocol level (which people probably don't bother doing anyway, like the CRLF crap in Windows)
Jun 07 07:40:43 <Maxdamantus> (note on the CRLF point: protocols like FTP have a "text" mode that you're meant to use when transferring text files, which will correctly translate to/from things like CRLF form; the idea is that CRLF is just a way of encoding text, so as long as you transcode it every time you move it to a different system, there shouldn't be a problem)
Jun 07 07:42:21 <Maxdamantus> But that's just silly. It should just be stored in the simplest way possible.
Jun 07 07:42:35 <Maxdamantus> s/possible/plausible/
Jun 07 07:42:35 <infobot> Maxdamantus meant: But that's just silly. It should just be stored in the simplest way plausible.
Jun 07 09:45:44 <auenfx4> you mean emoji in file names isnt a good thing? :P
Jun 07 09:46:28 <KotCzarny> we need emojis for basic unix tools!
Jun 07 09:47:23 <auenfx4> is there a programming language that is entirely in emojis?
Jun 07 09:47:56 <auenfx4> http://www.emojicode.org/
Jun 07 09:47:58 <auenfx4> aparently ues
Jun 07 09:48:09 <auenfx4> but, i'd probably use lolcode over it still
Jun 07 09:49:03 <KotCzarny> as if there would be a difference
Jun 07 09:53:26 <L29Ah_> what about lolicode
Jun 07 09:54:30 <bencoh> -_-
Jun 07 10:13:43 <Maxdamantus> What I'm saying has no effect on whether or not you can use emojis in filenames.
Jun 07 10:14:00 <Maxdamantus> Just that the kernel shouldn't have to know about emojis.
Jun 07 10:14:14 <Maxdamantus> or the definition of the filesystem format.
Jun 07 10:14:52 <Maxdamantus> Unicode already gives a really good way to represent emojis as bytes: UTF-8
Jun 07 10:16:22 <Maxdamantus> A lot of the time when designing software, the best way to support things is not to cram lots of features in that specifically tend to that thing.
Jun 07 10:16:29 <Maxdamantus> but rather to be generic enough to allow it.
Jun 07 10:17:52 <Maxdamantus> Linux filesystems are perfectly generic enough, since they all handle filenames as bytes, and UTF-8 is a very good way to represent Unicode as bytes, even versions of Unicode that came out after the filesystem was last updated.
Jun 07 10:18:33 <Maxdamantus> That seems to be the direction that Apple's new filesystem is going, since they're presumably well aware by now that what they did was completely wrong.
Jun 07 10:18:58 <L29Ah_> another filesystem?
Jun 07 10:19:22 <Maxdamantus> Yeah, APFS
Jun 07 10:20:03 <Maxdamantus> HFS+ does two really stupid things WRT unicode: use 16-bit code units (so things are meant to be encoded in UTF-16), and do unicode normalisation at the filesystem level.
Jun 07 10:20:10 <Maxdamantus> afaict, APFS does neither.
Jun 07 10:21:04 <L29Ah_> https://en.wikipedia.org/wiki/Apple_File_System#Design i'd say there are a lot of features you don't usually see at the filesystem level
Jun 07 10:22:09 <Maxdamantus> Like what? Encryption?
Jun 07 10:22:41 <Maxdamantus> Other than encryption, all the other things listed there should be standard features of new filesystems.
Jun 07 10:23:00 <Maxdamantus> eg, btrfs and ZFS are relatively new filesystems that do all of those things other than encryption.
Jun 07 10:23:09 <L29Ah_> multikey encryption, snapshots, data deduplication
Jun 07 10:23:16 <Maxdamantus> (ZFS might do encryption, dunno)
Jun 07 10:23:36 <Maxdamantus> actually, ZFS might not have cloning of files .. btrfs does.
Jun 07 10:24:07 <L29Ah_> i don't see the reason why snapshots should be done at the filesystem level instead of the lower one (as in lvm)
Jun 07 10:24:43 <L29Ah_> data deduplication/CoW may have been as well (as seen in fossil + venti), yet it didn't go mainstream for some reason
Jun 07 10:24:57 <Maxdamantus> Well, why not just do data integrity at the LVM level?
Jun 07 10:25:27 <Maxdamantus> It's perfectly feasible for something like LVM to be CoW and provide total data ordering to arbitrary filesystems that sit on top of it.
Jun 07 10:25:30 <L29Ah_> data integrity is done at the hardware level :]
Jun 07 10:25:45 <Maxdamantus> I don't trust data integrity done at the hardware level.
Jun 07 10:26:13 <L29Ah_> then it's time for you to use raid
Jun 07 10:26:35 <Maxdamantus> er, when I said integrity before, I more meant "crash protection"
Jun 07 10:26:37 <L29Ah_> i don't see what task the extra checksums at filesystem level would solve
Jun 07 10:26:59 <L29Ah_> maybe you wanted journaling?
Jun 07 10:27:10 <Maxdamantus> No, journalling is a really bad way of solving that.
Jun 07 10:27:18 <Maxdamantus> Journalling means you have to write everything twice.
Jun 07 10:27:27 <Maxdamantus> Which means your speed is at least halved.
Jun 07 10:27:32 <Maxdamantus> CoW is a much better way to do it.
Jun 07 10:27:34 <L29Ah_> it doesn't mean that
Jun 07 10:27:43 <Maxdamantus> Yes, it does. That's exactly what journalling is.
Jun 07 10:27:46 <L29Ah_> CoW doesn't impede journaling
Jun 07 10:27:58 <Maxdamantus> Write the new stuff into a temporary space, then write it to the actual space.
Jun 07 10:28:08 <L29Ah_> and you don't need to store checksums for CoW
Jun 07 10:28:14 <Maxdamantus> That way if you crash in the middle, you can ignore the stuff in the temporary space.
Jun 07 10:28:29 <Maxdamantus> or ignore the stuff in the actual space (and copy everything from the temporary space)
Jun 07 10:28:46 <Maxdamantus> note: practically noone actually uses full journalling on ext, only metadata.
Jun 07 10:29:12 <Maxdamantus> The best crash protection you can get from ext without full journalling is data=ordered, which has its own performance issues.
Jun 07 10:29:17 <L29Ah_> well, i meant CoW for data, journaling for metadata
Jun 07 10:30:07 <Maxdamantus> So you're writing the metadata twice? Why?
Jun 07 10:31:09 <L29Ah_> to avoid data loss on crashes (:
Jun 07 10:31:18 <Maxdamantus> You get that with CoW.
Jun 07 10:31:26 <Maxdamantus> and you don't have to write everything twice.
Jun 07 10:31:53 <L29Ah_> CoW over metadata?
Jun 07 10:32:00 <Maxdamantus> Also, you mentioned RAID .. RAID is a really bad solution to this as well.
Jun 07 10:32:03 <L29Ah_> where do i store the metadata about metadata then?
Jun 07 10:32:07 <Maxdamantus> It practically requires .. again .. journalling.
Jun 07 10:32:18 <Maxdamantus> So with RAID you have to write everything twice.
Jun 07 10:32:43 <Maxdamantus> If you have a really expensive hardware RAID controller, it might have hundreds of megabytes of really fast NVRAM to work as its journal.
Jun 07 10:33:11 <Maxdamantus> if you're using a less expensive RAID controller or software RAID, you're just fucked in terms of crash protection.
Jun 07 10:33:28 <Maxdamantus> Every RAID setup that doesn't have a journal is vulnerable to the write hole issue.
Jun 07 10:33:44 <Maxdamantus> Again, you need CoW to do this without journalling.
Jun 07 10:34:24 <Maxdamantus> Things like btrfs and ZFS offer RAID-like features (striping, parity, duplicating, etc), but using CoW instead of journalling.
Jun 07 10:34:42 <Maxdamantus> Well, btrfs' parity mode supposedly still has the write hole, last I checked.
Jun 07 10:35:36 <Maxdamantus> 22:32:03 < L29Ah_> where do i store the metadata about metadata then?
Jun 07 10:35:50 <Maxdamantus> You don't store it until you know all of the metadata is stored.
Jun 07 10:36:02 <Maxdamantus> "it" being the information in the superblocks.
Jun 07 10:36:46 <Maxdamantus> The superblock is a small block of information that is replicated multiple times on a single device.
Jun 07 10:36:57 <L29Ah_> :*
Jun 07 10:37:20 <Maxdamantus> When mounting the filesystem, you read all the instances of the superblock and look for the latest one that matches its own checksum.
Jun 07 10:38:09 <Maxdamantus> (this is in btrfs, other CoW filesystems might have other solutions, but they'll probably be essentially the same)
Jun 07 10:43:23 <sicelo> OT: what is currently best RSS reader for N900?
Jun 07 10:44:03 * L29Ah_ uses http://rss2email.blasux.ru/
Jun 07 10:45:10 <sicelo> i'm not looking for something web based
Jun 07 10:45:45 <bencoh> I'd tend to say cutenews, but...
Jun 07 10:46:00 <L29Ah_> it is not web based
Jun 07 10:47:09 <sicelo> bencobut ...
Jun 07 10:48:21 <sicelo> L29Ah_: i also don't want them in my mailbox
Jun 07 10:49:52 <bencoh> sicelo: I thought about using it with miniflux (since miniflux would prepare content remotely then provide me with a clean rss feed / clean articles)
Jun 07 10:50:16 <bencoh> I eventually ditched the whole idea and don't use any rss reader
Jun 07 10:50:36 <bencoh> just that cutenews seemed the most clean/efficient at that time
Jun 07 10:50:44 <bencoh> (compared to, say, feedingit)
Jun 07 10:51:45 <sicelo> thanks bencoh. will use cutenews then
Jun 07 10:52:47 <sicelo> i think i do need an rss reader. stock one is fine for me, but it has problems with others
Jun 07 11:01:37 <sicelo> what did you end up using?
Jun 07 11:02:09 <bencoh> no rss, no articles on the go
Jun 07 11:02:13 <bencoh> not a great loss anyway
Jun 07 11:17:36 <Wikiwide> What's the problem with built-in rss reader?
Jun 07 11:24:56 <sicelo> i do use it just fine most of the time - however it can't open for example https://news.ycombinator.com/rss
Jun 07 12:09:02 <DocScrutinizer05> Wikiwide: everything >1MB for el-v1.db is considered to bring down whole system to a grinding halt, due to sqlite accessing all that stuff and thus bringing swap hell to a system with only 250MB of RAM with only ~50MB of that available to userland
Jun 07 12:13:43 <Wikiwide> Hmm... Interesting... My point of view is that el-v1.db should be separated into many small files (though it would be hell to maintain), so that when user wants the last 10 messages, he doesn't have to load 1million-heavy history into RAM.
Jun 07 12:13:47 <L29Ah_> does sqlite always read the whole database to ram?
Jun 07 12:14:05 <L29Ah_> also i see 100MB free on my pretty much stock maemo
Jun 07 12:14:10 <Wikiwide> How do other programs do that? Log and load logs into GUI?
Jun 07 12:14:47 <DocScrutinizer05> Wikiwide: it's one of the known problems in maemo that Nokia devels never tested maemo for long term sustainability, neither with 1000s of contacts in database, nor for cleaning up any 'logging' files like particularly el-v1.db. You *could* write your own SQL cleanup jobs if you know SQL and understand how the db is structured
Jun 07 12:15:49 <DocScrutinizer05> for all those questions please refer to RDBMS-101
Jun 07 12:15:55 <Wikiwide> What would cleanup entail? I know basic SQL, but structure of el-v1.db is confusing.
Jun 07 12:16:07 <DocScrutinizer05> yes, that's a problem
Jun 07 12:17:05 <DocScrutinizer05> luckily I think the data isn't incremental at all, in a sense of "you can delete old records without any impact to new data"
Jun 07 12:17:45 <Wikiwide> I don't want to delete records.
Jun 07 12:18:00 <DocScrutinizer05> it's probably safe to delete just records with old timestamps, out of a set of very many similar types of records
Jun 07 12:18:13 <Wikiwide> Imagine life-logger.
Jun 07 12:18:23 <DocScrutinizer05> how would you shrink your el-v1.db without doing that?
Jun 07 12:19:06 <Wikiwide> Also, some time in distant future it could be possible to implement Find, and search All Messages for a word, sender, or something. 
Jun 07 12:19:19 <DocScrutinizer05> *shrug*
Jun 07 12:19:55 <Wikiwide> How to shrink? Split it, either chronically or channel-y or both, and load only pieces needed.
Jun 07 12:20:32 <DocScrutinizer05> you can't keep 1TB of data on a 512MB uSD, and you can'tkeep the data of a 50MB el-v1.db in a el-v1.db <1MB
Jun 07 12:20:40 <Maxdamantus> imo that data shouldn't even all be stored in sqlite.
Jun 07 12:21:28 <Maxdamantus> Simple log files for actual conversation data seems more sensible to me .. easier to maintain/trim.
Jun 07 12:21:33 <DocScrutinizer05> you can mv el-v1.db el-v1_1.db any time. Instantly reboot after that!
Jun 07 12:21:48 <Wikiwide> Mmm. Sqlite has good tools to go through data. Where and the like. Text file wouldn't be as easy to edit.
Jun 07 12:21:50 <Maxdamantus> and don't need special tools to search .. grep should just work.
Jun 07 12:23:03 <Wikiwide> Grep wouldn't do field search, like who-sender or which-channel or SMS-vs-IM
Jun 07 12:23:50 <L29Ah_> googling suggests sqlite won't load the whole database into ram unless explicitly requested to
Jun 07 12:24:08 <Wikiwide> I think that Quassel has similar Sql database for logs, or something. 
Jun 07 12:24:08 <DocScrutinizer05> of course not
Jun 07 12:24:15 <Maxdamantus> grep who-sender ~/irclogs/Freenode/#maemo.log
Jun 07 12:24:16 <DocScrutinizer05> read RDBMS-101
Jun 07 12:24:33 <L29Ah_> so i doubt it makes sense to rotate it
Jun 07 12:24:40 <Maxdamantus> grep '^..:.. <.Wikiwide>' ~/irclogs/Freenode/#maemo.log
Jun 07 12:24:47 <bencoh> Wikiwide: tbh when I'm looking for some information in my (say) text messages I just launch sqlite and fire a SELECT query
Jun 07 12:25:07 <Wikiwide> Okay, RDBMS-101. Will look it up.
Jun 07 12:25:11 <DocScrutinizer05> I doubt this chat makes sense to me, I think I'm missing parts thanks to /ignore
Jun 07 12:26:01 <DocScrutinizer05> https://www.ixquick.com/do/metasearch.pl?query=RDBMS-101  :-)
Jun 07 12:26:13 <Wikiwide> bencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is.
Jun 07 12:27:57 <DocScrutinizer05> a RDBMS is usually b-tree or somesuch, with indizes. depending on the operation you perform, it's necessary to sometimes walk the complete dataset which means you load every part of dataset to RAM, sequentially. Since linux fs and RDBMS both keep buuffers, you will inevitably hog some RAM with that
Jun 07 12:31:09 <DocScrutinizer05> and mere experience holds evidence that el-v1.db >1or2 MB brings device to a state that's close to "grinding halt", with delays in the range of 10 to 30 seconds to events that are supposed to be realtime, like dialer popping up on inbound call. You don't want that to happen 30s *after* inbound call started
Jun 07 12:32:13 * Maxdamantus has IRC logs that are gigabytes long
Jun 07 12:32:32 <Maxdamantus> irssi doesn't care .. all it does is open them in append mode and write to them.
Jun 07 12:33:06 <DocScrutinizer05> bencoh: make that a script, and augemt it so it could work across multiple database files, so users could keep archive in el-v1_*.db ?
Jun 07 12:33:10 <Maxdamantus> (on PC, not on N900, though N900 probably wouldn't suffer from having those large logs either)
Jun 07 12:33:18 <DocScrutinizer05> bencoh: that would be an extremely useful little app
Jun 07 12:33:30 <bencoh> DocScrutinizer05: it probably would, yeah
Jun 07 12:34:47 <DocScrutinizer05> my SQL is pretty dusty, thugh I have a comprehensive understanding of how things work and should work
Jun 07 12:35:12 <DocScrutinizer05> IOW I could do SQL but it takes ages
Jun 07 12:35:13 <Maxdamantus> imo if there were to be more advanced search features, it should just use the logs as a source of truth.
Jun 07 12:35:29 <bencoh> "the logs"?
Jun 07 12:36:06 <Maxdamantus> ie, once you've got logs that are >100 MiB long, you can generate extra search information that's stored in a cache.
Jun 07 12:36:35 <bencoh> some kind of index?
Jun 07 12:36:40 <Maxdamantus> If you think that search information might be corrupted, just delete it and have it regenerated.
Jun 07 12:36:40 <KotCzarny> maxd: that's assuming world is static, ie. doesn't change
Jun 07 12:36:52 <bencoh> KotCzarny: not necessarily
Jun 07 12:37:02 <bencoh> see mairix, notmuch, xapian, the likes
Jun 07 12:37:05 <Maxdamantus> KotCzarny: well, the index data would still be generated incrementally.
Jun 07 12:37:24 <bencoh> (xapian is actually the engine. mairix/notmuch are frontends)
Jun 07 12:37:37 <KotCzarny> there is a saying 'only cow doesnt change it's opinions'
Jun 07 12:37:53 <bencoh> only CoW?
Jun 07 12:37:57 <bencoh> :p
Jun 07 12:38:03 <KotCzarny> moo :P
Jun 07 12:38:21 <DocScrutinizer05> ~moo
Jun 07 12:38:22 <infobot> mooooooo! I am cow, hear me moo, I weigh twice as much as you. I'm a cow, eating grass, methane gas comes out my ass. I'm a cow, you are too; join us all! type apt-get moo.
Jun 07 12:41:00 <DocScrutinizer05> ((<Wikiwide> bencoh: nice! SQL SELECT aka search is a good example of how human-friendly sqlite database is.)) also a good example for the RDBMS needing to walk the *complete* dataset if it can't optimize the SELECT condition to use indizes
Jun 07 12:42:43 <DocScrutinizer05> and I think what wikiwide meant is "human friendly" of 4th generation languges like SQL at large
Jun 07 12:43:23 <DocScrutinizer05> the tricky part however starts with defining and understanding the data model
Jun 07 12:43:45 <DocScrutinizer05> *using* it is simple, after that
Jun 07 12:45:12 <DocScrutinizer05> SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE);  ;-P
Jun 07 12:46:16 <DocScrutinizer05> now is-relevant-to-wikiwide could be a stored procedure AIUI ;-)
Jun 07 12:47:10 <DocScrutinizer05> still ineffective until you get an index for it
Jun 07 12:47:36 <DocScrutinizer05> I prefer to use grep for such stuff, generally
Jun 07 12:48:19 <DocScrutinizer05> XML isn't all bad for those purposes
Jun 07 12:49:02 <DocScrutinizer05> not compatible with simple grep usually, but still simpler and more lightweight than the usual RDBMS
Jun 07 12:51:20 <DocScrutinizer05> for IRC I have plaintext logfiles (not on N900) and grepping in them is a matter of milliseconds, even in the 20+MB sized ones
Jun 07 12:52:33 <DocScrutinizer05> anyway maemo fremantle lacks any useful means (except for SMS iirc) to clean out old records form el-v1.db
Jun 07 12:52:53 <DocScrutinizer05> s/form/from/
Jun 07 12:52:53 <infobot> DocScrutinizer05 meant: anyway maemo fremantle lacks any useful means (except for SMS iirc) to clean out old records from el-v1.db
Jun 07 12:53:35 <DocScrutinizer05> heck, I even don't lnow of any simple archiver or export tool for the data in there
Jun 07 12:56:30 <Maxdamantus> I'm pretty sure grep is even going to be faster at most searches you'd want to do.
Jun 07 12:57:05 <Maxdamantus> because of DocScrutinizer05's point; unless you're doing some operation that the schema was specifically designed to make optimal, it's probably just searching through all the rows.
Jun 07 12:58:22 <Maxdamantus> I imagine in the best case you might be able to get faster searches for a particular time period, but if every message is logged with a timestamp and there are no clock skews, you can already do that using `look` from util-linux
Jun 07 12:59:03 <Maxdamantus> (`look` does a line-wise binary search on a file)
Jun 07 12:59:37 * Maxdamantus hacked together some program himself that did that, before realising `look` already existed
Jun 07 13:01:05 <Maxdamantus> I suspect still in that case, running an entire look process will be far quicker than simply opening an SQLite database
Jun 07 13:03:44 <Maxdamantus> Hm, though `look` doesn't seem to have an option to just produce an offset into the file.
Jun 07 13:04:08 <Maxdamantus> My program did that, so you could just pass the result as an argument to something like `less` to seek to the right position.
Jun 07 13:06:35 <Maxdamantus> Oh hey, my program can actually exec less for you.
Jun 07 13:08:42 <Maxdamantus> Funnily enough, I also have that mechanism built into my terminal.
Jun 07 13:09:20 <KotCzarny> yeah, probably set PAGER variable in bash
Jun 07 13:09:41 <Maxdamantus> in rxvt-unicode you can hold ctrl-shift and drag to show information about a character .. I modified my rxvt-unicode so whenever it calculates that information it does a binary search on UnicodeData.txt to show more interesting stuff.
Jun 07 13:10:24 <Maxdamantus> No noticable latency, because each lookup takes probably less than 1 ms
Jun 07 13:10:40 <Maxdamantus> only a 1.4 MiB file, but obviously binary search scales extremely well.
Jun 07 13:13:58 <Maxdamantus> $ time ./findline /usr/share/unicode-data/UnicodeData.txt 000{0..9}\; >/dev/null 
Jun 07 13:13:58 <Maxdamantus> real    0m0.001s
Jun 07 13:14:36 <Maxdamantus> https://gist.github.com/Maxdamantus/2e344f688d9715490cb0ea6dd6ffb8bd in case anyone's interested
Jun 07 13:14:58 <DocScrutinizer05> SELECT responsetext FROM ideal-database WHERE (is-relevant-to-wikiwide = TRUE) gets you a dozen or a few 10000s of lines/records out of a 500MB database in only a few milliseconds, while  SELECT responsetext FROM the-usual-database WHERE (responsetext LIKE "this is what I search for"); takes ages and gets beat by a simple grep hands down
Jun 07 13:15:33 * Maxdamantus just adapted it from his urxvt code one day at work when analysing some stuff from a database dump
Jun 07 13:23:49 <DocScrutinizer05> database dudes have a hard time eachs single time I show them a grep is faster than their RDBMS in particular cases ;-P
Jun 07 13:24:46 <DocScrutinizer05> "but but but...! it's not... errr... wel you can't.... ERRRR but it's NOT SQL!"
Jun 07 13:48:14 <Maxdamantus> fun test: sha1sum <(./findline -e -i /tmp/unicodedata-sorted.txt </tmp/unicodedata-sorted.txt) /tmp/unicodedata-sorted.txt
Jun 07 13:49:13 <Maxdamantus> (for every line in the file, looks for that line in the file using binary search)
Jun 07 13:50:22 <Maxdamantus> about 170k independent searches per second
Jun 07 13:51:39 <Maxdamantus> could easily be optimised, since there are particular hotspots that the binary searches all touch that could remain in memory (ie, the lines at 0.5, 0.25, 0.75, ...)
Jun 07 14:37:02 <parazyd> freemangordon: amprolla is finished i believe :)
Jun 07 14:37:23 <parazyd> should undergo some testing now by the 'big bosses'
Jun 07 14:37:59 <parazyd> will find out more when they wake up (also got a devuan meeting tonight i think)
Jun 07 15:57:41 <freemangordon> parazyd: great :)
Jun 07 16:13:17 <Wizzup> parazyd: thanks for going the extra mile and getting it into devuan. onwards
Jun 07 16:13:25 <Wizzup> s/extra/final/
Jun 07 16:13:25 <infobot> Wizzup meant: parazyd: thanks for going the final mile and getting it into devuan. onwards
Jun 07 16:23:38 <ff_> https://www.sqlite.org/lang_vacuum.html el-v1.db might help
Jun 07 16:25:48 <ff_> I done it, and el-v1.db is 14% smaller now, no data loss, I don't know about speed
Jun 07 16:26:55 <bencoh> :)
Jun 07 17:21:41 <sicelo> back to my rss issue - cutenews doesn't load on my N900. 
Jun 07 17:23:19 <sicelo> anyway, guess should try all of them one by one 
Jun 07 17:29:53 <sicelo> ah, silly problem. http://talk.maemo.org/showthread.php?t=95797&page=3
Jun 07 17:41:50 <parazyd> Wizzup: thank you for the help :)
Jun 07 18:10:37 <sixwheeledbeast> sicelo: seems a bit odd for it to be file permissions issue.
Jun 07 18:11:53 <sicelo> indeed, but at least on my installation that worked. i actually first adjusted the .desktop file, which didn't work
Jun 07 18:49:15 <sixwheeledbeast> Thanks I'll try it later if I get a minute
Jun 07 23:54:39 <Maxdamantus> Heh .. I mentined yesterday how HFS+ does unicode normalisation and that I wondered how it handled things between systems with different versions.
Jun 07 23:55:35 <Maxdamantus> Their solution is actually quite simple: the normalisation is just based on a particular version of unicode: 3.2, released in 2002
Jun 07 23:56:15 <Maxdamantus> so it simply doesn't do normalisation of newer code points.
Jun 07 23:56:47 <Maxdamantus> because if they did .. you'd have that major issue.
Jun 07 23:57:13 <Maxdamantus> (not that it still has major issues)
Jun 08 00:05:24 <Maxdamantus> doesn't still have*
Jun 08 00:49:28 <Wikiwide> I didn't know Maemo 2 existed: http://repository.maemo.org/pool/maemo2.1/free/source/h/hello-world-app/
Jun 08 00:54:15 <Wikiwide> Question : what's the difference between maemopad 2.4 in Diablo and maemopad 2.6 in Fremantle?
Jun 08 00:55:56 <Wikiwide> A large difference, according to dependencies. What is libhildonhelp0?
**** ENDING LOGGING AT Thu Jun 08 03:00:01 2017