**** BEGIN LOGGING AT Mon Dec 16 03:01:54 2019
**** BEGIN LOGGING AT Mon Dec 16 07:26:31 2019
Dec 16 09:18:48 <nag68> Please kindly assist to get Certification of non-use for < Red phosphorus>
Dec 16 09:18:57 <nag68> MPN: BBONE-BLACK-4G
Dec 16 09:19:17 <zmatt> nag68: ask your distributor/manufacturer
Dec 16 09:19:42 <nag68> please provide contact details
Dec 16 09:20:26 <zmatt> I'm sure you can find those on your distributor/manufacturer's website
Dec 16 09:20:33 <zmatt> that's where I'd look for them anyway
Dec 16 09:21:19 <zmatt> this is a community support channel for the beaglebone, we don't have any particular involvement with any of the manufacturers or distributors (there are multiple)
Dec 16 15:17:07 <kremlin> m
Dec 16 15:33:35 <Ragnorok> Moanin!  (here)  I've been trying to use the watchdog timer on a Black, and it's working perfectly.  Problem is every time I kick it the system log gets a line that says "watchdog didn't stop!".  But it does kick it.  I've been scouring the web for a number of hrs and haven't found much.
Dec 16 15:35:07 <Ragnorok> My recollection is it didn't do that when I started down this path; in that instance I was opening a fd and using ioctl to do everything, but that left the watchdog inaccessible.  I changed it to close the fd after each touch and this started to appear.
Dec 16 15:43:19 <Ragnorok> This is exactly the case.  I just put the code back to doing ioctl on a fd that's left open for the entire run, and nothing in the system log.  Any idea why closing the fd, then accessing the watchdog later, causes this message?
Dec 16 16:23:25 <Ragnorok> It doesn't seem to matter if I use ioctl or system(wdctl) to set the timeout.  If what I do closes the device it work perfectly but logs stop failures each time I kick it.
Dec 16 16:26:31 <Ragnorok> I'm on an older kernal b/c it's a level of effort to change that for this appliance.  Looks like maybe it's up to 4.14 or so?  I saw one whole ref to this being a bug in < 4.8, and comment to that ref saying >= 4.8 didn't actually fix it.
Dec 16 16:47:01 <zmatt> Ragnorok: the watchdog fd should normally never be closed
Dec 16 16:47:41 <zmatt> also, note that if you're using systemd, you'd normally let it manage the watchdog (if you want a watchdog on individual services, systemd offers that too)
Dec 16 16:48:22 <zmatt> the "watchdog didn't stop!" is a warning that the watchdog fd has been closed while leaving the watchdog running
Dec 16 16:49:02 <Ragnorok> Interesting.  If I don't close it, it's locked by my process, which isn't a service.
Dec 16 16:49:29 <zmatt> if your process isn't a service, how can it be responsible for the watchdog?
Dec 16 16:49:42 <Ragnorok> If I never run my process (that might keep it open) then it's accessible as expected.
Dec 16 16:49:59 <zmatt> what do you mean "accessible" ?  accessible by whom, for what?
Dec 16 16:50:15 <zmatt> normally only one process will ever touch the watchdog
Dec 16 16:50:29 <Ragnorok> I dnn't see the connection.  I can manage the watchdog using CLI w/o requiring a service.  When I say "accessbile", I mean by CLI / anything that's not my process.
Dec 16 16:51:27 <zmatt> that sounds like a hack, presumably it exists to offer a way to kick the watchdog before the watchdog service is running (e.g. in initramfs) or maybe for debugging, I don't know
Dec 16 16:51:54 <zmatt> normal use for the watchdog is that one process opens the fd and kicks it periodically
Dec 16 16:52:06 <zmatt> most commonly that process would be either systemd or watchdogd
Dec 16 16:52:12 <Ragnorok> I don't either.  Hours of scouring the web have only yielded info on using /dev/watchdog essentially from CLI.  Nothing about services.
Dec 16 16:52:29 <zmatt> I've never heard of it being used from the CLI, that's news to me
Dec 16 16:52:47 <Ragnorok> Well ioctl or cli.  I view them as similiar from my code's perspective.
Dec 16 16:52:59 <zmatt> no they're a huge difference
Dec 16 16:53:08 <zmatt> cli necessarily means open and close
Dec 16 16:53:24 <zmatt> as opposed to the normal pattern of leaving it open
Dec 16 16:53:28 <Ragnorok> Also nothing about close is bad wrt /dev/watchdog.
Dec 16 16:53:55 <zmatt> not sure, iirc it depends on kernel config whether that will stop the watchdog or not
Dec 16 16:54:01 <zmatt> and if it doesn't, you'll get that warning in the kernel log
Dec 16 16:54:02 <Ragnorok> It seems like I should write a service, then, and have my process tell that service to kick it?
Dec 16 16:54:46 <Ragnorok> If systemd is doing something along those lines, nothing has appeared regarding that, either.
Dec 16 16:55:07 <Ragnorok> In fact systemd hasn't appeared at all regarding searching for watchdog interaction.
Dec 16 16:55:13 <zmatt> just set RuntimeWatchdogSec=30s or something like that in /etc/systemd/system.conf
Dec 16 16:55:20 <Ragnorok> Except to say it's involved at boot & shutdonw.
Dec 16 16:55:30 <Ragnorok> But how do I kick it, then?
Dec 16 16:55:37 <zmatt> systemd will kick it
Dec 16 16:55:58 <Ragnorok> How do I have it reboot if my process crashes?
Dec 16 16:56:08 <zmatt> configure in your service config file
Dec 16 16:56:17 <Ragnorok> My process isn't a service.
Dec 16 16:56:44 <zmatt> very much sounds to me like it should be
Dec 16 16:56:56 <zmatt> otherwise it has no business rebooting the system
Dec 16 16:57:23 <Ragnorok> lol
Dec 16 16:58:12 <zmatt> if it runs as a systemd service you ensure that if your service crashes it will be restarted or (at your option) the whole system will be restarted
Dec 16 16:58:17 <zmatt> *you can ensure
Dec 16 16:58:42 <Ragnorok> I'll see how ugly is it to change it.  I dno't really care if it's a service or not.  It's not a service b/c scripting and debug are vastly simpler if I don't have more automagic in the way.
Dec 16 16:59:18 <zmatt> having systemd manage the watchdog also ensure that you can have multiple services that need to be kept alive... which you can't do with the hardware watchdog
Dec 16 16:59:48 <zmatt> what's your service written in?
Dec 16 17:00:09 <Ragnorok> My process is written in C++.
Dec 16 17:00:19 <zmatt> ah because you mentioned scripting, hence I wondered
Dec 16 17:00:48 <Ragnorok> I use scripting to do everything, as was done in days of yore before all this automagic got crammed in.
Dec 16 17:00:52 <zmatt> if it were nodejs then I would have had a handy module I wrote => https://github.com/dutchanddutch/node-sd-daemon#service-watchdog
Dec 16 17:02:00 <Ragnorok> Everything ofc being "job control" or whatever you want to call it.  The process is simpleton CLI application that just runs.
Dec 16 17:02:38 <zmatt> sounds like I'm going to be a happier person the less I know about it ;)
Dec 16 17:03:21 <zmatt> anyway, you're still free to open the watchdog manually, just don't close it
Dec 16 17:04:46 <zmatt> so wait... if your process isn't a service, how is it started again automatically after reboot?
Dec 16 17:04:50 <Ragnorok> Perhaps.  Either way I'll see about doing the service thing, since that's what it takes to make it work how I want.  Leaving it open doesn't really work b/c troubleshooting can be very difficult if I can't turn the watchdog off interactively.
Dec 16 17:04:54 <zmatt> since a watchdog is pretty useless without that
Dec 16 17:05:02 <Ragnorok> rc.local ofc.
Dec 16 17:05:12 <zmatt> in other words it *is* a service, just a horrid one
Dec 16 17:05:34 <Ragnorok> Horrid is a bit strong.  That's how things used to be done.
Dec 16 17:05:42 <zmatt> why not take whatever you stick into rc.local into a separate systemd service?
Dec 16 17:05:52 <zmatt> (note that rc.local is also just another systemd service)
Dec 16 17:06:00 <Ragnorok> That's what I was going to look into doing, to make a "a service".
Dec 16 17:06:12 <zmatt> hold on, I have a nice template
Dec 16 17:06:21 <zmatt> https://pastebin.com/KXVdTNrL
Dec 16 17:06:40 <Ragnorok> Thanks!  I'll work on this and see what happens.
Dec 16 17:16:48 <zmatt> Ragnorok: https://pastebin.com/RYfr9xfh  some additional useful directives for you
Dec 16 17:17:02 <Ragnorok> Roit toe.  Thanks again!
Dec 16 17:17:29 <zmatt> you can also have systemd just restart your service, and fall back to rebooting the system if your service fails too many times within a configured interval
Dec 16 19:08:22 <rbw> What causes the limit on simultaneous (virtual) serial connections that can be open? after 6 it will throw an exception.  Using Python's pyserial library and 12 devices connected on a 12-port USB hub.
Dec 16 19:08:42 <zmatt> hmm, there shouldn't be any
Dec 16 19:09:05 <rbw> always happens.  i'd need to juggle the connections.
Dec 16 19:09:19 <zmatt> what's the error you get?
Dec 16 19:09:32 <rbw> also, it doesn't happen on the pi4.  haven't tried a 3
Dec 16 19:09:36 <rbw> uh, let me dig that up!
Dec 16 19:13:20 <rbw> will take me a while to do it again, of course i didn't put it into my notes..
Dec 16 19:14:04 <zmatt> when asking for help with an error, it's generally useful to have the actual error at hand :P
Dec 16 19:16:12 <rbw> ha, yes my bad:
Dec 16 19:16:14 <rbw> Traceback (most recent call last):
Dec 16 19:16:15 <rbw> call last):
Dec 16 19:16:18 <rbw> serial.serialutil.SerialException: [Errno 5] could not open port /dev/ttyACM6: [Errno 5] Input/output error: '/dev/ttyACM6'
Dec 16 19:16:32 <rbw> full error when hitting the 7th device
Dec 16 19:16:59 <rbw> import serial
Dec 16 19:17:08 <zmatt> curious, anything in kernel log?
Dec 16 19:18:38 <rbw> while i consider myself knowledgeable with *nix, i don't know a lot about checking the logs!
Dec 16 19:18:46 <rbw> tail -f /var/log/kern.log ?
Dec 16 19:19:18 <zmatt> dunno, I'd generally use journalctl -k -f
Dec 16 19:20:15 <rbw> awesome, thanks for the command.  this looks like IRQ error
Dec 16 19:20:25 <zmatt> if you have a syslogd that writes to /var/log/kern.log then you can also consult that file, but I haven't used any syslogd in many years so I'm not sure what it writes where
Dec 16 19:20:39 <rbw> gotcha
Dec 16 19:20:39 <zmatt> can you share the error via pastebin.com or similar paste service?
Dec 16 19:23:00 <rbw> both kernel and python errors are above but i'm working on that
Dec 16 19:23:17 <zmatt> ?
Dec 16 19:23:47 <rbw> https://pastebin.com/vuVKjR9f
Dec 16 19:27:46 <rbw> is that what you meant..?
Dec 16 19:27:55 <zmatt> the last two lines yes
Dec 16 19:28:44 <zmatt> okay so it's about musb running out of endpoints... which makes sense, musb is a small usb otg controller, it's not really a full-featured usb host controller.  though I wonder why the rpi wouldn't be affected, I thought it also used musb?
Dec 16 19:29:10 <rbw> Ya that seemed curious to me.  does pi4 use two of those or something?
Dec 16 19:29:46 <zmatt> based on what I'm reading, apparently some kind of multiplexing should kick in when it runs out of endpoints, although then the question becomes why that isn't happening
Dec 16 19:31:56 <rbw> got a link to that kind of documentation?  i'm not terribly familiar with the BBB.  just realized i didn't mention it was a Black but either they're similar or you figured that out!
Dec 16 19:32:32 <zmatt> http://e2e.ti.com/support/processors/f/791/t/778528?Linux-AM3358-MUSB-driver-and-endpoint-usage   was the first google hit
Dec 16 19:35:37 <zmatt> which kernel version are you using?
Dec 16 19:37:48 <rbw>  uname -r
Dec 16 19:38:14 <zmatt> ?
Dec 16 19:38:37 <rbw> latest stretch IoT version?
Dec 16 19:38:48 <zmatt> you just said 20:37 < rbw>  uname -r
Dec 16 19:39:50 <rbw> i downloaded the latest stretch IoT version to card and did a number of updates.  but not sure how to verify kernel version directly
Dec 16 19:40:14 <zmatt> with uname -r
Dec 16 19:40:35 <rbw> ok, so i'm trying to say i wouldn't doubt that i screwed something up..
Dec 16 19:40:39 <zmatt> but okay, so probably 4.14
Dec 16 19:40:53 <zmatt> no I see no reason to assume that
Dec 16 19:41:16 <zmatt> I'm just trying to understand why this endpoint allocation scheme is failing
Dec 16 19:42:32 <zmatt> though in some sense the core issue is just overburdening of the simplistic musbhdrc controller, which was never meant to handle that many devices at the same time
Dec 16 19:43:38 <rbw> ya, that's probably just it.  i've been exploring it as an embedded controller for some usb devices.  workaround may have to be to juggle the connections.
Dec 16 19:45:21 <zmatt> still, the kernel has a workaround that should allow an arbitrary number of bulk endpoints... it just doesn't seem to be working for some reason
Dec 16 19:46:30 <zmatt> hmm, does CDC-ACM use an interrupt endpoint? lemme check
Dec 16 19:51:13 <zmatt> rbw: could you dump the descriptors of one of your devices with  lsusb -D /dev/bus/usb/001/xxx  (where xxx is the device number as listed in plain "lsusb")
Dec 16 19:52:39 <rbw> sure, one minute here..
Dec 16 19:53:50 <rbw> zmatt can I send it to you directly?
Dec 16 19:54:06 <zmatt> I think I understand what's happening... all available endpoint pipes are occupied by bulk endpoints, and subsequently the allocation for an interrupt endpoint fails.  that also explains the workaround in that E2E thread of preemptively forcing bulk endpoints to use the shared EP1
Dec 16 19:54:15 <zmatt> just use pastebin or similar paste service
Dec 16 19:56:07 <rbw> https://pastebin.com/UJ0trYpi
Dec 16 19:56:39 <zmatt> yeah, bulk in, bulk out, interrupt in
Dec 16 19:58:22 <rbw> well i'm impressed!  I have much to learn..  does that mean it is expected behavior for this type of device?
Dec 16 19:58:32 <zmatt> so the workaround in that thread might work for you too... although the guy said he ended up having to limit the endpoint-sharing to only bulk-in (so each bulk-out would still use its own dedicated musb endpoint)... so I don't know if you'd still run into problems with 12 ACM devices
Dec 16 20:00:12 <rbw> OK.  it makes sense on some level, but it's an area i'm not too familiar with!  Gives me a lot to go by though.  Thanks!
Dec 16 20:00:32 <zmatt> like I said: the core issue is just that the musbhdrc controller isn't designed to handle that many devices at the same time
Dec 16 20:01:53 <zmatt> using more than 4 ACM devices already involves workarounds in the kernel
Dec 16 20:02:44 <rbw> Trying to ask intelligently, it is designed for no more than 6 USB devices?  which are occupied when opening them as virtual serial devices?
Dec 16 20:03:16 <rbw> i know as much that there is some magic to map USB to the ACM/virtual serial devices..
Dec 16 20:03:58 <zmatt> it supports 15 endpoints simultaneously (excluding control endpoints).  Every usb serial (CDC-ACM) device uses 3 endpoints
Dec 16 20:04:18 <zmatt> it has nothing to do with ACM specifically, it's purely about the number of endpoints used
Dec 16 20:04:32 <zmatt> the hub also uses an endpoint
Dec 16 20:05:31 <Ragnorok> journalctl is very annoying.  When I do journalctl -f it keeps dumping every log possible until it gets to the bottom, then it does --Reboot-- and shows a paltry ten lines of the end of the log.  Is that normal?
Dec 16 20:05:46 <zmatt> Ragnorok: ehh what?
Dec 16 20:05:56 <rbw> @zmatt ok, that makes sense. although i do get to open 6 with the hub.  Anyway, this is awesome, thanks for your time!.
Dec 16 20:06:16 <zmatt> rbw: like I said, the kernel already has workarounds to be able to support more than it otherwise could
Dec 16 20:06:49 <rbw> ah, just not as many as i hit..
Dec 16 20:06:51 <zmatt> and it is apparently possible to further raise the limit somewhat using a kernel patch (at expense of performance)
Dec 16 20:07:17 <rbw> gotcha.  thanks again!
Dec 16 20:07:24 <zmatt> but 12 ACM devices would really be pushing it
Dec 16 20:07:42 <zmatt> Ragnorok: that sounds utterly bizarre
Dec 16 20:07:48 <Ragnorok> Since I'm forced to use this goofy systemd thing I'm also using its log, which means I have to use journalctl to see it.  It's always been a massive PITA.  Unless there's a way to run a service CLI w/o using systemd start, I have to use journalctl -f to see what it's doing, and it never worked right.
Dec 16 20:08:18 <Ragnorok> It also routinely fails to retain logs across boots, even though I have it configured to do so, and the dir it creates has content.
Dec 16 20:08:27 <zmatt> Ragnorok: I mean, that's not any different from being started by rc.local
Dec 16 20:08:41 <zmatt> weird, journal works quite well for me
Dec 16 20:08:52 <Ragnorok> Logs have never worked with that launch means either.
Dec 16 20:09:36 <Ragnorok> I have a logging wrapper that uses systemd for logging and it's fine for the current session.  Usually.  Prior boots are never complete.
Dec 16 20:09:53 <zmatt> while doing clear reboots?
Dec 16 20:09:57 <zmatt> *clean
Dec 16 20:10:13 <zmatt> also, what version of systemd are you using?
Dec 16 20:10:22 <Ragnorok> But once in a while it simply doesn't log things.  It's not rate limit, or if it, it's not telling me.  Clean being normal shutdown and power cycle?  Yes.
Dec 16 20:10:58 <zmatt> clean being software-requested shutdown yes (as opposed to unceremoniously cutting power)
Dec 16 20:11:20 <zmatt> if it's ratelimiting it will tell you
Dec 16 20:11:23 <Ragnorok> Right.  I always do a clean reboot unless I have no choice.
Dec 16 20:12:13 <zmatt> the journalctl -f thing sounds really odd to me, it should basically work just like tail -f
Dec 16 20:12:21 <zmatt> and I've never seen it do otherwise
Dec 16 20:12:26 <Ragnorok> I'm not clever enough to know what systemd version I'm using.  Let me go look.
Dec 16 20:12:34 <Ragnorok> I always see it do otherwise.  It's nighmarish.
Dec 16 20:12:38 <zmatt> systemctl --version
Dec 16 20:13:08 <Ragnorok> I'd expect that to tell me the version of systemctl, not systemd.
Dec 16 20:13:13 <zmatt> those are the same
Dec 16 20:13:32 <Ragnorok> It's Linux.  I would never assume that.  Says 215.
Dec 16 20:13:48 <zmatt> they're in one package
Dec 16 20:14:14 <zmatt> v215 ... release july 2014, ok
Dec 16 20:14:30 <zmatt> so, that might be a bit old :P
Dec 16 20:15:24 <Ragnorok> Everything here is a bit old.  It's that updating an appliance problem.  I don't have three weeks to fix all the broken packages right now.  I did try a kernel update earlier today and it was the expected heinous tragedy.
Dec 16 20:15:40 <zmatt> holy cow that's old... that's not even the latest version of debian jessie
Dec 16 20:15:46 <zmatt> *for
Dec 16 20:16:07 <Ragnorok> I just need a watchdog to work on thing that been working flawlessly for months.  That's it.
Dec 16 20:16:28 <zmatt> I mean, yeah, if I strain myself to think back that far then yeah maybe journal still had issues back then
Dec 16 20:16:46 <Ragnorok> But using a service, and journalctl, debugging is nightmarish.
Dec 16 20:16:46 <zmatt> so just use the watchdog manually like you were doing... just don't close it
Dec 16 20:17:09 <zmatt> I don't even know if service watchdogs were a thing yet in such an ancient version of systemd
Dec 16 20:17:59 <zmatt> I mean, journalctl has worked fine for years, but maybe not *that* many years, I don't really quite remember
Dec 16 20:18:00 <Ragnorok> It seems to build find.  It did log a couldn't stop the one time it seemed like it loaded, but I haven't yet determined where in the code it was when that happened.
Dec 16 20:18:16 <zmatt> couldn't stop = fd closed
Dec 16 20:18:57 <Ragnorok> My memory is it was better at some point in the past, but it's never really be super great at retaining logs across boots.  It gets ... fragmented.  Syslog was much more reliable.
Dec 16 20:19:56 <zmatt> you're using an ancient version, I'm inclined to blame any and all problems you're having with it on it simply being an ancient version unless you can reproduce them on an uptodate version :P
Dec 16 20:20:25 <Ragnorok> If I just leave it open like I was, then if anything at all goes wrong at any time, I can't kill the process manually without the watchdog resetting the system.
Dec 16 20:20:38 <Ragnorok> That is the simplest course of action.
Dec 16 20:21:10 <zmatt> I mean, that's how a watchdog is supposed to work
Dec 16 20:21:23 <zmatt> there's no difference between manually killing a process and it crashing
Dec 16 20:21:34 <Ragnorok> Had I know it would take three days to fail to get a simple watchdog timer working I may have spend that time trying to update it.
Dec 16 20:22:03 <zmatt> you can still avoid a system reset if you restart the service (or stop the watchdog) soon enough after killing your process
Dec 16 20:22:09 <Ragnorok> If I have access to the watchdog external to the process I can do a kill ; echo 'V' and stop it from rebooting if that's what I need to happen.
Dec 16 20:22:24 <zmatt> echo 'V' ?
Dec 16 20:22:42 <zmatt> if you kill the process, the fd is closed hence you'll be able to access the watchdog again
Dec 16 20:23:03 <Ragnorok> echo 'V' > /dev/watchdog shuts it off.  My xp is that doesn't work cleanly or I wouldn't be worried about it.
Dec 16 20:23:07 <zmatt> (do make sure you don't leak the fd to additional processes if you spawn any, make sure it's close-on-exec)
Dec 16 20:23:40 <Ragnorok> Perhaps I've been opening it incorrectly, then.
Dec 16 20:23:57 <zmatt> is the kernel configured to allow the watchdog to be stopped?
Dec 16 20:24:03 <Ragnorok> It is now.
Dec 16 20:25:27 <Ragnorok> Lastest madness is systemd is trying to CHDIR to my script file, not the WorkingDirectory I set.  Or that's what the log looks like.
Dec 16 20:25:29 <zmatt> anyway, I don't really have any experience with dealing with the watchdog since systemd takes care of that for me.  it does evidently close and reopen it when systemd reexecs (based on the warning in kernel log), so that at least seems to work fine
Dec 16 20:25:54 <zmatt> Ragnorok: what does the log say?
Dec 16 20:26:14 <zmatt> you have the strangest problems...
Dec 16 20:26:27 <Ragnorok> systemd[802]: Failed at step CHDIR spawning <path to script file>: No such file or directory.
Dec 16 20:26:46 <Ragnorok> I do.  It's my computer karma.  I routinely have problems that no one else has heard of.
Dec 16 20:26:54 <zmatt> okay so it's not trying to chdir to your script file
Dec 16 20:27:19 <zmatt> it's saying it's failing to chdir() as part of spawning your script
Dec 16 20:27:31 <zmatt> i.e. WorkingDirectory is set incorrectly
Dec 16 20:27:58 <zmatt> don't forget to systemctl daemon-reload after changing the .service file.  (current systemctl will warn you about this, but dunno if a 4.5-year old one does)
Dec 16 20:28:25 <zmatt> 5.5-year sorry
Dec 16 20:28:56 <Ragnorok> WorkingDirectory=<path> and ExecStart=<path>/<script>  I do that every time.  It's just another in a litany of annoyances getting a service to work.
Dec 16 20:29:44 <Ragnorok> Ah.  Now it says failed to parse notify specifier.
Dec 16 20:30:00 <Ragnorok> That's progress.
Dec 16 20:30:18 <zmatt> yeah sorry I don't know what specifiers do or don't work in a systemd version that old
Dec 16 20:30:48 <Ragnorok> I searched.  I'll do it again.
Dec 16 20:30:58 <zmatt> like I said, I don't know if service watchdogs were even supported yet, so you might want to check that before putting more effort into this
Dec 16 20:31:18 <zmatt> i.e. check the manpages on your system, matching the systemd version you use
Dec 16 20:31:36 <Ragnorok> The image I have on my Black doesn't have man pages.
Dec 16 20:31:57 <zmatt> you know, I'm glad I'm not you right now :D
Dec 16 20:33:28 <zmatt> if it doesn't understand Type=notify I'd be pretty worried
Dec 16 20:33:49 <Ragnorok> It didn't complain on type.  It complained on NotifyAccess.
Dec 16 20:33:54 <zmatt> oh
Dec 16 20:33:56 <zmatt> then omit that
Dec 16 20:34:04 <zmatt> the default is probably fine
Dec 16 20:35:48 <Ragnorok> I found what may be a man page for 215 that contains everything possibly imaginable except any [Service] Section data.  Nice.
Dec 16 20:36:14 <zmatt> normally that would be in the "systemd.service" and "systemd.exec" manpages
Dec 16 20:36:51 <Ragnorok> I see.  I was in the system.unit part.  Duh.
Dec 16 20:37:04 <zmatt> I mentioned that in my service template :)
Dec 16 20:37:11 <zmatt> in the comments
Dec 16 20:37:51 <Ragnorok> https://manpages.debian.org/jessie/systemd/systemd.service.5.en.html  claims NotifyAccess is allowed.  Perhaps it's not the 215 I think it is?
Dec 16 20:38:38 <Ragnorok> The default says nothing will allow notification.
Dec 16 20:40:11 <zmatt> you said you had v214, not not v215
Dec 16 20:40:20 <zmatt> read the last sentence of the description of NotifyAccess
Dec 16 20:41:01 <zmatt> still it seems unlikely NotifyAccess was introduced that late, I'd assume it's a very old option... check for typos?
Dec 16 20:41:02 <Ragnorok> <Ragnorok> It's Linux.  I would never assume that.  Says 215.
Dec 16 20:41:14 <zmatt> oh my bad
Dec 16 20:41:16 <Ragnorok> Ok. I'll take it out.
Dec 16 20:41:47 <zmatt> I guess the 4 from 2014 somehow blended into the version number in my mind or something
Dec 16 20:42:08 <zmatt> well, if you're using v215 then it should definitely be supported, so if it's complaining about it something weird is going on
Dec 16 20:42:11 <Ragnorok> The bottom of that file says "systemd 215.  Does that mean it's 215 man page or it that just there to confuse searches?  lol
Dec 16 20:42:12 <zmatt> or there's a typo
Dec 16 20:42:31 <zmatt> well v215 is the latest and last version for jessie
Dec 16 20:42:38 <Ragnorok> Ok.
Dec 16 20:43:03 <Ragnorok> So that and WatchdogSec failed for whatever reason.
Dec 16 20:43:26 <Ragnorok> Oh I see.  This isn't smart enough to do end of line comments.
Dec 16 20:44:32 <Ragnorok> There we go.
Dec 16 20:44:38 <zmatt> it is nowadays
Dec 16 20:44:46 <zmatt> I think
Dec 16 20:44:57 <zmatt> actually not sure
Dec 16 20:45:15 <zmatt> I see I only have EOL comments on lines that are commented out
Dec 16 20:46:03 <Ragnorok> I guess i have to set the PID as well.  It's getting the wrapper script PID.  That at least makes sense.
Dec 16 20:46:37 <zmatt> ot use NotifyAccess=all
Dec 16 20:46:39 <zmatt> *or
Dec 16 20:46:57 <Ragnorok> That wold probalby be simpler.  It's not like there's more than one process anyway.
Dec 16 20:46:59 <zmatt> changing the mainpid is probably the wrong thing to do if your wrapper script continues to live
Dec 16 20:47:05 <zmatt> oh it doesn't?
Dec 16 20:47:14 <zmatt> can't you use exec in your wrapper script then?
Dec 16 20:48:25 <zmatt> otherwise you will have to change the MainPID before your wrapper exits, since I'm pretty sure the service will stop when the main process exits
Dec 16 20:48:30 <Ragnorok> It does actually.  I left it that way b/c it was that way to start with and I didn't want to do more work.  I just set NotifyAccess to all.
Dec 16 20:48:31 <zmatt> crap, brb, shopping
Dec 16 21:07:54 <zmatt> back
Dec 16 21:09:38 <Ragnorok> It's running now but it won't stop properly.  Normally the start wrapper is backgrounded with & and when the stop wrapper is called it does some tidying up and exits.  Now when systemd calls the stop wrapper everything just dies.
Dec 16 21:10:33 <zmatt> normally systemd just expects services to handle SIGTERM, but you can configure a custom ExecStop if necessary
Dec 16 21:10:44 <Ragnorok> I did configure one.
Dec 16 21:11:11 <Ragnorok> That would be the stop wrapper that's not working the way it did before this whole service thing.
Dec 16 21:11:21 <zmatt> what does it do?
Dec 16 21:12:44 <Ragnorok> The stop wrapper checks that another process is stopped and then does a SIGINT on the main one.  The start wrapper captures the exit code and does differnt things depending on what it got.
Dec 16 21:13:10 <zmatt> okay sounds like stuff that systemd would normally do for you
Dec 16 21:13:16 <zmatt> (you can change the SIGTERM to SIGINT btw)
Dec 16 21:13:33 <Ragnorok> Except I see no way to have systemd do different things based on return code.
Dec 16 21:13:47 <zmatt> it depends a bit on how many different things I guess
Dec 16 21:14:51 <Ragnorok> Three, one of which is run a script to modify if it should run that or one of the other two.
Dec 16 21:15:35 <Ragnorok> This whole complex thing is based on trying to get it do something moderately repeatable based on battery operation, or wall operation.
Dec 16 21:15:38 <zmatt> anyway, if your start wrapper remains running then it should definitely the main process, earlier you implied it didn't when you said "It's not like there's more than one process anyway"
Dec 16 21:16:03 <Ragnorok> I wasn't thinking about the wrapper.  My bad.
Dec 16 21:16:34 <Ragnorok> If I can get it to shut off cleanly I may be about where I want to be.
Dec 16 21:17:03 <zmatt> when the main process exits, everything gets killed by systemd... you wouldn't want any processes you spawned to cause systemd to continue considering your service to be "running" after the main process exits
Dec 16 21:17:41 <Ragnorok> Right.  In this case "main" would be the wrapper, right?  That's what systemd is launching.
Dec 16 21:17:50 <zmatt> yes, so the default should be fine
Dec 16 21:18:19 <zmatt> hmm, maybe your stop-wrapper exits before its work takes effect?
Dec 16 21:19:06 <zmatt> wait, I'm dumb
Dec 16 21:19:09 <Ragnorok> My expectation was when I call the stop wrapper, systemd will wait for the main process to exit.  It sounds like the stop wrapper should wait instead.
Dec 16 21:19:22 <zmatt> ExecStop is performed after the service exits, not as a mechanism to stpo the service
Dec 16 21:19:29 <Ragnorok> Then systemd will see everything is shut down when I issue a stop.
Dec 16 21:19:32 <zmatt> I think
Dec 16 21:19:32 <zmatt> or not
Dec 16 21:19:36 <zmatt> maybe I'm confused
Dec 16 21:19:40 <zmatt> I've never used that option I think
Dec 16 21:19:51 <zmatt> no, never mind
Dec 16 21:19:52 <Ragnorok> Commands to execute to stop the service started via ExecStart=
Dec 16 21:19:59 <zmatt> it does contain commands to stop the service
Dec 16 21:20:02 <zmatt> yeah
Dec 16 21:20:16 <Ragnorok> Ah.  Next sentence says this needs to wait or it'll kill everything.
Dec 16 21:21:27 <zmatt> btw, you can have an ExecStopPost which uses the exit code of the main process to perform some action, so that could act as a substitute for a wrapper
Dec 16 21:22:31 <Ragnorok> Ah.  So I could split up the start wrapper and put the clean-up bits in there, seems like.
Dec 16 21:22:34 <zmatt> but yeah it would require some rework
Dec 16 21:22:49 <Ragnorok> Been doing that all converstaion.  (shrug)
Dec 16 21:23:50 <Ragnorok> I don't see a way to get the exit status like I can in bash.  Did I miss that?
Dec 16 21:23:59 <zmatt> it's just different models... the degree of management done by systemd isn't quite compatible with the freedom, lack of oversight, and requirement to manage things manually you have with a bunch of a scripts
Dec 16 21:24:29 <zmatt> read the last paragraph of the description of ExecStopPost.. oh wait maybe that's not in v215, lemme check
Dec 16 21:24:46 <zmatt> it might not be
Dec 16 21:25:04 <zmatt> either it wasn't implemented yet or it wasn't documented :P
Dec 16 21:25:10 <Ragnorok> I didn't get that from "environment variable substitution".
Dec 16 21:26:47 <zmatt> yeah I meant the last paragraph of the description of ExecStopPost... in the documentation of the version I have on my system :P
Dec 16 21:26:51 <zmatt> (see e.g. https://manpages.debian.org/testing/systemd/systemd.service.5.en.html )
Dec 16 21:27:17 <zmatt> I'm guessing it wasn't available yet in v215
Dec 16 21:27:51 <Ragnorok> Guess I'll find out.  I can always hack it with an env var.
Dec 16 21:28:10 <Ragnorok> Wish they put this in alpha order.
Dec 16 21:28:59 <zmatt> just use dbus to query systemd for the exit code ;-)
Dec 16 21:29:01 * zmatt ducks and runs
Dec 16 21:29:30 <Ragnorok> lol
Dec 16 21:44:24 <Ragnorok> Every time I start my service I get a dump of every journalctl entry in the system.  I'm just doing sd_journal_printv().  It's not complicated.
Dec 16 21:44:43 <zmatt> ???
Dec 16 21:45:32 <Ragnorok> My process logs to the journal using sd_journal_printv.  Every time it start (using the service) I get a full dump of all entries.
Dec 16 21:45:56 <zmatt> get a full dump of all entries where?
Dec 16 21:46:04 <Ragnorok> I have journalctl -f running in another window.
Dec 16 21:46:15 <zmatt> ehmmmmmm
Dec 16 21:46:21 <zmatt> are you sure your system isn't haunted?
Dec 16 21:46:32 <Ragnorok> Ok.  Not EVERY time.  Just some times.
Dec 16 21:47:15 <Ragnorok> It could be.  I'm working on it.  My very presence could be considered haunting.
Dec 16 21:47:46 <zmatt> have you considered getting an updated systemd from jessie-backports? :P
Dec 16 21:48:27 <zmatt> I don't really know how to help you here, I've never seen or heard or anything even remotely as strange as what you've been describing
Dec 16 21:48:58 <zmatt> although it seems more likely that journalctl is somehow getting really confused (and is dumping lines it already dumped) than that there's actually stuff being logged
Dec 16 21:49:30 <zmatt> actually given the weird behaviour you mentioned from journalctl -f
Dec 16 21:50:43 <zmatt> try adding to journalctl:  -b -0 -o short-monotonic
Dec 16 21:52:55 <Ragnorok> I really hate automagic.  Everything I do leads to ever more hoop-jumping to try to get where I stared w/o the automagic.  This always happens.
Dec 16 21:53:11 <Ragnorok> That command doesn't tail.  It does the same as journalctl -b.
Dec 16 21:53:28 <zmatt> I meant in addition to -f
Dec 16 21:54:13 <Ragnorok> For one run it's worked well.  I'll keep using it and see what happens.
Dec 16 21:57:24 <Ragnorok> It looks like systemd nuked my pid file when I stopped the service.
Dec 16 21:57:57 <zmatt> why would systemd touch your pid file, or even know it exists
Dec 16 21:58:28 <Ragnorok> Nah, for some reason it's not creating one any longer.  Nice
Dec 16 21:58:55 <Ragnorok> Commented out too many lines.
Dec 16 22:09:43 <Ragnorok> I really feel this is a waste of time.  I can't get this to do what I want b/c the post stop doesn't have the data I need.  Tomorrow I'm going to hope I can put all the crap back together and get the watchdog fd to close on process exit.  That's all I really needed.  But at least I know a lot more about systemd, for that's worth.  Thanks for all your suggestions.
Dec 16 22:11:24 <zmatt> yep that makes a lot of sense, given the headaches you're having with the ancient systemd and the awkward fit of your existing scripts into a systemd service
Dec 16 22:11:50 <Ragnorok> Well I had to try, you know.  There it is.  Have an evening.  Or whatever it is there.
Dec 16 22:12:04 <zmatt> "get the watchdog fd to close on process exit" sounds like an odd thing, since all fds of a process are closed when it exits
Dec 16 22:12:18 <zmatt> maybe you were previously just not waiting for the process to have exited before trying to open the watchdog device?
Dec 16 22:12:33 <Ragnorok> I mean make sure it's closing.  I've seen it not.  Unless wait pid doesn't really wait.
Dec 16 22:12:50 <zmatt> it should
Dec 16 22:13:05 <zmatt> also make sure you don't accidently leak the fd, although that shouldn't be possible if you don't have any child processes
Dec 16 22:13:14 <zmatt> just to be sure, make sure the fd is cloexec
Dec 16 22:13:26 <zmatt> (O_CLOEXEC should be used by default anyway)
Dec 16 22:14:25 <Ragnorok> That's also what I meant by make sure it's closed.  Use that.  (grin)  Thanks again!  L8r...
Dec 16 22:14:36 <zmatt> good night!
**** BEGIN LOGGING AT Mon Dec 16 22:39:25 2019
**** ENDING LOGGING AT Tue Dec 17 02:59:58 2019