16:30:06 <djmitche> #startmeeting weekly
16:30:06 <bb-supy> Meeting started Tue Sep  6 16:30:06 2016 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:06 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:30:06 <bb-supy> The meeting name has been set to 'weekly'
16:30:12 <djmitche> #topic Introduction
16:30:16 <djmitche> #nick bdbaddog gracinet
16:30:28 <djmitche> #link https://titanpad.com/buildbot-agenda
16:31:19 <djmitche> who else is here?  looks like a pretty slim agenda
16:31:53 <tardyp> hi
16:32:12 <djmitche> Hi!
16:32:22 <djmitche> Nothing is jumping out at me from the weekly-updates email
16:32:25 <djmitche> #topic Week In Review
16:32:29 <djmitche> anything to highlight?
16:32:45 <tardyp> I'd like to discuss buildbotNetStatistics: https://github.com/buildbot/buildbot/pull/2393
16:33:56 <djmitche> great
16:34:12 <tardyp> once this lands, I'd like to release rc3
16:34:15 <djmitche> #info Pierre has been working on the statistics reporting discussed last week
16:34:17 <djmitche> ++
16:34:42 <skelly> what's left for final release?
16:34:52 <tardyp> skelly: not much
16:35:56 <djmitche> does http://trac.buildbot.net/ticket/3604 block the final release?
16:36:01 * djmitche looking at patch
16:36:25 <gracinet> is there a big difference currently between 0.9.0 branch and master ?
16:36:27 <tardyp> I'd like to see people install rc3, so that we can know it works before we can say its good for release (and the stats serer is there to have an idea of who installed it)
16:36:35 <tardyp> 3604 is a few hour of work
16:36:45 <tardyp> it can happen after actually
16:37:00 <bdbaddog> Is there a way to keep track of which installs report so we don't double count?
16:37:02 <tardyp> gracinet: there is the hyper work, and refacto of the latent workers
16:37:25 <gracinet> tradyp, ok, gotta catch up on hyper (not right now)
16:37:28 <tardyp> I was planning to look at source IP
16:37:45 <skelly> how will NAT affect that?
16:38:02 <tardyp> we will only see one IP
16:38:38 <tardyp> I dont want to expose internal IP address or master hostnames in the data
16:38:52 <tardyp> maybe we can hash the hostname though
16:39:00 <bb-github> [13buildbot] 15djmitche commented on pull request #2393 144a5a9ca: If we're tracking by source IP, we should mention it here! 02https://git.io/viZ2l
16:39:02 <djmitche> Could we have the master generate a uuid and cache it in its master directory or the DB?
16:39:10 <bdbaddog> you see the ip of the BB server, which could be internal IP, and could conflict with others.. given a lot of small shops run in 192.168.1.x,
16:39:10 <tardyp> yeah
16:39:21 <bdbaddog> exactly a UUID.
16:39:27 <tardyp> master already does something similar  for multimaster
16:39:30 <bdbaddog> what I wast hoping for.
16:39:36 <gracinet> +1 for UUID indeed
16:39:37 <tardyp> we can just hash the master identifier
16:39:37 <djmitche> tardyp: that'd be even better
16:39:42 <djmitche> ++
16:39:53 <tardyp> I would like to avoid doing db upgrade at that point
16:40:00 <djmitche> yes, that makes sense
16:40:39 <gracinet> If there's already some kind of uuid, no need to make yet another one, unless you'd want one per multi-master setup of course
16:41:14 <djmitche> #info discussion of how to track unique installs, so we can distinguish two checkins from the same master
16:42:02 <djmitche> #info plan to release rc3 with the BuildbotNetStatistics support installed, to test the service and also see how many rc3 installs are running
16:42:23 <bb-github> [13buildbot] 15tardyp commented on issue #2393: comments from meeting: need to add unique identifier to the request that identifies a master install 02https://git.io/viZ25
16:42:30 <tardyp> there are a few fixes in rc3 also
16:43:03 <djmitche> #info rc3 also has some bugfixes and other improvements
16:43:13 <djmitche> ok
16:43:29 <djmitche> #topic 0.8.x release
16:43:38 <djmitche> I had a query directly to me via email about the docs for 0.8.14
16:43:57 <djmitche> sa2ajj: is p1 coming soon? :)
16:44:10 <bdbaddog> Probably a good idea to send a blast to mailing list about BuildbotNetStatistics prior to RC3?
16:44:44 <tardyp> bdbaddog: will do
16:45:30 <djmitche> #agreed will notify the mailing list about the BuildbotNetStatistics service before releasing rc3
16:45:30 <gracinet> I'm wondering, not so much at ease with calling home in general, but it would make sense to backport it to 0.8.x, isn't it ?
16:45:51 <djmitche> hm
16:45:53 <djmitche> not a bad idea
16:45:58 <djmitche> at least to 0.8.14p1
16:46:03 <gracinet> I know there are a lot of not up-to-date 0.8 so that would defeat the purpose
16:46:09 <skelly> so much for p1 coming soon then ;)
16:46:33 <tardyp> that would be pretty easy, the code is generic enough
16:46:41 <gracinet> oh no, don't want to delay anything, it can make sense afterwards anyway
16:46:50 <djmitche> yeah, worth considering
16:47:06 <djmitche> #info considering adding the BuildbotNetStatistics service to the latest 0.8.x too
16:47:17 <gracinet> like, "we are 6months after 0.9.0 final, there are still xx% reported 0.8.14 out there "
16:47:17 <djmitche> although I really can't see those stats being reliable
16:47:21 <djmitche> yeah
16:48:04 <djmitche> OK, let's talk TLS
16:48:08 <djmitche> #topic TLS Support
16:48:13 <djmitche> gracinet: take it away :)
16:49:02 <gracinet> ok, so the thing is : if the worker were using the standard endpoint connection system, then we'd have TLS and other interesting options basically for free
16:49:37 <gracinet> last week, tardyp pointed me at an integration test (test_worker_comm) to play with, and I started experimenting
16:50:22 <gracinet> that test uses actually its own simplified worker, but it demonstrated easily TLS capabilities (that's what currently in the PR)
16:50:33 <gracinet> (PR 2387)
16:50:40 <gracinet> except for issues in Windows
16:50:59 <djmitche> what are the windows issues?
16:52:03 <gracinet> I didn't investigate them yet (it's in the appveyor report https://ci.appveyor.com/project/djmitche/buildbot/build/1.0.1412)
16:52:39 <djmitche> ah, ok
16:52:49 <djmitche> that integration test is a great way to work on this issue, though!
16:53:39 <gracinet> Then I turned to actually change the code in the worker
16:54:19 <gracinet> getting the integration test to work wasn't easy (that's me being not so twisted-fluent)
16:54:38 <gracinet> and in real life it worked right away
16:54:56 <gracinet> the thing is, most of the connecting / reconnecting code has to be just dropped
16:55:29 <gracinet> and while that wasn't to the best of my knowledge tested at all, I can't imagine doing it without tests
16:55:49 <djmitche> dropped because using endpoints is much easier?
16:55:55 <gracinet> so, in short, good progress, a major refactor in which many duties are actually transferred to twisted
16:56:02 <djmitche> that's great!
16:56:05 <gracinet> yes, djmitche, that's about it
16:56:09 <djmitche> I think this integration test is the main test for that functionality
16:56:12 <tardyp> gracinet: I think at the time of writting
16:56:13 <djmitche> it's hard to test any other way really
16:56:18 <tardyp> reconnectingfactory wasn't in tw
16:56:36 <tardyp> and at that time, we weren't so much into unit testing
16:57:29 <gracinet> djmitche: the existing integration test actually implements its a full simplified PB factory etc from scratch
16:57:50 <gracinet> so it's more about testing behaviour of the master wrt to worker attaching/detaching than something else
16:58:13 <tardyp> yes, test_worker_comm is not really integration test
16:58:22 <tardyp> there is too much mocking in that test
16:58:26 <gracinet> not of the worker anyway :-)
16:58:48 <djmitche> hm
16:59:08 <djmitche> so I guess the question is, how best to test that before changing it/
16:59:09 <djmitche> ?
16:59:31 <gracinet> I'll provide a new test, exerting the actual worker code
16:59:44 <gracinet> actually I have one right now, entangled with test_worker_comm
17:00:36 <gracinet> so, I'd stay that this still needs more work, which I'm ok to do
17:00:52 <tardyp> sounds great!
17:01:05 <gracinet> because there's not a single line changing in the master code, the end result will have a broad compatibility range
17:01:22 <djmitche> haha, sounds great :)
17:01:36 <gracinet> but I wouldn't release it with buildbot-worker 0.9.0
17:02:07 <djmitche> ok
17:02:10 <gracinet> what's left to do is
17:02:10 <gracinet> - cleanup expand push new test
17:02:31 <djmitche> #info lots of progress on using endpoints, which open up all of the network support of twisted (TLS, v6, systemd sockets, etc.)
17:02:36 <gracinet> - investigate TLS issues with Windows and (maybe) certificate validation (I cheated)
17:02:37 <djmitche> #info will not be in 0.9.0 though
17:02:42 <djmitche> haha
17:03:04 <gracinet> well, I supplied a cert for localhost, together with the authority used to sign it
17:03:17 <gracinet> but did not ask for verification
17:03:22 <djmitche> right
17:03:48 <gracinet> and, last item to be done, but not least, reimplement features that aren't part of Twisted's reconnection options for services
17:04:00 <gracinet> keepalive, graceful shutdown
17:05:17 <gracinet> I realised afterwards, btw : systemd sockets are meaningful for listeners (i.e., the master), and you can already use them I suppose
17:05:57 <djmitche> I suppose that's like xinetd?
17:06:41 <gracinet> Not familiar with that, but yes it draws ideas from inetd also
17:06:53 <gracinet> Typically, systemd would bind the socket, then pass it down to the managed daemon
17:07:48 <gracinet> with the option to actually start the daemon only if something comes down that socket (and they use that also if I understand well to get on-demand services, even if it's not a network service)
17:08:08 <djmitche> I see
17:08:11 <bdbaddog> sounds very similar to xinetd…
17:08:20 <djmitche> so not terribly useful for buildbot, since the master needs to be running even when there are no connections
17:08:27 * gracinet reading wikipedia for xinetd
17:08:41 <djmitche> xinetd is pretty much the same as inetd, with a better config format
17:10:27 <djmitche> ok, well, that's a great update
17:10:32 <djmitche> if there's nothing else, we can wrap up
17:10:34 <gracinet> I don't know, maybe systemd provides some useful management options that twisted doesn't
17:11:37 <cmouse> well, systemd has socket activation
17:11:46 <cmouse> which is basically same as (x)inetd without stdin
17:13:32 <djmitche> ah, cool
17:13:41 <djmitche> anyway, I'm sure someone smart can hook that up with endpoint syntax :)
17:13:46 <djmitche> #endmeeting