16:04:46 <djmitche> #startmeeting weekly
16:04:46 <bb-supy`> Meeting started Tue Jun 20 16:04:46 2017 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:04:46 <bb-supy`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:04:46 <bb-supy`> The meeting name has been set to 'weekly'
16:04:49 <djmitche> hi everyone :)
16:04:54 <warner> howdy
16:04:58 <djmitche> i don't get gcal notifications now that it's a separate calendar :/
16:05:02 <djmitche> #topic Introduction
16:05:05 * djmitche waves to warner
16:05:11 <rjarry> ola
16:05:22 <djmitche> https://docs.google.com/document/d/1SAiv3Tk3lACPFVaCNExTjeU7z7NKCKCfP9_2AZ5lpjE/edit - agenda
16:05:39 <djmitche> #topic Week in Review
16:05:46 <djmitche> tardyp: all yours :)
16:07:27 <djmitche> https://github.com/buildbot/buildbot/pull/3333 -- interesting, allowing masters to run with an in-memory DB
16:07:30 <tardyp> finding the report email
16:07:32 <djmitche> I'm curious how that's useful
16:07:52 <tardyp> yes it is, it allows to remind what happen
16:08:01 <tardyp> actually the week was quite active
16:08:04 <tardyp> after the release
16:08:19 <djmitche> I meant how an in-memory DB is useful -- the weekly email I find very useful!
16:08:37 <rjarry> djmitche: about that sqlite :memory: feature, I strongly discourage using that in production
16:08:59 <djmitche> me too!!
16:09:04 <tardyp> indeed, this can be useful for developer environment though
16:09:09 <djmitche> yep
16:09:25 <tardyp> so we got lots of PR, but the bigger one are still in the open Pipe
16:09:46 <tardyp> stash
16:09:52 <tardyp> hook refactorization
16:10:10 <tardyp> also discussion on IRC on how we can improve the hooks to get more info from the REST apis
16:10:19 <rjarry> tardyp: btw, I've still got to get unit tests landing on grid_view
16:10:24 <tardyp> which requires the hook refactorization to make it async
16:10:35 <tardyp> grid view from rjarry is another big one
16:10:37 <djmitche> #info New PRs for stash (bitbucket) support
16:10:45 <rjarry> I think this should be ready tomorrow
16:10:52 <tardyp> secret providers is nearly finished
16:10:56 <djmitche> #info and hooks
16:11:04 <djmitche> rjarry: that's awesome :)
16:11:17 <tardyp> also I tried to join you djmitche about reclaim and unclaimexpired
16:11:18 <djmitche> #info almost finished with grid view and secret providers
16:11:28 <djmitche> oh, I'll have a look
16:11:37 <rjarry> djmitche: thanks :) actually, that was a lot less work that I expected
16:11:38 <tardyp> unclaimexpired is not called in nine, so I removed it in https://github.com/buildbot/buildbot/pull/3346
16:11:59 <tardyp> rjarry: I am glad that you found the JS development workflow not so hard
16:12:24 <djmitche> #info And 0.9.8 was released!
16:12:35 <rjarry> tardyp: your live coding video helped a lot :)
16:12:40 <djmitche> awesome
16:12:40 <tardyp> nic
16:12:47 <djmitche> anything else new this week?
16:12:55 <tardyp> so we got a lot of activity on IRC
16:13:15 <tardyp> lots of new people who are looking like building pretty large scale CI
16:13:44 <tardyp> kfunk today annouceed that he is porting is 450+ builders from eight to nine
16:13:48 <djmitche> wow
16:13:52 <djmitche> ++
16:13:56 <tardyp> rjarry: is in that level of scale as well
16:14:17 <djmitche> I hope those deployments are smooth - that's awesome
16:14:31 <djmitche> warner: we've come a long way :)
16:14:34 <tardyp> s0undt3ch: looks like pretty good scale also
16:14:49 <warner> warms my heart :)
16:14:56 <tardyp> and frodox is quite active as well
16:14:57 <djmitche> #info there are several large-scale deployments planned for 0.9.8
16:15:00 <rjarry> heheh
16:15:13 <djmitche> #topic Want To Write A Book?
16:15:16 <tardyp> so its cool to see so much people here
16:15:17 <djmitche> warner: ^^ take it away..
16:15:20 <djmitche> indeed!
16:15:22 <warner> hey folks.. for those who don't know me, I wrote the first version of buildbot a zillion years ago. I haven't been around much for the last 5, er, 10 years.
16:15:23 <warner> 
16:15:28 <warner> I stopped by to see if anyone would be interested in writing a chapter about buildbot for an upcoming book on Twisted
16:15:34 <warner> at PyCon last month I met an editor at Apress named Todd Green, and he's trying to assemble people to write one chapter each on different projects that use Twisted. It could be kind of like the Architecture of Open-Source Applications book, except it'd be twisted-focussed
16:15:36 <s0undt3ch> mine is only growing bigger ;)
16:15:40 <warner> there will probably also be some introductory material on Twisted basics at the start, and then some more chapters on libraries in the twisted ecosystem (like treq and automat, probably some database libs), and hopefully some future-looking material at the end (py3, asyncio).. he's looking for authors for those too
16:15:45 <warner> I've agreed to write chapters on Tahoe-LAFS and magic-wormhole
16:15:52 <warner> but I think Buildbot is still one of the largest (and most widely used) Twisted-based applications out there, and the broad range of protocols it needs to speak serves as a great example for why you need something like Twisted
16:15:57 <warner> (imagine trying to handle all the sorts of status clients and worker nodes and changesources and stuff with just a bunch of threads and semaphores :)
16:16:03 <warner> so anyways, if this sounds interesting to you, i can connect you with Todd.. just drop me an email. warner@lothar.com
16:16:06 <tardyp> are you really typing that fast?
16:16:17 <rjarry> lol, I was gonna ask that :D
16:16:22 <warner> nope :)
16:16:40 <warner> pre-prepared remarks :)
16:16:44 <rjarry> warner: nice to meet you anyway :)
16:16:53 <warner> rjarry: :)
16:17:17 <tardyp> it looks like a very good project
16:17:24 <warner> so I think the book could be pretty interesting, and it might be a good opportunity for someone who wants to get some exposure and/or show off their writing skills
16:17:38 <warner> and I think the Twisted world could use more information about buildbot and how/why it works
16:17:50 <tardyp> I'd be interrested, but I am just starting a new job, and already the agenda is starting to fill very fast
16:17:58 <warner> a lot of the problem with learning Twisted is being motivated, since many toy problems/projects could be implemented without it
16:18:32 <warner> whereas buildbot is a good example of one which really does need that level of protocol support and "concurrency"/asynchronousness/ whatever you want to call it
16:18:38 <warner> tardyp: yep, I hear ya
16:18:38 <djmitche> #info open opportunity to write a chapter about Buildbot in a book on Twisted
16:18:43 <rjarry> tardyp: I hope you have air conditioning at your new job
16:18:49 <djmitche> warner: the chapter could probably highlight some of the growing pains, too
16:19:02 <tardyp> s0undt3ch last week talk on how buildbot was master than Jenkins.
16:19:04 <djmitche> callbacks -> inlineCallbacsk
16:19:16 <tardyp> I think twisted is a lot responsible for that
16:19:25 <warner> yeah, it could focus on the "why", or the "how", or changes over time
16:19:45 <warner> if y'all know of other twisted-related projects, Todd is looking for more chapters too, not just buildbot
16:19:59 <warner> or projects which evaluated twisted but then chose to use something else, that would be pretty valuable too
16:20:00 <tardyp> could it be a team project?
16:20:02 <warner> yeah, sure
16:20:14 <djmitche> if it's a team thing, I could help add some of the historical perspective
16:20:19 <djmitche> but I don't want to lead it :)
16:20:30 <tardyp> I think if you ask to each of us, that would be difficult, but if we make it a team, we might manage to get something out
16:20:44 <djmitche> agreed
16:20:59 <warner> I haven't talked specifics with him like royalties or anything, but writing books doesn't generally pay very well anyways, so small$$ / len(authors) is still roughly== small$$ :)
16:21:19 <tardyp> ah. yes
16:21:28 <djmitche> I suspect the best option would be to pipe those proceeds to Conservancy
16:21:35 <tardyp> I think this is good for OSS to get a book
16:21:39 <djmitche> however small they might be :)
16:21:51 <djmitche> yeah, and BB is at an age where a book makes a lot of sense
16:22:08 <tardyp> we got the feedback from time to time: where is the BB book?
16:22:11 <warner> it could also be the jumping-off point for a book just about buildbot
16:22:15 <tardyp> there are already tons about Jenkins
16:22:38 <tardyp> yes, a first contact with the edition ecosystem
16:22:38 <warner> you might talk to Todd about that.. I dunno what he's looking to tackle, but it's worth a shot
16:23:29 <djmitche> tardyp: are there people who have been active in the project recently that you think might like to lead the chapter authorship?
16:23:33 <warner> I know the guy who runs No Starch press, I'd be happy to introduce somebody here to him as a buildbotbook person
16:23:34 <tardyp> so do we have already the twisted chapters?
16:24:07 <warner> none of this twisted book is written yet, Todd is still trying to gather ideas/titles/outlines/people together
16:24:18 <tardyp> I think we are at a phase of the project where we have more and more very good potencial contributors
16:24:19 <warner> it's in the planetary nebulae phase of book-formation :)
16:24:40 <tardyp> but I'd say most of them are already too shy to step in more heavily
16:24:49 <tardyp> still too shy I mean
16:25:13 <tardyp> so what is the planning ?
16:25:33 <tardyp> like we need to write it for 2017? Q12018? Q2?
16:26:11 <warner> dunno, that's still up in the air too
16:26:36 <djmitche> so maybe concretely, I can send a note to devel@ asking for authors
16:26:38 <warner> the overall outline is just something he and I sketched out at pycon last month :)
16:26:44 <djmitche> and we can all keep an eye out for a good match?
16:26:49 <warner> yeah, that's probably the best next step
16:26:52 <djmitche> ok
16:27:03 <djmitche> #action djmitche to send a note to devel@ asking for authors
16:27:11 <tardyp> good!
16:27:18 <warner> if anyone reading the log of this meeting thinks this sounds interesting, please email me (warner@lothar.com) and I'll help you get in touch with Todd
16:27:23 <djmitche> awesome :)
16:27:27 <warner> thanks!
16:27:28 <djmitche> any other topics for today?
16:27:33 <djmitche> thanks for dropping by :)
16:27:46 <tardyp> about event.buildbot.net
16:27:51 <djmitche> #info Discussion of a Buildbot-specific *book*, but let's write the chapter first
16:28:03 <djmitche> #topic events.buildbot.net
16:28:22 <tardyp> some people ask access to the data
16:28:38 <tardyp> how about putting it "open data"
16:29:00 <tardyp> The whole db  turns into a 250MB json
16:29:36 <djmitche> what are the privacy implications?
16:30:11 <tardyp> there is no private data in there only the instanceids
16:30:21 <djmitche> ok
16:30:31 <djmitche> and we have this opt-in/opt-out approach, with the default being minimal info, right?
16:30:46 <djmitche> so we've basically told users this info is public already -- it's just limited to project contributors at the moment?
16:30:46 <tardyp> yes
16:30:56 <tardyp> there are several options.
16:31:02 <djmitche> bdbaddog1: ^^ thoughts on this?
16:31:07 <tardyp> we can give access on demand to kibana
16:31:22 <bdbaddog1> can you filter fields (aka ACL access)?
16:31:24 <tardyp> or we can put the json available to everyone linked from buildbot.net
16:31:51 <tardyp> no. if you give access to kibana, it has to be to trusted people, as they'll have write access
16:32:29 <bdbaddog1> hrm… could we push to a google sheet and filter fields like IP in the process? then make it read only to the world?
16:32:37 <tardyp> example data event:
16:32:41 <tardyp> https://www.irccloud.com/pastebin/jxed5sw0/
16:33:11 <tardyp> there are 185080
16:33:13 <tardyp> events
16:33:33 <tardyp> thas the kind of open data where excel doesn't shine
16:33:39 <bdbaddog1> gotcha.
16:33:40 <djmitche> i do worry about the IPs
16:33:49 <djmitche> maybe we could do a filtered dump with IPs hashed?
16:33:52 <tardyp> the IP are our internal IP
16:33:55 <djmitche> oh
16:33:58 <bdbaddog1> can we pull the json and filter IP and instance ID?
16:34:14 <tardyp> yes, we would have to remove that IP
16:34:21 <tardyp> instanceID is necessary imho
16:34:26 <djmitche> so the instlalId has no useful info in it (just an identifier, useful for determining duration of installation)
16:34:29 <bdbaddog1> by our do you men the installation, or the endpoint to the call home logic
16:34:30 <tardyp> this is what identify the evolution of an install
16:34:38 <djmitche> and the IPs are basically always the same
16:34:49 <bdbaddog1> by our IP i meant
16:34:49 <tardyp> so that you can know how old  is the instance
16:35:14 <djmitche> bdbaddog1: yeah, the server end of the call-home logic
16:35:15 <tardyp> yes it is the internal IP of events.buildbot.net in our network
16:35:21 <bdbaddog1> gotcha.
16:35:24 <tardyp> it is added by logstash automatically
16:35:34 <djmitche> in that case I'm happy to publish this as-is
16:35:41 <bdbaddog1> +1
16:35:42 <tardyp> we can remove all logstash generated data
16:35:54 <djmitche> it may be worth adding a link to the docs where users choose what level of opt-in they want
16:35:56 <bdbaddog1> so pull json from logstash, publish on buildbot.net ?
16:36:20 <bdbaddog1> maybe gzip'd json?
16:36:32 <djmitche> #info discussion of making the Buildbot usage data public, for download from buildbot.net
16:36:33 <tardyp> so we can do is to have a cronjob which creates the json every week from elasticsearch
16:36:41 <tardyp> and clean it up
16:36:42 <bdbaddog1> +1
16:36:52 <tardyp> then it will publish on ftp or anywhere on our infra
16:36:53 <djmitche> yeah +1 from me too
16:37:02 <tardyp> then we link it in buildbot.net
16:37:10 <bdbaddog1> maybe each json is a weeks data?
16:37:26 <tardyp> maybe indeed
16:37:35 <djmitche> I suspect most uses of this are incidental
16:37:41 <djmitche> I'm kind of curious why anyone but us cares, to be honest
16:37:43 <tardyp> with a cgi to cat all that
16:37:54 <djmitche> so probably that's more work than it's worth
16:38:08 <tardyp> It allows to answer: how many master are using windows and p4
16:38:25 <tardyp> is buildbot a right choice for windows and p4
16:38:31 <djmitche> hm, good point
16:38:50 <tardyp> maybe we could have some student taking this data and generate report
16:38:59 <djmitche> well I think my only advice would be, do what's simplest and expect users of the data to put it in the shape they want
16:39:18 <djmitche> a cgi to combine multiple files seems like more work than necessary :)
16:39:28 <tardyp> I plan to also release a jupyter notebook which loads the data
16:39:33 <djmitche> oh, I like that
16:39:41 <tardyp> I already have it
16:40:31 <djmitche> #agreed will make the event data public and available for download, with a Jupyter notebook to load the data
16:40:48 <djmitche> well, we're 12 minutes over time -- should we wrap up?
16:40:55 <tardyp> yep
16:40:56 <tardyp> thanks
16:41:00 <rjarry> o/
16:42:27 <djmitche> #endmeeting