16:31:33 <djmitche> #startmeeting weekly
16:31:33 <bb-supy> Meeting started Tue Aug  2 16:31:33 2016 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:31:33 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:31:33 <bb-supy> The meeting name has been set to 'weekly'
16:31:38 <djmitche> I got distracted :)
16:31:41 <djmitche> #topic Introduction
16:31:55 <djmitche> Agenda: https://titanpad.com/ep/pad/view/buildbot-agenda/aW9h9kMAWD
16:32:00 <tardyp> well now it is done .. https://buildbot.buildbot.net/
16:32:03 <tardyp> down
16:32:14 <tardyp> nm lets go for the meeting
16:32:22 <djmitche> ok
16:32:29 <djmitche> roll-call -- who else is around?
16:32:34 <djmitche> sa2ajj by chance? rutsky__?
16:32:43 <djmitche> bill's not here :(
16:32:43 <rutsky> hi!
16:32:53 <verm__> hey
16:33:15 <djmitche> yay!
16:33:21 <djmitche> ok, getting started
16:33:22 <sa2ajj> djmitche: i am
16:33:29 <sa2ajj> i'm not sure for how long though :/
16:33:29 <djmitche> wow, big week :D
16:33:33 <djmitche> #topic week in review
16:33:41 <djmitche> metabuildbot PR#58 - recreate the virtualenv if it is too old
16:33:55 <djmitche> This is helping to eliminate "stale" virtualenvs in the metabuildbot, right?
16:33:58 <djmitche> and maybe also save space?
16:34:05 <djmitche> #info metabuildbot PR#58 - recreate the virtualenv if it is too old
16:34:19 <sa2ajj> how long a venv creation takes?
16:34:27 <tardyp> well this is more to fix automatically bad environments and avoid botheing the owner of the workers
16:34:44 <sa2ajj> i mean unless it's really critical to be really fast, then we need optimise it
16:34:52 <tardyp> because of those  issues with ansible, I haven't be able to measure much
16:34:52 <sa2ajj> otherwise i think we just should create every time
16:35:16 <tardyp> I know that pip did some optimizations recently, but I am not sure how much we can count on the real infra
16:35:51 <sa2ajj> e.g. in one of our local projects 2mins is considered "long"
16:36:19 <djmitche> buildbot.buildbot.net is running again (I ran 'supervisorctl restart buildmaster')
16:36:25 <sa2ajj> and then they optimise (and make errors in those optimisations)
16:37:01 <sa2ajj> if we want to optimise, we might very well have a nightly job to create tarballs for those venvs and then just unpack them
16:37:13 <djmitche> that's tricky with different pythons
16:37:16 <sa2ajj> (virtualenv is notorious for not being relocatable though)
16:37:19 <djmitche> right
16:37:24 <djmitche> I think this is a good short-term fix
16:37:33 <djmitche> let's see how it works out now that metabuildbot is running again
16:37:40 <tardyp> I d like to try it and see how it works
16:37:49 <djmitche> I think optimization is a secondary goal
16:37:51 <sa2ajj> another option could be to run 'pip install -U --dry-run' and see if it reports anything
16:37:52 <djmitche> if that
16:38:02 <tardyp> I am more interrested in stability
16:38:08 * sa2ajj nods
16:39:30 <sa2ajj> so consensus: take it in, right?
16:39:39 <sa2ajj> s/: /is: /
16:40:03 <bb-trac> [trac] #3588/defect (new) updated by tardyp (2) If I take an existing buildmaster, adjust the configuration so it has a ...) http://trac.buildbot.net/ticket/3588
16:40:13 <djmitche> yep
16:40:20 <djmitche> #info    http://trac.buildbot.net/ticket/3588 - duplicate tags causes DB inconsistency
16:40:32 <djmitche> I think this means tags=['a', 'b', 'a']?
16:40:47 <tardyp> I am not sure but it looks like indeed
16:40:56 <djmitche> o
16:40:56 <djmitche> ok
16:41:00 <djmitche> so pretty easy to prevent from happening :)
16:41:02 <sa2ajj> just dedup?
16:41:03 <tardyp> this guy reported on ml, but then didn't answer
16:41:13 <djmitche> ah, ok
16:41:21 <tardyp> I would rather error early rather than dedupe automaticaly
16:41:23 <djmitche> easy enough to list(set(tags))
16:41:24 <djmitche> right
16:41:27 <djmitche> either one
16:41:50 <djmitche> #info Buildbot PR#2332 Increased size of builder identifier from 20 to 50
16:42:14 <djmitche> I think this is a pretty minor change.  IIRC the reason I chose 20 is that it appears in an AMQP message
16:42:22 <djmitche> sorry, in a routing key
16:42:29 <djmitche> and the limit for a routing key atom is 23 iirc
16:42:31 <tardyp> its not
16:42:34 <tardyp> only the id
16:42:48 <tardyp> but that is good to know
16:43:08 <djmitche> oh, ok, great
16:43:10 * sa2ajj . o O (good opportunity to put a comment next to that constant... :))
16:43:11 <djmitche> carry on then!
16:43:17 <tardyp> maybe that was for the initial buildrequest messaging design
16:43:19 <djmitche> true :D
16:43:21 <djmitche> right
16:43:34 <djmitche> ok then
16:43:39 * djmitche does 0.8.0 first since sa2ajj may have to leave
16:43:47 <djmitche> #topic 0.8.x release
16:44:00 * djmitche cedes the stage to sa2ajj
16:44:24 <sa2ajj> there's not much to say: my lovely vacation started with computer failure
16:44:38 <djmitche> oh no, that doesn't sound very relaxing
16:44:43 <sa2ajj> and then i just did not do *anything* related to programming, even though i managed to recover the computer
16:45:10 <sa2ajj> the fun part was: system and home partitions on an encrypted lvm volume :)
16:45:16 <sa2ajj> anyway
16:45:22 <djmitche> been there, done that
16:45:30 <sa2ajj> in finland we have "coming back from vacation" time
16:45:44 <sa2ajj> so it's not very time demanding (in three weeks it's gonna be different)
16:45:52 <sa2ajj> i checked the PRs
16:46:00 <sa2ajj> there was only one really 0.8.x related
16:46:07 <sa2ajj> so next steps:
16:46:17 <sa2ajj> * finalise the release notes
16:46:31 <sa2ajj> (= check them to see if it's all clear)
16:46:42 <sa2ajj> and then follow cutting-the-release page
16:46:44 <djmitche> this is for 0.8.13 right?
16:46:46 * djmitche taking notes
16:46:48 <sa2ajj> yes
16:46:50 <djmitche> #info next steps for 0.8.13: finalize release notes, cut the release
16:46:59 <djmitche> great, so no further blocking bugs or PRs
16:47:07 <sa2ajj> i do not think so
16:47:12 <djmitche> #info no known bugs or PRs blocking the release
16:47:14 <djmitche> awesome
16:47:20 <sa2ajj> i really want this to be _maintenance_ release
16:47:25 <djmitche> welcome back, too!
16:47:27 <djmitche> yes, indeed
16:47:39 <sa2ajj> so *i* did not do any features backporting (like cmake step)
16:47:46 <djmitche> right
16:47:59 <sa2ajj> i *hope* to have before our next weekly meeting
16:48:00 <djmitche> #info no features backported to 0.8.13 beyond those for which contributors provided an eight patch
16:48:12 <djmitche> #info hope to have release within a week
16:48:15 <djmitche> ok!
16:48:23 <djmitche> tardyp, how about 0.9.0rc1?
16:48:30 <djmitche> #topic 0.9.0rc1 release
16:48:31 <djmitche> or rc2?
16:48:51 <tardyp> so we got a bunch of reports from dave on ml
16:49:00 <tardyp> I wanted to fix those before rc2
16:49:11 <tardyp> but I did not have time to fix them yet
16:49:23 <djmitche> #info got some bug reports on the mailing list for rc1, so rc2 is delayed until those are fixed
16:49:44 <tardyp> the backlog needs some work http://trac.buildbot.net/query?status=accepted&status=assigned&status=new&status=reopened&milestone=0.9.0&group=status&order=priority
16:50:28 <djmitche> looks like mostly new stuff, ok
16:50:47 <djmitche> #info accumulating a backlog of bugs to fix (currently 10)
16:51:51 * djmitche goes on to Hyper.sh
16:51:55 <djmitche> #topic Hyper.sh integration
16:52:08 <djmitche> looks like there is continued close cooperation with the Hyper folks!
16:52:15 <djmitche> any details to add there?
16:52:51 <tardyp> I mostly finished my hyper integration
16:53:13 <tardyp> the current PR are more fixes on the latent stuff that I encountered when testing
16:53:18 <tardyp> but nothing hyper related
16:53:18 <djmitche> great!
16:53:33 <djmitche> #info Hyper.sh integration is mostly complete; remainder are small fixes to latent support
16:53:40 <tardyp> I'd like to get some reviews from @tomprince but he looks like maybe on vacation
16:53:56 <djmitche> he's a good choice, yes
16:53:56 <tardyp> well those are not small fixes actually
16:54:02 <djmitche> #undo
16:54:02 <bb-supy> Removing item from minutes: <ircmeeting.items.Info object at 0x806916f90>
16:54:08 <djmitche> #info Hyper.sh integration is mostly complete; remainder are important fixes to latent support
16:54:33 <tardyp> so we have a meeting on thursday with hyper folks to show the demo
16:54:38 <djmitche> I don't think anything has changed with infrastructure, aside from whacking a few jails into operation
16:54:39 <djmitche> ah, ok
16:54:49 <djmitche> #info Meeting with Hyper people on Thursday to demo the results
16:54:51 <tardyp> they want to prepare a blog post to try buzzing arround the integration
16:55:03 * sa2ajj . o O (that message from bb-supy was useful...)
16:55:26 <djmitche> well, at least it says it's doing something :)
16:55:27 <sa2ajj> wow
16:55:31 <sa2ajj> that'd be great
16:55:33 <djmitche> tardyp: that's awesome
16:55:43 <djmitche> I think we had the opportunity to do something cool like this with Docker
16:55:44 <djmitche> back in the day
16:55:53 <djmitche> and I definitely dropped the ball
16:56:01 <djmitche> so I'm glad you're doing this!
16:56:13 <tardyp> :)
16:56:30 <djmitche> cmouse: any update on openstack v3?
16:56:36 <djmitche> I think we were still waiting for a bug?
16:56:45 <djmitche> #topic MOSS Projects
16:57:32 <djmitche> So Gervase from Mozilla has asked me and Bill (separately) about progress on spending the MOSS funds
16:57:48 <djmitche> we indicated we would spend them within 1 year of their being granted, and we're about 8 months into that year now
16:58:15 <djmitche> rutsky: AIUI your bounty is waiting on you to work out payment with SFC, so I think we're OK there (correct me if I'm wrong)
16:58:30 <djmitche> but we still have no takers on the EC2 bounty
16:58:31 <djmitche> http://trac.buildbot.net/ticket/3392
16:58:39 <djmitche> #info Mozilla is looking for status
16:58:48 <djmitche> #info slave -> worker bounty is on track
16:58:57 <djmitche> #info EC2 support bounty has no takers yet
16:59:22 <djmitche> verm__ was suggesting changing the requirements for the bounty
16:59:30 <rutsky> djmitche: yes, I think almost all work on 2340 is done (at least all significant parts), and I need to initiate processes of enterpreneur registration here in Russia in order to receive bounty
16:59:33 <bdbaddog> rutsky: let me know the status and I'll update MOSS for that bounty.
16:59:34 <djmitche> I'm open to that idea, especially if there's someone who would work on it
16:59:45 <djmitche> bdbaddog: ohai! :D
16:59:58 <bdbaddog> :) back from the ether.
17:00:02 <djmitche> I think gerv's asked you the same questions he asked me so I've sort of lost track of things, but he'll ask again if we've missed something
17:00:24 <bdbaddog> yup. just asked status of bounty 1 and 2.
17:00:27 <djmitche> fwiw I'm happy to call 2340 complete for purposes of payment -- the remainder is miniscule in comparison to what's been done
17:00:39 <bdbaddog> o.k. I'll pass that along to MOSS.
17:00:39 <djmitche> I'd love to have that completed but I trust that rutsky will follow through
17:01:07 <bdbaddog> It's all in 0.9.x release right?
17:01:10 <djmitche> yues
17:01:10 <rutsky> currently I'm quite occupied from my new job and I haven't investigated yet what I need to do to receive bounty here
17:01:10 <bdbaddog> minus the details
17:01:41 <bdbaddog> o.k. so 99% done + figuring out details of payment to "bounty hunter" :0
17:01:42 <bdbaddog> :)
17:01:47 <djmitche> rutsky: I'd think of December as a soft deadline for getting that taken care of -- Mozilla will probably start bugging us to disburse or return the funds at that point
17:01:57 <djmitche> (of course, it took them four months to give us the funds, so we have some leeway)
17:02:02 <djmitche> yep
17:02:13 <rutsky> I hope I'll finish everything 2340-related during August
17:02:13 <djmitche> so I'm open to ideas regarding the EC2 bounty
17:02:26 <djmitche> * change it somewhat, with someone in mind to whom the changes would better appeal
17:02:43 <djmitche> * change it completely to a new and different bounty (again for which we know there is some interest)
17:02:51 <djmitche> * more aggressively market the bounty as it is
17:02:57 <rutsky> at September our gov in Russia returns from vacation and I don't want to stumble with new laws that they will release :)
17:03:10 <djmitche> haha!
17:03:10 <tardyp> ahah
17:03:11 <sa2ajj> side note: hyper.sh buzzing might help ec2 bounty
17:03:20 <djmitche> sa2ajj: good point
17:03:32 <djmitche> tardyp: it wouldn't hurt to bring this up in the conversation with Hyper
17:03:50 <tardyp> well ec2 is their direct competitor
17:04:13 <djmitche> haha, that's true
17:04:27 <djmitche> we could change the bounty to be more generally about latent support, as verm__ suggested
17:04:31 <tardyp> I personally think hyper is much better for the ec2 market ( even for moz)
17:06:10 <djmitche> you and verm__ are saying similar things :)
17:06:29 <tardyp> indeed
17:06:39 <tardyp> ec2 is so 2010 :)
17:06:41 <djmitche> ok, well, of those ideas, I'm not capable of doing the third (marketing)
17:07:38 <djmitche> and I don't have a good sense for any interested parties in either of the first two
17:08:20 <tardyp> my thought is that the bounty too low for the actual ammount of investment needed for a newcomer
17:08:28 <djmitche> for a newcomer, definitely
17:08:42 <djmitche> so, we don't have scope to increase the amount.  Should we scope it down, then?
17:08:43 <sa2ajj> so that goes to info: ec2 is competitor to hyper
17:08:59 <sa2ajj> bounty can be extended to cover _latent_ build workers...
17:09:02 <djmitche> #info ec2 is a competitor to hyper
17:09:23 <cmouse> djmitche: oh right
17:09:26 <djmitche> could we "extend" to cover latent workers while also reducing the difficulty?
17:09:46 <sa2ajj> i think we could offer bigger bounty for _new_ latent build workers
17:09:57 <sa2ajj> and smaller for ironing out details for existing ones.
17:10:13 <sa2ajj> i'd say significantly smaller, but still _a_ bounty
17:10:59 <sa2ajj> and then somewhere in between a bounty for maintaining compatibility among different versions of the same environment
17:11:13 <djmitche> all within the US$5000 we have?
17:11:18 <sa2ajj> oh
17:11:22 <djmitche> and with no single bounty < US$1000, which is a limitation of SFC's
17:11:25 <sa2ajj> looks like i missed a lot :)
17:11:33 <djmitche> I'm not trying to say "no", just sharing the limitations :)
17:11:59 <sa2ajj> what bou
17:12:02 <sa2ajj> oops
17:12:58 <sa2ajj> what about 1k for bringing an existing one to the "latest" level, 3k for improving compatibility b/w versions, 5k for a new one?
17:13:15 <djmitche> 1+3+5 > 5, but we could do 1+3
17:13:18 <djmitche> or even 2+3
17:13:42 <tardyp> so for http://trac.buildbot.net/ticket/3393
17:13:44 <sa2ajj> well... is there a page to read about bounties?
17:13:53 <djmitche> http://trac.buildbot.net/wiki/BountyProgram
17:13:54 <tardyp> I think I resolved it with my masterShutdown PR
17:13:58 <sa2ajj> thx
17:14:10 <tardyp> http://trac.buildbot.net/ticket/2935 is really EC2 specific
17:14:20 <tardyp> I dont think it is very difficult to fix
17:14:50 <tardyp> Providing a failsafe method of managing EC2 instances that will reliably prevent over-provisioning and lost EC2 instances. Proposals should outline one or more specific approaches to this problem.
17:14:59 <tardyp> I dont really understand it
17:15:01 <bb-trac> [trac] #2340/enhancement (assigned) updated by rutsky (empty comment) http://trac.buildbot.net/ticket/2340
17:15:11 <djmitche> tardyp: oh, good point wrt masterShutdown
17:15:23 <sa2ajj> so 10k is for slave->worker transition
17:15:32 <djmitche> the last one is "prevent this from happening again"
17:15:33 <sa2ajj> and 5k is what's left for a specific item
17:15:36 <djmitche> where the first few points are "fix this bug"
17:15:43 <djmitche> sa2ajj: right, that's our allocation
17:16:05 <tardyp> so this is something I did try to handle in hyper PR
17:16:08 <sa2ajj> ok, then what's i wrote above does not really apply
17:16:45 <tardyp> the idea is to make sure the latent workers are connected to application lifecycle stopService, and make sure that they never leak instances open
17:16:58 <djmitche> ok
17:17:01 <djmitche> maybe we should just pay you?
17:17:01 <tardyp> of course, this does not take in account master crash
17:17:06 <djmitche> I'm not really joking
17:17:19 <bb-trac> [trac] #2340/enhancement (assigned) updated by rutsky (empty comment) http://trac.buildbot.net/ticket/2340
17:17:37 <tardyp> well I actually would prefer this bounty bring in new contributors
17:17:55 <sa2ajj> so if the question about how to get people to want that specific bounty
17:18:11 <djmitche> agreed
17:18:51 <sa2ajj> i'd think harder what is involved (there're already 4 points) and assign bounties (>= 1k) for unrelated ones
17:19:10 <tardyp> I think, for the ammount of the bounty, probably points 1 and 3  would be better
17:19:12 <sa2ajj> at least, on the first sight it seems that some of those items are not completely related
17:19:26 <djmitche> ok
17:19:42 <tardyp> which is: documentation and tutorial + fixes for the spot instances
17:19:48 <djmitche> so it sounds like, establish a well-defined and tested way of running Buildbot with ec2
17:19:59 <sa2ajj> unfortunately, i have *no* experience with EC2 (so far i always used bare metal)
17:20:02 <djmitche> and making spot-instance requesting more reliable
17:20:35 <sa2ajj> and if those two items are the ones, 1 < 2
17:20:49 <sa2ajj> so either 1+4 or 2+3 might be the way
17:20:50 <bb-trac> [trac] #3393/enhancement (assigned) updated by dustin (https://github.com/buildbot/buildbot/pull/2329 may also address this.) http://trac.buildbot.net/ticket/3393
17:21:01 <djmitche> ok
17:21:11 <djmitche> I'll make that change, and re-send email to the devel@ and users@ lists
17:21:29 <djmitche> #action djmitche to modify EC2 bounty project as discussed and re-advertise it to mailing lists
17:21:56 * sa2ajj nods
17:22:04 <djmitche> #info suggestion is to limit the project to (1) establish a well-defined and tested way of running Buildbot with EC2 and (2) make spot-instance requests more reliable
17:22:20 <bb-trac> [trac] #3393/enhancement (assigned) updated by tardyp (I *think* this is addressed, however, I did not test ec2 recently) http://trac.buildbot.net/ticket/3393
17:22:22 <djmitche> #info broken up into two separate bounties, $1k < x < $5k
17:22:28 <djmitche> ok, cool
17:22:33 <djmitche> any other business to bring up, or should we wrap?
17:22:43 <sa2ajj> btw, rutsky, you did a great job!  bounty well earned!
17:23:09 * sa2ajj has no AOB
17:23:22 <sa2ajj> s/B$/Bs/
17:23:31 <rutsky> sa2ajj: thanks :)
17:24:14 <djmitche> #endmeeting