16:30:15 <djmitche> #startmeeting weekly
16:30:15 <bb-supy> Meeting started Tue Jan 12 16:30:15 2016 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:30:15 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:30:15 <bb-supy> The meeting name has been set to 'weekly'
16:30:22 <djmitche> #topic Introduction
16:30:28 <djmitche> #nick verm__
16:32:19 <djmitche> https://public.etherpad-mozilla.org/p/taskcluster-weekly-meeting
16:32:32 <djmitche> whoops, wrong meeting
16:32:37 <djmitche> https://titanpad.com/buildbot-agenda
16:32:37 <infobob> http://paste.pound-python.org/show/rcWtUzBxooeYuqMbtcTh/ (repasted for djmitche)
16:32:45 <djmitche> lol, <-- dummy
16:33:01 <rutsky> hi, all!
16:33:05 * djmitche waves
16:33:28 <djmitche> #topic Bug 2340 - renaming
16:33:33 <rutsky> let me check bold on titanpad...
16:34:07 <rutsky> > officially, public interfaces are those that are documented in the .rst files, and no more
16:34:11 <rutsky> this sound great
16:34:35 <rutsky> but in practice I think others still using private methods (at least non underscore named)
16:34:55 <djmitche> we've typically tried to be "nice" to people who might have read the source and used a function, since that's how things used to be
16:35:04 <djmitche> but I don't think any of the identifiers you've mentioned fall into that category
16:35:08 <djmitche> yes
16:35:29 <rutsky> ok, so public only that is documented
16:35:50 <djmitche> yes, at this point I think we've documented most of the things we believe people are using
16:36:14 <djmitche> tardyp: you around?
16:36:36 <rutsky> > Maybe provide flag for create-master to force treating deprecated-name-warnings as errors
16:36:55 <rutsky> so I do no changes in new default buildbot.tac?
16:37:17 <rutsky> currently I added commented code how to force warnings or ignore warnings
16:37:22 <djmitche> no options anyway.  I think a new buildbot.tac should default to the error level
16:37:34 <djmitche> that's good, yes
16:37:55 <rutsky> ok, so default for new configs = error?
16:38:26 <djmitche> I think so anyway :)
16:38:26 <rutsky> if I explicitly add setting warnings to errors, that such configs will stop working when we will remove all deprecated fallback stuff
16:39:13 <rutsky> We *can* make deprectation fallback machinery deprecated at some point...
16:39:15 <djmitche> ideally new installs (so, people running create-master) shouldn't have any old terminology
16:39:24 <rutsky> djmitche: agree
16:39:41 <djmitche> yes, I think we'd consider removing the old name support in 1.0
16:40:21 <rutsky> so at the point of pre 1.0 version import of buildbot.worker_transition from user code will be deprecated and raise warnings?
16:40:22 <djmitche> the case I can see where a new user uses old terminology is copy-pasting from something they see online
16:40:30 <djmitche> in which case errors with a nice message will help
16:40:47 <tardyp> hi all
16:40:47 <djmitche> oh, why would user code be importing that?/
16:40:53 <rutsky> hi tardyp!
16:40:55 <djmitche> hi pierre :)
16:41:18 <rutsky> user code may import buildbot.worker_transition to explicitly set warnings to errors
16:41:31 <djmitche> oh I see
16:41:54 <djmitche> well, they'd get an error when they tried to make that import, which seems reasonable :)
16:42:02 <djmitche> but we can solve that when we do 1.0 -- let's not get ahead of ourselves!
16:42:39 <rutsky> So there are two transitions: from 0.8 to 0.9, when "slave" staff may be used but raise warnings, and from 0.9 to 1.0 when whole fallback machinery should be removed
16:43:09 <rutsky> It's nice to think early about such things, they make future changes much more easy
16:43:36 <djmitche> true
16:44:17 <rutsky> I suggest to not change new buildbot.tac (don't set explicit warnings or errors), but document how warnings can be treated as errors or how to ignore them (this is actuall done)
16:44:47 <djmitche> so by default someone running create-master, then writing c['slaves'] = .. would get a warning?
16:44:56 <rutsky> So if new user will play nicely and use only new API he will be not required to change anything related to slave stuff on 0.9-> 1.0 transition
16:45:02 <rutsky> djmitche: yes
16:45:25 <rutsky> and by default in 0.9 allow all "slave" stuff, but with warnings
16:46:29 <rutsky> any objections?
16:46:41 <djmitche> ok
16:46:43 <djmitche> sounds great
16:46:43 <tardyp> i am ok
16:46:57 <rutsky> ok, so let's move forward...
16:47:18 <rutsky> I will add BUILDBOT_DEPRECATED_WORKER to the TODO list
16:47:25 <djmitche> tardyp: what do you think of SlaveBuilder -> WorkerBuilder
16:47:31 <djmitche> as a name
16:48:04 <tardyp> well I was not found of SlaveBuilder, I am not of WorkerBuilder
16:48:17 <tardyp> but I dont have smarter solution :-}
16:48:20 <djmitche> me neither, but I didn't have a better idea
16:48:21 <djmitche> heh
16:48:24 <djmitche> WorkerForBuilder?
16:48:50 <tardyp> slightly better indeed
16:48:59 <djmitche> ok
16:49:03 <djmitche> let's go with that, then
16:49:13 <djmitche> in the absence of an even-slightly-better  option!
16:49:19 <rutsky> ok, so WorkerForBuilder
16:49:30 <djmitche> yes
16:49:45 * djmitche summarizes for meeting notes
16:50:18 <djmitche> #info rutsky is doing lots of work on this project - WIP at https://github.com/buildbot/buildbot/pull/1943
16:50:33 <djmitche> #info currently working on renaming stuff in master-part
16:50:43 <djmitche> #info roadmap at https://public.etherpad-mozilla.org/p/bounty-app-2340
16:51:02 <rutsky> afk for a few minutes, sorry...
16:51:16 <djmitche> #agreed new users of 0.9.x and higher will get warnings when using old names
16:51:28 <djmitche> #agreed plan to remove support for old names in 1.0 (or at least revisit the idea)
16:51:44 <rutsky> I'm here
16:51:46 <djmitche> #agreed confusingly-named SlaveBuilder will be renamed to WorkerForBuilder
16:52:34 <djmitche> #agreed only "public" names will have compatibility added.  For the most part, that means documented names (in the .rst files), but can also include names users may have found in the source code
16:52:41 <djmitche> rutsky: anything else to cover?
16:52:52 <rutsky> BUILDBOT_DEPRECATED_WORKER please
16:53:36 <djmitche> ok -- the idea is that this would override the in-code setting of the warning level?
16:53:48 <rutsky> yes
16:54:02 <djmitche> do we need to have both options, then?
16:54:14 <rutsky> I think yes
16:54:29 <rutsky> one thing is that we use Python warnings machinery
16:54:29 <djmitche> why? (not arguing, just asking)
16:55:19 <rutsky> actually not sure
16:55:29 <djmitche> heh :)
16:55:34 <rutsky> I'm sure that I want to use Python warnings stuff
16:55:42 <rutsky> they are convenient and pretty powerfull
16:55:53 <djmitche> it seems like *one* way to specify this would be best -- if that's an environment variable, I think that's fnie (as it can also be specified in buildbot.tac)
16:55:55 <tardyp> +1
16:56:01 <rutsky> env var may be convenient to "smoke-test" user setup without changing code
16:56:06 <djmitche> yeah
16:56:19 <djmitche> BUILDBOT_DEPRECATED_WORKER=1 buildbot checkconfig .
16:56:43 <tardyp> ah. because checkconfig does not load the .tac
16:56:53 <tardyp> I would prefer an option to checkconfig
16:56:53 <rutsky> python warnings provides filters and catching functionality that I heavily use in tests
16:56:54 <djmitche> oh, good point
16:56:59 <djmitche> ok
16:57:06 <rutsky> option for checkconfig is +1
16:57:38 <tardyp> generally env variable seems like a good idea, and then you regret it
16:57:44 <tardyp> thas my experience
16:57:46 <djmitche> haha, true
16:57:52 <rutsky> yep)
16:57:53 <djmitche> especially since it's the only one for buildbot
16:58:10 <djmitche> so should we agree to not use an env var, barring some great reason to add one?
16:58:17 <djmitche> preferring buildbot.tac options and checkconfig options?
16:58:23 <rutsky> ok, let's postpone it until real need for it
16:58:27 <djmitche> ok
16:58:31 <rutsky> djmitche: yes
16:58:33 <tardyp> yes : buildbot.tac options and checkconfig options?
16:58:53 <djmitche> #agreed warning level will be controlled with buildbot.tac, and with an option to checkconfig (allowing users to "smoke-test" their config for use of old names)
16:58:58 <djmitche> ok :)
16:59:05 <djmitche> should we talk about docker now?
16:59:16 <rutsky> not yet
16:59:24 <rutsky> let's dicuss "How to not get my changes rotten?"
16:59:43 <rutsky> tardyp: do you plan to do significant changes in BB soon?
16:59:48 <djmitche> oh, sorry, yes
17:00:17 <tardyp> rutsky: I'd like to release a new beta with the new data module
17:00:33 <tardyp> I also would like to better document the REST api
17:00:45 <rutsky> I think I need about week to complete "slave"->"worker" transition and document it
17:00:54 <tardyp> I am looking a lot at http://raml.org/
17:01:22 <tardyp> and would like to use this as the source spec for all the rest api derive stuff we have
17:01:49 <tardyp> so that may eventually include some refactoring, indeed
17:01:55 <tardyp> but the refactoring can wait
17:02:10 <tardyp> the doc generation is for me the more important
17:02:12 <rutsky> sorry, afk
17:02:34 <djmitche> tardyp: doyou think you could ship the new data API in the next week or so, then we can have a change-freeze for rutsky to land his changes, then work on the refactoring?
17:02:35 <tardyp> I think we need the data api doc as we have now, but also a REST api doc, which is more explicit
17:02:41 <rutsky> I'm here
17:02:43 <djmitche> yes
17:02:58 <tardyp> djmitche: of course
17:03:11 <djmitche> rutsky: does that work for you?
17:03:43 <djmitche> I think we'll implement the "freeze" by just checking with you before merging any PR's, and holding off on PRs that wouuld conflict
17:03:43 <rutsky> ok, so I will wait for Pierre's changes in Data API? It's ok for me
17:04:00 <djmitche> if those are still pending in a week, we can consider changing the order
17:04:01 <djmitche> cool
17:04:30 <rutsky> ok, I will not touch Data API stuff until Pierre's confirmation
17:04:46 <djmitche> #agreed pierre will ship a beta with the newly-landed data module next week, after which we will have a "change freeze" for rutsky to land his work, followed by some Data API work
17:04:58 <djmitche> rutsky: I think the thing to avoid is the data module? tardyp?
17:05:03 <rutsky> so plan is: finish most of master part; Pierre's release of Data API; merge of master + docs
17:05:12 <djmitche> as in, the angular code that is a client to the data API
17:05:22 <djmitche> yes, sounds good
17:05:56 <rutsky> ok
17:06:14 <rutsky> Is Bill around?
17:06:28 <djmitche> I don't think so :(
17:06:54 <rutsky> ah, ok. Anyway I don't think "master" part splitting is a good idea
17:06:59 <djmitche> I think you should largely proceed as if this is your project at this point -- it's substantially complete
17:07:02 <djmitche> yes
17:07:19 <djmitche> ok, docker?
17:07:34 <rutsky> docker! :)
17:07:41 <djmitche> #topic best practices for running worker and master in docker containers
17:08:13 <djmitche> #info there are Dockerfiles in master/contrib, but no good description of how to use them
17:08:17 <djmitche> https://github.com/rutsky/buildbot/blob/worker_API_rename_POC2/master/contrib/docker/worker/README.rst
17:08:51 <rutsky> djmitche: this is README that I've done mostly for myself, so it's no generic (yet)
17:09:06 <djmitche> #info we are using these images to run the new linux worker that is replacing knuth.r.igoro.us
17:09:24 <djmitche> I think the question then is, are these good guidelines, and what else should we suggest?
17:10:20 <djmitche> #info looking for feedback on simple recommendations for using docker to run buildbot
17:10:44 <rutsky> worker is packaged is pretty good IMO, at least if we are use temp-like workers that data is not important
17:10:49 <djmitche> myself, I found that docker image very easy to use and I followed the steps in yoru readme
17:10:52 <djmitche> right
17:10:58 <rutsky> and current worker Dockerfile can be reused pretty well
17:11:32 <djmitche> good
17:11:39 <rutsky> for master part... it's not so simple because of master.cfg --- where to place it? how it should be updated?
17:11:46 <djmitche> we should avoid making it too complicated -- if users want to do fancy things, that's up to them :)
17:11:50 <djmitche> right
17:12:12 <rutsky> sorry, afk
17:12:22 <djmitche> from the other docker images I've looked at, many recommend specializing by e.g., FROM buildbot/master \n ADD master.cfg /master.cfg
17:13:56 <bb-trac> [trac] #3407/defect (v:) created by dustin (weekly summary has encoding issues) http://trac.buildbot.net/ticket/3407
17:15:06 <rutsky> I'm here
17:15:48 <rutsky> oh, I don't like adding master.cfg in image...
17:16:01 <rutsky> at least while master.cfg completely defines testing process
17:16:19 <rutsky> e.g. if I need to add new branch/repo/project, I need to change master.cfg
17:16:34 <rutsky> and it's kind of painful to recreate image etc...
17:16:54 <djmitche> true - I guess another option is to mount it (-v) or provide a URL to it (-e)
17:17:24 <rutsky> as I mentioned before, I store master.cfg (with python deps) in git, clone in on the host, and mount using -v
17:17:55 <djmitche> yeah - that seems a little complicated to suggest as the base, though
17:18:18 <djmitche> but maybe suggesting using -v /path/to/your/config:/master
17:18:20 <rutsky> djmitche: btw do you considered how versioning of images of BB will be done in the Docker Hub?
17:18:32 <djmitche> I was versioning by bb version
17:18:50 <rutsky> hm...
17:19:35 <djmitche> https://hub.docker.com/r/buildbot/buildbot-worker/tags/
17:19:47 <rutsky> that's ok, I think. But in general I would expect to see something like sem.versioning for Dockerfile
17:20:10 <djmitche> it seems to match what mysql and postgres do
17:20:18 <djmitche> e.g., mysql/mysql-server:5.6 installs the 5.6 server
17:20:44 <rutsky> yeah, and I'm not sure that this is right way to do it :)
17:20:56 <rutsky> but let's stay with your approach
17:21:18 <djmitche> "like everyone else does it" is often a good approach for something that's not a core competency
17:21:22 <djmitche> ok :)
17:21:37 <djmitche> #info discussion of how to specify configuration to the master container
17:21:44 <djmitche> #info discussion of how to tag master and worker containers
17:21:53 <rutsky> I think master Docker configuration requires further investigation and testing --- I can't suggest anything better then use "-v" for config, and I don't have time to investigate this issue
17:22:04 <djmitche> #agreed to version similarly to how mysql or postgres do -- container tag matches software version
17:22:14 <djmitche> yeah, that should be fine
17:22:34 <rutsky> djmitche: also I like how you named "buildbot/buildbot-worker"
17:22:48 <djmitche> ok, cool
17:23:29 <rutsky> it's better than "buildbot/worker" or simply "worker", because when others will fork config they will have properly named unambiguous image name, like "rutsky/buildbot-worker"
17:23:34 <djmitche> #topic replacing pep8 + pyflakes with flake8
17:23:39 <djmitche> yes, agreed :)
17:24:00 <tardyp> ok. I already use flake 8 in my editor
17:24:04 <djmitche> ok
17:24:11 <djmitche> so this is really just a change in validate.sh?
17:24:21 <rutsky> I opened ticket for this (http://trac.buildbot.net/ticket/3403), and made some POC commits in my branch
17:24:26 <djmitche> it wouldn't involve a lot of soruce-code changse (which might conflict with rutsky's work)
17:24:30 <tardyp> there are also a bunch of makefile targets
17:24:37 <djmitche> right
17:24:38 <tardyp> but validate is enough
17:24:40 <rutsky> tardyp: yes
17:24:57 <rutsky> and source code can be simplified with flake8
17:25:05 <rutsky> e.g. remove most of _hush_pyflakes
17:25:26 <rutsky> and yes, djmitche, it will conflict with my work
17:25:42 <rutsky> I suggest to do it after we will merge my PR
17:25:50 <djmitche> yes, that makes sense
17:25:57 <djmitche> and maybe disabling pyflakes entirely for the moment?
17:26:18 <rutsky> also Travis should be changed (I made it in my branch, but in a dirty way)
17:26:38 <rutsky> flake8 is most probably will be required for my PR
17:26:55 <rutsky> I think I will be required to use from ... import *
17:27:07 <rutsky> and some magic with renaming
17:27:21 <rutsky> maybe I will include basic swithching to flake8 in my PR?
17:27:37 <rutsky> without refactoring code to replace hush_pyflakes with "# noqa"
17:28:03 <rutsky> and after merging renaming I can do hush_pyflakes -> "# noqa"
17:28:46 <djmitche> yes, that makes a lot of sense
17:29:19 <rutsky> > Is "do only pep8 check" step is required? It's definitely useful, but can we live with only "do pep8+pyflakes" and "ignore" pyflakes output?
17:29:33 <djmitche> I didn't undersatnd that sentence..
17:29:38 <rutsky> currently there is "make pep8" and "make pyflakes" target
17:29:45 <djmitche> ah
17:29:53 <rutsky> flake8 may replace it with "make flake8"
17:29:53 <djmitche> I don't think they need to be separate, no
17:30:01 <djmitche> that should be fine
17:30:09 <rutsky> djmitche: ok, that I was tried to ask :)
17:30:18 <djmitche> :)
17:30:28 <djmitche> ok, I need to get running.. anything else to address?
17:30:54 <rutsky> I think no, thanks!
17:30:58 <djmitche> #agreed we want to transition to just running flake8, which is a combination of pep8 and pyflakes
17:31:10 <bb-trac> [trac] #3403/undecided (assigned) updated by rutsky (empty comment) http://trac.buildbot.net/ticket/3403
17:31:18 <djmitche> #agreed for the renaming work, rutsky will make minimal changes to just switch to flake8 but not make source-code changes
17:31:22 <rutsky> I assigned flake8 to myself
17:31:24 <djmitche> ok, great!
17:31:28 <djmitche> #endmeeting