16:55:10 <rutsky_> #startmeeting weekly
16:55:10 <bb-supy> Meeting started Tue Aug 23 16:55:10 2016 UTC and is due to finish in 60 minutes.  The chair is rutsky_. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:55:10 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:55:10 <bb-supy> The meeting name has been set to 'weekly'
16:55:28 <rutsky_> #chair tardyp #chair tardyp
16:55:28 <bb-supy> Warning: Nick not in channel: #chair
16:55:28 <bb-supy> Current chairs: #chair rutsky_ tardyp
16:55:36 <rutsky_> #chair rutsky
16:55:36 <bb-supy> Current chairs: #chair rutsky rutsky_ tardyp
16:56:11 <rutsky_> #info agenda https://titanpad.com/ep/pad/view/buildbot-agenda/EyMlQpy7gu
16:56:23 <rutsky_> #topic Introduction
16:56:33 <rutsky_> Hi everyone!
16:56:41 <rutsky_> lets do roll call, who is around?
16:56:48 <tardyp> hu
16:56:49 <bdbaddog> bdbaddog here.
16:57:21 <rutsky_> #topic Week in Review
16:58:22 <tardyp> the week was quite quiet
16:58:32 <tardyp> I am preparing rc2
16:59:14 <rutsky_> #info Pierre backported recent fixes to 0.9.0 branch and preparing rc2 release
16:59:49 <rutsky_> #info now we have nice 256-colors support in job terminal output
17:00:29 <rutsky_> ok, lets move on
17:00:49 <rutsky_> #topic 0.9.0rc1 update
17:01:11 <tardyp> there is a PR with rc2 release note
17:01:12 <rutsky_> tardyp: Is this topic about rc1 update or maybe rc2 release?
17:01:24 <tardyp> rc1 already out
17:01:26 <gracinet> hi guys
17:01:30 <tardyp> hi gracinet
17:01:32 <rutsky_> #undo
17:01:32 <bb-supy> Removing item from minutes: <ircmeeting.items.Topic object at 0x806a4ba90>
17:01:38 <rutsky_> #topic 0.9.0rc2 update
17:02:09 <rutsky_> #info rc2 release PR is awaiting review: https://github.com/buildbot/buildbot/pull/2348
17:02:35 <rutsky_> sa2ajj: are you here to talk about 0.8.x release?
17:02:47 <tardyp> I have verified that autobahn now works with tw16.3
17:03:04 <tardyp> so I will release the constraint on setup.py
17:03:14 <rutsky_> tardyp: in rc2?
17:03:21 <tardyp> yes
17:03:40 <tardyp> and I will also put some very simple protractor tests
17:03:50 <tardyp> got tired of testing it manually
17:03:53 <rutsky_> protractor tests?
17:04:28 <rutsky_> oh, I see its e2e for Angular
17:04:29 <tardyp> e2e with selenium
17:04:50 <rutsky_> tardyp: sounds great!
17:05:47 <rutsky_> #info latest Twisted release (16.3) works with Autobahn, so rc2 will not set upper limit on Twisted version
17:06:06 <rutsky_> tardyp: do you plan to release rc2 today?
17:06:15 <tardyp> yes
17:06:21 <tardyp> I am nearly done with the testing
17:06:39 <rutsky_> #info Buildbot 0.9.0rc2 will be released today
17:06:52 <rutsky_> anything else about rc2?
17:07:37 <tardyp> nope
17:07:47 <rutsky_> anyone want's to talk about 0.8 release? Or this topic is from previous meetings?
17:08:47 <rutsky_> lets skip 0.8 release topic
17:09:15 <rutsky_> #topic Run Metabuildbot test suite before merging pull requests
17:09:24 <tardyp> sa2ajj: there?
17:09:29 <rutsky_> Typical aproach I've seen
17:09:31 <rutsky_> one of github org members reviews PR and marks is as "safe to test"
17:09:33 <rutsky_> bot sees such comments in PR and starts testing
17:09:34 <rutsky_> bot updates PR status using regular GitHub Status API
17:10:06 <tardyp> this is something I want to do with hyper
17:10:18 <rutsky_> I saw that users do this pipeline using their own bots and using Jenkins
17:10:31 <tardyp> hyper let you start fresh container for everu build
17:10:51 <tardyp> this allows to do tests without fearing about compromising security
17:10:58 <rutsky_> tardyp: do you plan to test PRs unconditionally?
17:11:03 <tardyp> yes
17:11:54 <rutsky_> this is a good approach, but *in addition* to it it would be nice to have ability to run tests for reviewed PRs or from users in white list
17:12:42 <tardyp> why?
17:13:00 <tardyp> if we have safe unconditional testing
17:13:09 <rutsky_> because containers is not always an option (e.g. for windows users)
17:13:30 <bdbaddog> so if I push a pull request which runs CC for a spambot army it will get run unconditially.
17:13:30 <rutsky_> for Metabuildbot it will be sufficient, yes
17:13:53 <rutsky_> I'm thinking about adding general functionality to Buildbot
17:13:56 <tardyp> ah. indeed. this is a flow that can be implemented indeed
17:14:10 <tardyp> probably with a change_hook and a change filter
17:14:21 <tardyp> instead of answering to PR event, answer to PR comment event
17:14:40 <rutsky_> tardyp: is it possible to hook PR comment event right now?
17:14:41 <bdbaddog> does/would the container block outbound connections? or have whitelisted outbound connections?
17:14:46 <tardyp> there might be everything there already actually
17:15:06 <tardyp> have to dig into github hook doc
17:15:18 <tardyp> but if jenkins can do it, this should be possible
17:15:30 <tardyp> I hope they dont poll....
17:16:03 <rutsky_> IMO this feature may significantly improve BB position as CI runner for modern project
17:16:05 <tardyp> bdbaddog: the container dont block outbound connections, but is killed after a while
17:16:23 <tardyp> so it can be taken in control but not for so much time
17:16:31 <bdbaddog> still..
17:16:39 <bdbaddog> or send sapm.
17:16:40 <bdbaddog> spam.
17:16:42 <tardyp> like for travis
17:16:57 <rutsky_> I'll open ticket for implementing such pipeline in Buildbot, is nobody against it
17:17:03 <bdbaddog> I agree for as a general feature for BB it would be great.  But for metabuildbot we should be careful
17:17:14 <tardyp> every feature is great!
17:18:12 <gracinet> about listening for comment events, indeed it seems not to be far «  If you'd like to handle other events (see Event Types & Payloads for more information), you'd need to subclass GitHubEventHandler and add handler methods for the corresponding events. For example…  » http://docs.buildbot.net/latest/manual/cfg-wwwhooks.html
17:18:29 <tardyp> for me this is much better to have the contributor not wait for a maintainer before getting CI feedback
17:19:04 <gracinet> https://developer.github.com/v3/activity/events/types/#pullrequestreviewcommentevent
17:21:08 <bb-trac> [trac] #3598/enhancement (v:master) created by rutsky (Implement pipeline for integration with GitHub (or other provider) that will allow to safely run BB tests for PR) http://trac.buildbot.net/ticket/3598
17:21:16 <rutsky_> lets move on, and discuss details of this feature later, maybe in the ticket
17:21:30 <rutsky_> #topic nine.buildbot.net is down
17:21:53 <tardyp> I need to put it back online
17:22:01 <rutsky_> who can take a look at nine.bb.net logs?
17:22:12 <tardyp> it is not very useful right now so I did not bother yet
17:22:20 <rutsky_> ah, ok
17:22:28 <tardyp> I want to use it eventually to run the PR CI
17:22:43 <tardyp> but its not ready yet
17:22:51 <rutsky_> #action tardyp bring up nine.buildbot.net
17:22:56 <tardyp> +1
17:23:06 <rutsky_> do we plan to replace buildbot.buildbot.net with nine version?
17:23:16 <rutsky_> maybe after 0.9.0 release?
17:23:57 <tardyp> once we have a working nine.buildbot.net yes
17:24:02 <tardyp> that is my plan
17:24:09 <bdbaddog> maybe buildbot = nine, and then eight.buildbot.net ?
17:24:17 <bdbaddog> for eight?
17:24:35 <rutsky_> bdbaddog: +1
17:24:48 <rutsky_> lets decide this later, when nine will be ready
17:24:53 <bdbaddog> sure.
17:25:01 <rutsky_> #topic Failing tests on metabuildbot
17:25:20 <bb-trac> [trac] CuttingReleases edited by tardyp (empty comment) http://trac.buildbot.net/wiki/CuttingReleases
17:25:25 <rutsky_> there are 4 type of errors right now
17:25:42 <rutsky_> error: Could not read 03e9fac82cd879e4a63820de0b3b7c1bcafc0911
17:25:44 <rutsky_> error: Could not read dec8425746491f56757a2ded7793778b904da8c2
17:25:45 <rutsky_> error: Could not read 03e9fac82cd879e4a63820de0b3b7c1bcafc0911
17:25:47 <rutsky_> error: Could not read dec8425746491f56757a2ded7793778b904da8c2
17:25:49 <rutsky_> fatal: pack has 53 unresolved deltas
17:25:50 <rutsky_> fatal: index-pack failed
17:25:51 <rutsky_> program finished with exit code 128
17:25:53 <rutsky_> http://buildbot.buildbot.net/builders/sqla%3D0.8.7-sqlam%3D0.9.8/builds/222/steps/git/logs/stdio
17:25:54 <tardyp> t
17:26:01 <tardyp> this looks like borken workdir
17:26:08 <rutsky_> I believe something wrong with .git or free space
17:26:24 <rutsky_> who can take a look at this?
17:26:57 <djmitche> ack, I'm sorry guys -- too many things going on today :(
17:26:57 <tardyp> koobs: can you look at this?
17:27:00 <tardyp> this is on koobs-freebsd10
17:27:10 <rutsky_> hi, Dustin!
17:27:18 <tardyp> only him has access to that machine
17:27:27 <rutsky_> tardyp: koobs, probably this is original error in the log: stat: No such file or directory: /usr/home/buildbot/buildbot/sqla_0_8_7-sqlam_0_9_8/build/.buildbot-patched
17:27:41 <tardyp> no
17:27:46 <koobs> hmm.
17:28:05 <tardyp> this log is on all the builds
17:28:20 <rutsky_> tardyp: ah, ok
17:30:01 <rutsky_> #action koobs please take a look at git failures at koobs-freebsd10 worker
17:30:33 <koobs> lol. :)
17:30:37 <koobs> am looking now
17:30:52 <rutsky_> virtualenv setup fails due to a missing enchant library: http://buildbot.buildbot.net/builders/docs/builds/546/steps/virtualenv%20setup/logs/stdio
17:31:25 <rutsky_> how hard is to get enchant C library on workers?
17:31:36 <tardyp> this is on buildbot.net worker
17:31:46 <rutsky_> yes
17:31:53 <tardyp> so this is about adding one line in a ansible file
17:31:54 <rutsky_> we need it only on worker that builds docs
17:32:08 <bdbaddog> is there a wheel?
17:32:48 <rutsky_> bdbaddog: wheel with linux binary?
17:33:20 <tardyp> no its a native package that is needed
17:33:31 <tardyp> and buildbot.net is a freebsd worker
17:33:59 <bdbaddog> only a wheel for win32 and mac with the binary it looks like.
17:35:20 <koobs> tell me if you need to port/package the enchant lib/things
17:35:30 <koobs> s/the/any
17:35:58 <rutsky_> tardyp: can you point me to a live of ansible where worker environment is configured?
17:36:26 <tardyp> what do you mean?
17:36:44 <rutsky_> where should enchant library be installed?
17:37:08 <rutsky_> I think this is configuration of buildbot.net worker: https://github.com/buildbot/buildbot-infra/blob/master/jail-bslave1.yml
17:37:08 <tardyp> on the worker?
17:37:59 <tardyp> probably need to add some stuff in venv_package
17:38:03 <tardyp> but I am not sure
17:38:07 <rutsky_> tardyp: I'm trying to figure out which line should be added in which file to get this fixed :)
17:38:34 <rutsky_> who is familiar with BB ansible configuration and can help with this issue?
17:38:52 <koobs> [root@10-STABLE-amd64:/usr/home/buildbot/buildbot/sqla_0_8_7-sqlam_0_9_8/build] git fetch -t git://github.com/buildbot/buildbot.git master
17:38:55 <koobs> testing.
17:39:13 <koobs> fatal: pack has 53 unresolved deltas
17:39:16 <koobs> fatal: index-pack failed
17:39:24 <koobs> how to fix
17:39:49 <rutsky_> koobs: IMO easiest way is to reclone buildbot.git
17:39:51 <tardyp> koobs: rm -rf .git
17:40:11 <koobs> Script/step should check for this status and do it #action
17:40:15 <koobs> Will fix locallty
17:40:46 <rutsky_> koobs: this is unusual behaviour of git, probably some major malfunction of something
17:40:54 <koobs> possibly,
17:40:57 <djmitche> rutsky_: I can help with ansible
17:41:05 <koobs> [root@10-STABLE-amd64:/usr/home/buildbot/buildbot/sqla_0_8_7-sqlam_0_9_8/build] git fsck
17:41:05 <rutsky_> koobs: I had such errors with diying hard drive
17:41:08 <koobs> Checking object directories: 100% (256/256), done.
17:41:10 <koobs> Checking objects:   1% (1466/103673)
17:41:31 <tardyp> sudo pkg install textproc/enchant
17:41:32 <tardyp> looks like installing enchant
17:41:36 <koobs> https://dpaste.de/9xde/raw
17:41:39 <koobs> git fsck output
17:41:50 <koobs> running fetch again
17:41:51 <rutsky_> djmitche: we are trying to fix this error by installing enchant on docs worker: http://buildbot.buildbot.net/builders/docs/builds/546/steps/virtualenv%20setup/logs/stdio
17:42:04 <koobs> nope, still fails.
17:43:11 <tardyp> rm -rf build
17:43:17 <tardyp> buildbot will reconstruct it
17:43:18 <djmitche> ok
17:43:29 <koobs> tardyp: that for me ?
17:43:37 <tardyp> koobs: yes
17:43:43 <koobs> donne
17:43:46 <koobs> done*
17:43:54 <koobs> was it only freebsd10 that failed?
17:43:58 <koobs> i couldnt see any other builder fails
17:44:15 <koobs> rutsky_: btw, workers are Vm guests (as per host info)
17:44:37 <koobs> tardyp: force rebuild if you could
17:44:40 <koobs> while im awake
17:44:43 <koobs> its 3:44am :)
17:44:46 <tardyp> koobs: I did
17:44:49 <koobs> gracias
17:45:27 <rutsky_> it works
17:45:34 <koobs> link to build?
17:45:34 <rutsky_> koobs: thanks!
17:45:42 <rutsky_> http://buildbot.buildbot.net/builders/sqla%3D0.8.7-sqlam%3D0.9.8/builds/224
17:45:53 <koobs> danke
17:45:58 <rutsky_> git update succeeded
17:46:11 <koobs> that looks like bb-linux3?
17:46:14 <koobs> linux4
17:46:30 <koobs> http://buildbot.buildbot.net/buildslaves/koobs-freebsd10
17:46:36 <koobs> http://buildbot.buildbot.net/builders/sqla%3D0.8.7-sqlam%3D0.9.8
17:46:38 <rutsky_> koobs: oh, you right
17:46:50 <koobs> git update suceeded
17:46:53 <koobs> lovely
17:47:02 <djmitche> do you need this too:
17:47:03 <djmitche> py27-enchant-1.6.5_6           Spellchecking library for Python based on the Enchant library
17:47:11 <koobs> #action Make git update step check for success and do Something TM if it fails
17:47:18 <tardyp> djmitche: this is done in virtualenv
17:47:22 <koobs> djmitche: you can pip that, it should compile
17:47:27 <koobs> if thats how you like to do things
17:47:28 <tardyp> so shouldn't be needed
17:47:31 <djmitche> ok
17:47:35 <koobs> just need the clib normally
17:47:46 <koobs> ASSUMING upstream author has their cextension bits done well :)
17:47:47 <tardyp> and maybe the -dev
17:47:50 <koobs> Otherwise, welcome to porting
17:47:58 <koobs> Our clibs come with libs and headers
17:48:03 <djmitche> https://github.com/buildbot/buildbot-infra/pull/135
17:48:05 <koobs> no -dev package separation
17:48:09 <tardyp> not sure if -dev is a think in freebsd
17:48:11 <bb-github> [13buildbot-infra] 15djmitche opened pull request #135: add enchant (06master...06enchanting) 02https://git.io/v6HN0
17:48:13 <tardyp> ok
17:48:14 <koobs> it isnt
17:48:38 <koobs> nor is -dbg but we may separate debug symbols out soon
17:48:43 <koobs> among other cool things
17:48:44 <rutsky_> djmitche: thanks!
17:48:50 <koobs> ok sleep time
17:48:52 <koobs> night all
17:49:06 <tardyp> koobs: night
17:49:07 <tardyp> https://pypi.python.org/pypi/buildbot/0.9.0rc2
17:49:36 <rutsky_> koobs: thanks for the help!
17:49:36 <tardyp> signed and smoke tested!
17:49:53 <rutsky_> tardyp: great, thanks!
17:50:12 <rutsky_> on buildbot-linux4 tests fails due to some issue with Python: http://buildbot.buildbot.net/builders/db-postgres/builds/609/steps/test%20master/logs/stdio
17:50:37 <bb-github> [13buildbot-infra] 15djmitche closed pull request #135: add enchant (06master...06enchanting) 02https://git.io/v6HN0
17:51:16 <rutsky_> what is buildbot-linux4 and why it has Python 2.6 on it?
17:51:29 <bb-meta> build #223 of sqla=0.8.7-sqlam=0.9.8 is complete: Success [3build successful]  Build details are at http://buildbot.buildbot.net/builders/sqla%3D0.8.7-sqlam%3D0.9.8/builds/223
17:51:41 <djmitche> rutsky_: it's a really old VM at mozilla
17:51:45 <djmitche> CentOS 5 I think
17:53:22 <rutsky_> djmitche: and it has Python 2.6 *and* 2.7 on it?
17:53:50 <djmitche> yep
17:54:08 <rutsky_> ok, then we need to fix db-postgres build environment to use 'python2.7' instead of 'python'
17:54:24 <rutsky_> djmitche: do you know where this can be fixed?
17:54:27 <djmitche> ok, enchant is installed
17:54:49 <djmitche> hm, I wonder how it broke..
17:54:54 <djmitche> I think that's in the metabuildbot config
17:55:52 <tardyp> enchant is a new requirement
17:57:39 <djmitche> sorry, I wonder how db-postgres on buildbot-linux4, which has been running fine for a long time, suddenly started using python2.6
17:57:51 <tardyp> I got to go. I'll send the annoucement tomorrow when the rc2 PR will be merged and forward merged to master (in order to have published RN)
17:58:39 <rutsky_> tardyp: ok, thanks for your help!
17:59:02 <rutsky_> tardyp: will you take a look at buildbot-www build failures later?
17:59:55 <tardyp> Ok
18:00:32 <rutsky_> #info lots of work done on fixing metabuildbot build failures, thanks to koobs, tardyp and djmitche!
18:00:52 <djmitche> hm, I still see no enchant :( http://buildbot.buildbot.net/builders/docs/builds/547/steps/virtualenv%20setup/logs/stdio
18:00:57 <rutsky_> #action tardyp take a look at buildbot-www build failures on metabuildbot
18:02:56 <djmitche> whoops, installed it in the wrong jail
18:03:05 <bb-github> [13buildbot-infra] 15djmitche pushed 1 new commit to 06master: 02https://git.io/v6HhJ
18:03:05 <bb-github> 13buildbot-infra/06master 14e2f7cb9 15Dustin J. Mitchell: install enchant in the right place
18:04:52 <rutsky_> djmitche: regarding db-postgres build on buildbot-linux4: should we explicitly set python to Python 2.7 or we need to investigate why default python changed (if it actually changed)?
18:05:08 <djmitche> I very much doubt anything changed on that host
18:05:13 <djmitche> I'm pretty much the only one who would login to it, and I haven't
18:05:30 <djmitche> so I think that we should fix the config to specify python2.7
18:06:24 <djmitche> I think docs just went green
18:06:47 <bb-meta> build #548 of docs is complete: Success [3build successful]  Build details are at http://buildbot.buildbot.net/builders/docs/builds/548
18:07:21 <rutsky_> great! thanks Dustin!
18:09:08 <rutsky_> djmitche: am I correct that this line should be changed to add Python 2.7 specification to db-postgres? https://github.com/buildbot/metabbotcfg/blob/master/builders.py#L385
18:11:17 <rutsky_> djmitche: please review https://github.com/buildbot/metabbotcfg/pull/60
18:11:57 <rutsky_> I think this is the last error (except buildbot-www at which Pierre will take a look)
18:12:07 <rutsky_> any other topics to discuss?
18:12:09 <gracinet> congrats
18:12:59 <rutsky_> #endmeeting