16:33:55 <djmitche> #startmeeting weekly
16:33:55 <bb-supy> Meeting started Tue May  3 16:33:55 2016 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:33:55 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:33:55 <bb-supy> The meeting name has been set to 'weekly'
16:33:55 <bb-supy`> Meeting started Tue May  3 16:33:55 2016 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:33:55 <bb-supy`> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:33:55 <bb-supy`> The meeting name has been set to 'weekly'
16:34:01 <djmitche> uh, whut
16:34:24 <rutsky> more, more meetings! :)
16:34:25 <djmitche> ok
16:34:33 <djmitche> #topic Introduction
16:34:41 <djmitche> Meeting Agenda: https://titanpad.com/buildbot-agenda
16:35:00 <djmitche> #topic Development week-in-review
16:35:31 <djmitche> Looks like a good week for PRs, but less so for bugs
16:35:55 <djmitche> lots of "fix" in the pull requests
16:36:00 <djmitche> Are we mostly in bug-fix mode?
16:36:15 <tardyp> we are not
16:36:41 <tardyp> I actually am mostly the one who is doing bug fix, but I got busy on other matters :-/
16:36:49 <tardyp> and rutsky finish his tasks
16:37:03 <tardyp> but the others PRs are more for contributions from other people
16:37:07 <djmitche> ok
16:37:09 <djmitche> that's great too
16:37:20 <tardyp> there are though a bunch of bug fixes without trac
16:37:34 <djmitche> #info We have a healthy number of PRs, both bug fixes and contributed enhancements
16:37:38 <tardyp> pple are starting to embrace nine, and find bugs, and find missing features for them
16:37:54 <djmitche> excellent
16:37:56 <tardyp> so yes, it is overall very good
16:38:14 <djmitche> #info lots of improvements are to nine features, so it is gaining momentum
16:38:27 <djmitche> thoughts on another beta? rc?
16:39:45 <tardyp> I wanted to do it this weekend, but could not attend the meeting.
16:39:52 <tardyp> so next week end could be good.
16:40:03 <djmitche> ok -- an RC, if I remember correctly?
16:40:08 <tardyp> rutsky: will you have a working buildbot-worker package?
16:40:26 <tardyp> the rc is if we are done on the bug
16:40:31 <rutsky> it's working, but I haven't done master changes that I want to do
16:40:39 <djmitche> ok
16:40:39 <tardyp> but we have imho enough changes to deserve another beta
16:40:44 <rutsky> better not to release worker until I finish it
16:40:54 <djmitche> so probably best to plan for a beta this weekend
16:40:54 <rutsky> since I need to put incompatible changes there
16:40:55 <tardyp> rutsky: ok any ETA?
16:41:07 <rutsky> I hope to finish during this week
16:41:24 <tardyp> ok. so the plan is beta9 this week end, for a release on monday
16:41:41 <tardyp> and then I'll ask you if I package buildbot-worker
16:41:51 <tardyp> or if we go workerless like we did for beta8
16:42:04 <djmitche> #agreed will release beta9 on Monday, potentially without buildbot-worker (like beta8)
16:42:10 <rutsky> tardyp: ok
16:42:10 <djmitche> cool!
16:42:29 <djmitche> it will be really nice to get that shipped -- I expect we'll see a BIG uptick in this sort of contribution
16:42:36 <djmitche> if the early-adopters are this helpful :)
16:42:42 <tardyp> yes
16:42:53 <djmitche> #topic service2.buildbot.net downtime
16:43:07 <djmitche> #info One of our three servers failed on Friday, and Amar brought it back up on Monday
16:43:32 <djmitche> #info Some work nearby in OSUOSL dislodged its power cable, and the monitoring didn't alert Amar
16:43:33 <rutsky> who should be pinged in such cases?
16:43:39 <djmitche> ideally sysadmins@
16:43:48 <djmitche> although as I found out later, that list is hosted on service2
16:43:54 <djmitche> so my messages didn't go anywhere!
16:43:54 <rutsky> hehe)
16:43:58 <tardyp> could we setup an  external monitoring service?
16:43:58 <sa2ajj> ironic
16:44:07 <tardyp> ideally outside of the infra
16:44:17 <djmitche> yeah, I don't know of any that are free
16:45:10 <gracinet> what kind of monitoring do you have in mind ? a simple ping ?
16:45:14 <sa2ajj> https://www.statuspage.io/pricing ?
16:45:16 <djmitche> that would be enough, yeah
16:45:43 <tardyp> I found that https://uptimerobot.com/
16:45:55 <gracinet> but that doesn't solve the email problem
16:45:56 <djmitche> sa2ajj: $29/mo is a lot for an org with no real income :)
16:46:01 <sa2ajj> nah, i thought i'd show something else :(
16:46:08 <sa2ajj> sorry
16:46:09 <djmitche> ah, I like uptimerobot
16:46:16 <djmitche> insofar as it's free :)
16:46:30 <djmitche> ok, I'll file a bug to set that up
16:47:19 <sa2ajj> uptimerobot might be the tool for now...
16:47:49 <djmitche> at least then people can check status.bb.n
16:47:49 <djmitche> or whatever we choose to use
16:48:04 <sa2ajj> status.bb.n sounds like a good place :)
16:48:16 <djmitche> hm, that's not that helpful actually -- we got pings here that alerted us to the issue in a fairly timely fashion
16:48:19 <tardyp> yes, but it can't be hosted on our infra
16:48:27 <djmitche> we really should have just called amar sooner
16:48:35 <djmitche> or, it turns out, filed an OSUOSL ticket
16:48:49 <djmitche> oh, uptimerobot does do alerting
16:48:54 <djmitche> sorry, thinking of another service
16:48:57 <djmitche> I'll set that up
16:49:09 * sa2ajj . o O (action...)
16:49:12 <bb-trac> [trac] #3545/enhancement (v:0.8.12) created by dustin (set up uptimerobot.com) http://trac.buildbot.net/ticket/3545
16:49:15 <gracinet> I see SMS in the list :-)
16:49:34 <djmitche> #agreed will set up uptimerobot.com to alert us of failures like this more quickly in the future
16:49:36 <tardyp> SMS is not free, but I would expect email is
16:50:43 <djmitche> OK, cool
16:51:00 <djmitche> I'm honestly not too upset -- I think this is about the expected uptime given our investment level :)
16:51:18 <djmitche> but it could have been less with better human communication, and that was on me :)
16:51:26 <djmitche> #topic Bug 2340 Update
16:51:30 <djmitche> rutsky: ^^ go! :)
16:51:36 <tardyp> agreed on the not too upset
16:51:48 <rutsky> it's still WIP :)
16:51:50 <tardyp> we are a volunteer team
16:51:53 <rutsky> I'd like to discuss https://github.com/buildbot/buildbot/pull/2184
16:52:03 <rutsky> I want to remove usePTY from worker
16:52:13 <rutsky> according to documentation it's deprecated
16:52:43 <rutsky> and it interfere with slave/worker transition
16:53:08 <rutsky> any objections for removing usePTY?
16:53:19 <sa2ajj> djmitche: btw, do you have twitter account? (sorry the rest)
16:53:27 <djmitche> sa2ajj: I don't, no
16:53:32 <sa2ajj> ok
16:53:39 <bb-github> [13buildbot] 15tardyp commented on issue #2138: its not 4496 its http://trac.buildbot.net/ticket/2673... 02https://git.io/vwHK4
16:53:40 <djmitche> rutsky: it's been deprecated for a long time, so I think that's OK
16:53:53 <tardyp> no objection for killing default usePTY
16:54:01 <tardyp> as it is best configured in the builder
16:54:12 <gracinet> this has nothing to do with the PTY options in ShellCommand, right ?
16:54:30 <rutsky> gracinet: it's related
16:54:44 <djmitche> gracinet: but that option will remain
16:54:48 <rutsky> previsouly you could specify usePTY in worker configuration and in ShellCommand
16:54:57 <rutsky> now it will be only in ShellCOmmand
16:55:13 <gracinet> suits me, no pbm (did not even notice the worker config option)
16:55:22 <rutsky> ok, then I merging deletion of usePTY in worker
16:55:40 <djmitche> yay!
16:55:50 <bb-github> [13buildbot] 15rutsky pushed 4 new commits to 06master: 02https://git.io/vwHKi
16:55:50 <bb-github> 13buildbot/06master 14ed677db 15Vladimir Rutsky: remove worker-side usePTY support...
16:55:50 <bb-github> 13buildbot/06master 1419f3ddd 15Vladimir Rutsky: fix flake8
16:55:50 <bb-github> 13buildbot/06master 1409e9aff 15Vladimir Rutsky: add workaround for using buildbot-worker with current master
16:55:59 <rutsky> any objections for merging https://github.com/buildbot/buildbot/pull/2182 ?
16:56:02 <djmitche> so still hoping for a release-able buildbot-worker this week?
16:56:03 <rutsky> tomprince: you here?
16:56:13 <djmitche> he wasn't going to be abel to be here, no
16:56:16 <rutsky> djmitche: I think yes
16:56:27 <rutsky> oh...
16:56:56 <rutsky> I would like to merge https://github.com/buildbot/buildbot/pull/2182 since I think tomprice mistaken in his comment
16:57:15 <rutsky> and that PR already has merge-me label :)
16:58:04 <bb-github> [13buildbot] 15tardyp closed pull request #2182: Use buildbot-worker for local worker (06master...06use-buildbot-worker-for-local-worker) 02https://git.io/vw1mU
16:58:05 <djmitche> haha, done :)
16:58:19 <tardyp> I think tomprince says that he is including some module in the slave namespace
16:58:28 <djmitche> #info removed usePTY worker-side configuration option (PR 2184)
16:58:31 <bb-github> [13buildbot] 15rutsky commented on issue #2138: @tardyp ok, thanks, I'll fix this. 02https://git.io/vwH6I
16:58:57 <djmitche> #info planning to have a release-able buildbot-worker this week, after which we can make a release candidate
16:59:15 <tardyp> in integration.py, he adds:
16:59:18 <tardyp> try:
16:59:18 <tardyp> from buildslave.bot import BuildSlave
16:59:18 <tardyp> except ImportError:
16:59:18 <tardyp> BuildSlave = None
16:59:25 <rutsky> tardyp: I haven't noticed anything slave-related in https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/integration/test_latent.py
16:59:33 <tardyp> I will fix it in my next integration PR
17:00:02 <rutsky> oh, that file! He haven't noted that file
17:00:16 <rutsky> tardyp: ok, thanks
17:00:28 <rutsky> any questions regarding bug2340?
17:01:06 <tardyp> not much. hopefuly we can close that for eow
17:01:23 <rutsky> tardyp: which intergration.py do you mean?
17:01:37 <rutsky> tardyp: I don't see slave in https://github.com/buildbot/buildbot/blob/master/master/buildbot/test/util/integration.py either
17:01:56 <rutsky> tardyp: or is `from buildslave.bot import BuildSlave` is in PR?
17:03:07 <tardyp> indeed, you are right
17:03:11 <tardyp> I ndeed to rebase
17:03:32 <tardyp> you fixed it in your PR I just merged. Sorry for confusion
17:03:44 <rutsky> ah, ok
17:03:51 <rutsky> next topic?
17:04:31 <djmitche> #topic Dropping Python 2.6
17:04:45 <rutsky> yay!
17:04:48 <djmitche> #info Proposal is to drop master-side support for Python-2.6
17:05:21 <rutsky> agenda has some for and against dropping 2.6: https://titanpad.com/buildbot-agenda
17:05:53 <rutsky> is anyone present here who interested in Python 2.6 support for buildbot master?
17:05:57 <rutsky> anish?
17:05:58 <djmitche> #info pros/cons assembled before the meeting - https://pastebin.mozilla.org/8869523
17:06:18 <djmitche> sa2ajj: I think you referred to this as a dealbreaker?
17:07:52 <djmitche> typically we've dropped support for older things when we've been forced by some strong motivation
17:08:00 <rutsky> maybe nine introduces addition deps that will already not available on old platform, like RHEL5, so there is already no way to run BB master on them?
17:08:23 <tardyp> is RHEL5 still supported? I see that it hasn't got an update in 2015
17:08:36 <rutsky> we have guanlecoja that requires pretty new nodejs/etc, but BB master can be installed with prebuilt web UI
17:08:37 <tardyp> https://fr.wikipedia.org/wiki/Red_Hat_Enterprise_Linux#RHEL_5
17:08:48 <tardyp> oups sorry for fr.wikipedia
17:09:10 <skelly> still in "production 3 phase"
17:09:23 <rutsky> tardyp: that link gives me `Exception encountered, of type "BadMethodCallException"`, btw :)
17:10:06 <djmitche> From previous experience with this, I think we can assume there are users for whom this will be a hardship
17:10:08 <skelly> dependencies have dropped 2.6 so nine has capped upper versions of e.g. twisted
17:10:10 <rutsky> according to Wiki RHEL5 has "End of Extended Lifecycle Support" up to 30 November 2020
17:10:28 <gracinet> 13 years !
17:10:31 <skelly> only paying customers get those last three years
17:10:34 <djmitche> arguments that it's not updated, etc. are good, but don't help those folks update their infrastructure
17:11:16 <djmitche> so I think the way to look at this is, what do you say to the user who's stuck on RHEL5 and not able to upgrade
17:11:31 <tardyp> buildbot eight
17:11:37 <djmitche> ok
17:11:50 <gracinet> we're speaking strictly of the master, right ?
17:11:55 <rutsky> gracinet: yes
17:11:56 <djmitche> yes
17:12:16 <gracinet> I asked the same question last time, just wanted to make it very explicit :-)
17:12:25 <skelly> if eight has a security issue, would a fix be made?
17:12:26 <djmitche> yeah
17:12:36 <djmitche> skelly: yes, assuming it was practical
17:12:44 <tardyp> yes
17:13:05 <djmitche> https://github.com/buildbot/botherders/blob/master/policies/support.md
17:13:35 <djmitche> I suspect we'll see a big chunk of users still on 0.8.x for a long while anyway
17:13:42 <tardyp> agreed
17:13:49 <rutsky> does anyone know how even Python 2.6 is being installed on RHEL5?
17:13:59 <rutsky> I think RHEL5 out of the box has Python 2.4
17:14:11 <rutsky> and users already do some additional steps to install 2.6
17:14:14 <djmitche> 2.5, but agreed
17:14:15 <djmitche> right
17:14:33 <djmitche> I think the point is, they are probably already facing a huge mountain of work to upgrade to nine
17:14:44 <djmitche> and this is likely to tip the balance into "nope, not going to upgrade"
17:15:53 <djmitche> That this has now come up at least twice suggests that there is a strong, if diffuse desire to do this among active BB hackers
17:16:28 <djmitche> so I think if we're comfortable with potentially leaving a few more users behind, then we should drop 2.6 compat
17:16:42 <gracinet> if py2.6 gets supported in bb 9.0, how long would that extend ?
17:16:50 <djmitche> not very long
17:17:07 <djmitche> I'd be much more positive about dropping support for 2.6 if it wasn't at exactly 0.9.0
17:17:14 <djmitche> since so much else is changing in that version
17:17:30 <gracinet> I have no personal interest in 2.6, but find it a bit sad to carry i t so close to the 0.9 release and droping it at the last moment
17:17:33 <tardyp> also, there is this: http://blog.technotesdesk.com/how-to-install-python-2-7-on-rhel-5/
17:17:37 <rutsky> djmitche: do you prefer to drop 2.6 in 0.9.1?
17:17:50 <tardyp> which is not distro specific
17:17:59 <rutsky> I think we can extend 2.6 support up to 0.9.1
17:18:13 <rutsky> and see then how much new users will still have 2.6
17:18:37 <tardyp> and also the fact that beyond python we might have userspace library dependencies that we are not aware of on such old distros
17:19:36 <rutsky> is there rpm package with Builbot for RHEL?
17:20:18 <djmitche> rutsky: I'd be more comfortable with that -- we could even put in a big warning in the release notes
17:20:38 <djmitche> but at least then users would have a stepping-stone to upgrade to 0.9.0 first, then to 2.7, then to 0.9.1
17:20:53 <tardyp> it does not make sense to me
17:21:26 <tardyp> 2.7 is a very easy upgrade
17:21:37 <djmitche> ok
17:21:39 <skelly> plenty of things got pushed from 0.9.0 that there would be a large impetus to either wait for a .1 or similar or be stuck with 0.9.0 for a while
17:21:52 <djmitche> true
17:21:53 <djmitche> yeah
17:21:59 <djmitche> I'm getting increasingly convinced :)
17:22:09 <djmitche> and I don't really have a right to a strong opinion here anyway
17:22:14 <djmitche> being neither a user nor an active hacker
17:22:37 <djmitche> it seems that the majority opinion is we should drop 2.6 support
17:23:24 <rutsky> I think most of developers would like to use newer and shinier versions of libraries/tools :)
17:24:10 <tardyp> I think we are one of the last python project to drop py26
17:24:23 <djmitche> haha, ok
17:24:39 <tardyp> especially with that big codebase
17:24:44 <djmitche> #agreed Buildbot will drop Python-2.6 support with the 0.9.0 release
17:24:45 <rutsky> there is ansible
17:24:51 <djmitche> #info (see chat log for details of the decision)
17:24:59 <djmitche> even ansible's requirements are increasing rapidly
17:25:08 <rutsky> which is developed by RH guys and I have feeling that they support even Py 2.4 :)
17:25:48 <rutsky> so, what is the plan of dropping Py 2.6?
17:26:00 <djmitche> tomprince proposed it, so let's let him do the PR :)
17:26:06 <tardyp> we just remove py2.6 from travis and metabb
17:26:13 <djmitche> that just means adding relnotes, adjusting the docs, and adjusting the various testing systems
17:26:18 <djmitche> and appveyor?
17:26:20 <rutsky> + update docs
17:26:36 <rutsky> appveyor runs only on 2.7 if I remember correctly
17:26:37 <tardyp> appveyor/circle ci do not have test matrix
17:27:37 <rutsky> and we immediately start wiping 2.6 support from the source code?
17:27:56 <rutsky> ah, regarding travis
17:28:04 <rutsky> we would like to run worker tests on 2.6
17:28:14 <rutsky> so it's not just removing configurations
17:29:06 <rutsky> also it would be nice to test integration between 2.7 master and 2.6 worker
17:29:07 <tardyp> rutsky: indeed
17:29:07 <djmitche> rutsky: I don't think we need to be aggressive about it
17:29:17 <djmitche> this is not a good time to be introducing bugs and test failures etc
17:29:44 <tardyp> I will be, with my next integration tests
17:29:45 <djmitche> I can't think of any issues we've had regarding python versions between master and worker
17:29:53 <djmitche> so I'm not sure how much value such an integration test would have
17:29:58 <djmitche> but definitely testing worker on 2.6 is important!
17:30:21 <djmitche> and if that's easy with the integration testing framework, then all the better :)
17:30:22 <tardyp> the integration tests have been flaky, yes, but all the flakyness were actual real bug
17:30:29 <djmitche> cool
17:30:33 <rutsky> tardyp: can you describe your plans regarding integration tests?
17:30:38 <tardyp> which help me to be confident on the robustness of the code
17:31:00 <tardyp> To run the integration tests with taking in account BUILDBOT_DB_URL_FOR_TEST
17:31:07 <tardyp> for now, they only run with sqlite on disk
17:31:44 <rutsky> I've sketched e2e tests that just creates master, worker, start both, stop both, and I'm thinking about direction to extend this (see https://github.com/buildbot/buildbot/pull/2136).
17:32:34 <rutsky> if we finished py2.6 dropping topic, lets discuss something else
17:32:36 <tardyp> this will allows for very fast integration tests for dev (by using in memory sqlite)
17:32:37 <tardyp> and also test e2e with all real db supported
17:32:37 <tardyp> yes, my e2e are not really e2e in the perfect sense
17:32:43 <rutsky> for example ideas regarding e2e
17:32:56 <tardyp> but I think this is the best compromise to  test real life, and still being easy to write
17:33:08 <rutsky> tardyp: agree
17:33:34 <djmitche> #topic E2E / Integration Testing
17:33:39 <rutsky> I need real e2e tests to have at least smoke testing for new worker
17:34:21 <rutsky> I already discovered that it is critical to name `buildbot-worker.bat` instead of `buildbot_worker.bat`, thanks to this e2e and appveyor
17:34:38 <rutsky> I would like to discuss ideas regarding real e2e testing
17:34:56 <tardyp> I am not so found of spawning masters and slaves in new processes
17:35:12 <rutsky> we can run `buildbot create-master ...`/`buildbot-worker create ...` with custom configurations to test something
17:35:35 <rutsky> tardyp: it's e2e, I think this is how it is supposed to be
17:36:18 <rutsky> I understand that it's really hard to write good tests in that way that covers a lot of code
17:36:41 <rutsky> but I would like to have at least some smoke testing, that whole system is actually working
17:36:48 <tardyp> rutsky: agreed, this is real e2e
17:37:17 <rutsky> we could put tutorial steps into e2e to check that documentaation is still valid
17:37:18 <tardyp> but I think there are not so much usecase you can test with this that you can't with python level integration tests
17:37:36 <rutsky> tardyp: I agree
17:37:57 <tardyp> this reminds me of jupyter
17:38:11 <tardyp> they have some feature to verify tutorials automatically
17:38:12 <djmitche> there are always tradeoffs -- integration and E2E tests are harder to keep up to date
17:38:16 <djmitche> and it's harder to define "success"
17:38:25 <djmitche> do they watch the logfiles for certain messages?
17:38:36 <djmitche> they also tend to fail by hanging
17:38:39 <rutsky> djmitche: this is only thing they do atm :)
17:38:50 <djmitche> yeah
17:38:57 <tardyp> if we do tutorial validation, for me that would be by parsing the rst
17:39:12 <djmitche> at the risk of sounding like a broken record
17:39:29 <djmitche> we should probably get 0.9.0 out the door and work on its known weaknesses :)
17:39:49 <djmitche> E2E could probably help us avoid embarassing things like buildbot_worker.bat
17:39:53 <rutsky> djmitche: I agree 0.9.0 is higher priority than e2e
17:40:00 <djmitche> ok
17:40:15 <rutsky> but I honestly can't test my changes on Windows except on AppVeyor
17:40:17 <djmitche> then after 0.9.0 this seems like a great idea :)
17:40:19 <rutsky> at least legally :)
17:40:19 <djmitche> true
17:41:01 <rutsky> ok, then I'll do what I'll do in that PR and lets see if this will be useful
17:41:36 <rutsky> currently it's just create master + create worker + start both + stop both + check exit codes/output message as I described earlies
17:41:40 <rutsky> earlier
17:42:23 <rutsky> tardyp: while checking examples from docs is super feature IMO, it's usually quite hard
17:42:48 <rutsky> tardyp: best thing that we might do, is to include in RST parts of actual tests, as does Qt for example
17:43:05 <rutsky> *parts of tests master.cfg
17:43:44 <djmitche> seems sensible
17:43:55 * djmitche makes wrapping up motions
17:44:33 <djmitche> not to dissuade further discussion of course!
17:44:37 <djmitche> #endmeeting