16:59:34 <djmitche> #startmeeting weekly
16:59:34 <bb-supy> Meeting started Tue Feb  5 16:59:34 2019 UTC and is due to finish in 60 minutes.  The chair is djmitche. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:59:34 <bb-supy> Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:59:34 <bb-supy> The meeting name has been set to 'weekly'
16:59:50 <djmitche> #topic Introduction
16:59:54 <djmitche> http://bit.ly/2rup31x
17:00:11 <djmitche> This is a 30-minute meeting.. anyone is welcome to speak up on any topic..
17:00:17 <djmitche> #nick tardyp
17:00:44 <djmitche> who else is around?
17:00:49 <bdbaddog> Greetings!
17:01:01 <djmitche> hi!
17:01:11 <djmitche> #topic Week in Review
17:01:17 <djmitche> it's been an "interesting" week..
17:01:29 <bdbaddog> for sure our first CVE?
17:01:37 <tardyp> hey!
17:01:43 <tardyp> not first. my first
17:01:46 <tardyp> third for bb
17:02:26 <tardyp> The week was pretty cool until friday.
17:02:45 <tardyp> we had lots of cleanup patches about py3, and I planned to make the 2.0 release over the week end
17:02:48 <djmitche> is that public? Should we not include it in the notes?
17:03:04 <tardyp> no its public now
17:03:33 <djmitche> #info CRLF injection vulnerability in /auth/logout?redirect= discovered; CVE number pending
17:03:34 <tardyp> We receive an alert on our botherder list
17:03:39 <tardyp> which got in my spam box
17:03:46 <tardyp> and skelly pinged me about it.
17:03:53 <bdbaddog> ditto.. much email from builtbot still hitting my spam box.
17:03:54 <djmitche> ugh
17:04:11 <djmitche> I think it's because our mailserver is old-fashioned
17:04:22 * djmitche adds agenda item
17:04:32 <tardyp> the vulnerability wasn't to complicated. The fix was easy
17:04:33 <bdbaddog> oh weekly summary got through this week.
17:04:49 <tardyp> then some discussion on how to create a CVE
17:04:56 <tardyp> and I made a release with it
17:05:02 <bdbaddog> as I understand it mainly you need a couple DNS records added to vastly imrpove email deliverablilty
17:05:07 <tardyp> both with 1.8.1 and  2.0.0
17:05:17 <rjarry> hi
17:05:20 <tardyp> so 1.8.1 is just t1.8.0 plus the fix, and the new last one to support 2.7
17:05:21 <bdbaddog> So baseline for 2.0.0 is 3.5 or 3.6?
17:05:28 <djmitche> tardyp: there was a suggestion to announce on another list, too? oss-sec or something?
17:05:31 <tardyp> bdbaddog: right
17:05:51 <tardyp> djmitche: indeed. I did not announce it yet
17:06:07 <tardyp> I assume that our users are more reading our lists than oss-sec
17:06:24 <djmitche> yeah
17:06:25 <djmitche> ok
17:06:28 <tardyp> but probably why create a CVE if not publishing it to oss-sec
17:06:33 <djmitche> true :)
17:06:34 <tardyp> I need to do it
17:07:09 <tardyp> we also looked a bit at the event.buildbot.net and saw that we are still 50/50% py2/py3
17:07:21 <tardyp> I'd expect the release of 2.0 will speed up thing
17:07:24 <djmitche> #info Fix is in 1.8.1 and 2.0.0; will announce to oss-sec soon
17:07:37 <djmitche> #info 2.0.0 requires Python 3.6
17:07:38 <tardyp> another important event for 2.0.0 is that we have a big migration script in that release
17:07:49 <rjarry> djmitche: py 3.5 works too
17:07:52 <tardyp> and we got one user who CTRL-C the upgrade-master
17:08:02 <tardyp> which corrupted its db.
17:08:08 <djmitche> #undo
17:08:08 <bb-supy> Removing item from minutes: <ircmeeting.items.Info object at 0x807014bd0>
17:08:10 <djmitche> #info 2.0.0 requires Python 3.5
17:08:27 <rjarry> tardyp: bigger migration script than usual?
17:08:32 <djmitche> corrupted the sqlite itself, or the DB was OK but the data it contained was scrambled?
17:08:50 <tardyp> rjarry: I guess yes, as it is adding a bunch of change in the FK
17:09:01 <rjarry> hmm
17:09:29 <bdbaddog> tardyp.. can we have a copy then upgrade then remove backup in teh migration?
17:09:31 <tardyp> djmitche: well the migrate got interrupted in the middle, and tmp_table was kept which prevented another upgrade-master to start
17:09:50 <tardyp> bdbaddog: I would expect any decent op to do that.
17:09:53 <rjarry> in any case, what p12tic mentioned "catch all signals and ignore them" seems like a sane thing to do
17:10:01 <tardyp> but our users are not always experiemented ops
17:10:07 <bdbaddog> sure.
17:10:14 <tardyp> so we shouldn't require that from them
17:10:27 <djmitche> backups of sqlite are possible (but potentially large) but of mysql/postgres are not something we could do automatically
17:10:30 <rjarry> but making a copy of the sqlite file can take a lot of space
17:10:31 <tardyp> at least warn that upgrade-master should not be interrupted is mho important
17:11:05 <djmitche> #info a user interrupted a long-running migration task and corrupted data (left temp table around); considering ways to mitigate that risk
17:11:07 <tardyp> and maybe as suggested a catch of KeyboardInterrrupt
17:11:09 <rjarry> that's especially true for sqlite and mysql, pg will recover without problems
17:11:18 <tardyp> I am not sure how to do that reliably
17:11:28 <rjarry> tardyp: trap all signals
17:11:42 <rjarry> not only KeyboardInterrupt
17:11:49 <djmitche> that's not practical
17:11:53 <djmitche> some are not trappable
17:11:56 <tardyp> yes
17:11:56 <djmitche> some are important
17:12:06 <rjarry> sure, but we can at least trap
17:12:11 <djmitche> but trapping sigint is practical (I think trapping the signal, vs. catching KeyboardInterrupt)
17:12:19 <rjarry> SIGTERM, SIGINT, SIGUSR1 and SIGUSR2
17:12:39 <rjarry> djmitche: if you install a sig handler for SIGINT, KeyboardInterrupt is not risen anymore
17:12:54 <djmitche> right, that seems a good solution -- by the time the exception is caught it's too late
17:12:56 <djmitche> anyway
17:13:04 <djmitche> is there an issue on file for this?/
17:13:10 <rjarry> not yet
17:13:20 <rjarry> I'll create it
17:13:21 <djmitche> #action rjarry to file issue about interrupting migrations
17:13:21 <djmitche> ;)
17:13:27 <djmitche> #topic Python 2.7 deprecation
17:13:36 <rjarry> and I'll fix it while on the way :)
17:13:40 <djmitche> #info still about 50/50% python2/python3 now, based on telemetry
17:13:54 <djmitche> #info hopefully the Buildbot 2.0.0 release will help move that toward python3
17:14:04 <djmitche> any other things to discuss on this topic?
17:14:17 <rjarry> about that 50/50
17:14:29 <rjarry> I think we should give a few weeks for users to update
17:14:41 <bdbaddog> Have you seen this site: https://pypistats.org/packages/buildbot. they're rolling up the pypi bigtable stuff.
17:14:48 <rjarry> 2.0.0 was only released 3 days ago
17:16:06 <tardyp> cool pypistats
17:16:18 <djmitche> rjarry: yes, not suggesting we do anything now
17:16:19 <tardyp> I was looking for such a service for long
17:16:49 <rjarry> btw, debian already has 2.0.0 integrated in unstable
17:17:02 <rjarry> it should migrate to testing before the end of the week
17:17:05 <djmitche> well, three days, that's time enough! ;)
17:17:18 <djmitche> #topic Mailing list posts flagged as spam
17:17:24 <tardyp> and maybe they first try with py2 and then try with py3
17:17:48 <rjarry> default python version in the linux distros may be the reason why :s
17:18:06 <rjarry> I can't speak for windows, my mother does not let me
17:18:10 <djmitche> yeah, it seems reasonable
17:18:10 <djmitche> haha
17:18:34 <djmitche> so, mailing list posts -- I can get a bug flied in bb-infra, and bring it up on the sysadmins list
17:18:51 <djmitche> we might be able to fix locally, or consider hosted alternatives (please not google groups)
17:18:57 <djmitche> other input?
17:18:59 <bdbaddog> BTW. PY 3.8a1 has dropped so might be worth someone giving it a try. For SCons it broke much of our regression suite.
17:19:05 <tardyp> I did try to train gmail recognise buildbot.net as a non spammer
17:19:18 <tardyp> but there is not anymore a "not spam" button.
17:19:32 <tardyp> i just copy it back to the buildbot folder and hope this will train the IA
17:19:39 <bdbaddog> I see such.. not spam or just clear the spam tag.
17:19:44 <tardyp> gmail is so confident they removed the not spam button.
17:19:56 <tardyp> or maybe its because I still use Inbox
17:20:00 <bdbaddog> hmm. I still see items with a button click her if not spam.
17:20:04 <djmitche> haha, one or the other
17:20:11 <bdbaddog> could be.. Inbox never took for me.
17:20:20 <djmitche> I added filters which help, but there's still a big "this is spam but your filter said not to throw it out"
17:20:23 <djmitche> https://screenshots.firefox.com/72Kffet8p5ZA2ikX/mail.google.com
17:20:35 <djmitche> https://github.com/buildbot/buildbot-infra/issues/220
17:20:48 <tardyp> djmitche: do you learn dutch?
17:21:32 <djmitche> ja :)
17:21:42 <djmitche> haha, I forgot about that
17:22:09 <tardyp> it also apear in your email replies. funny..
17:22:20 <tardyp> Pierre Tardy shrift
17:22:27 <djmitche> yeah, sorry
17:22:37 <djmitche> hm, I have a "Geen Spam" button..
17:22:49 <tardyp> anyways..
17:23:24 <djmitche> #topic Kubernetes Workers
17:23:27 <djmitche> anything new?
17:24:19 <tardyp> I did change the account to standard paid account
17:24:27 <tardyp> from trial account
17:24:38 <tardyp> and got my quota raise to 16VMs (I think)
17:24:58 <djmitche> "Votre facture mensuelle Google Cloud Platform & APIs est disponible. " lol
17:25:02 <tardyp> so we now have a bit more CPU, and can at least run one pipeline fully in parallel
17:25:02 <djmitche> (also in spam)
17:25:03 <tardyp> hehe
17:25:13 <djmitche> excellent
17:25:21 <tardyp> the build now runs in 16min
17:25:23 <djmitche> #info switched to a paid GCP account with a higher quota
17:25:27 <djmitche> is that faster?
17:25:33 <tardyp> which is about 2 minutes more than what we had in hyper
17:25:48 <tardyp> the builds themselfs are a bit faster
17:25:59 <tardyp> but we have 2-3 minutes slow start
17:26:00 <bdbaddog> Dumb question for the crowd.. How much would a threadripper 1 U cost to throw in the datacenter to handle this load? would that be better than going GCP?
17:26:05 <djmitche> #info builds are a little faster, but startup is a little slower
17:26:26 <tardyp> the billing for January was $20
17:26:50 <tardyp> about $13 for the base infra
17:26:53 <djmitche> bdbaddog: my tendency is to get away from owning hardware -- it's been a lot of work to maintain it
17:27:00 <tardyp> and the rest for the on demand VMs for builds
17:27:31 <bdbaddog> indeed.. HW can be a pain.
17:27:55 <djmitche> that's about what I'm paying for freecinc.com
17:28:32 <djmitche> any other news there?
17:28:46 <djmitche> ..and, any updates on GSoC?
17:29:26 <tardyp> do you have 32 CPU on demand with that price?
17:29:56 <tardyp> also, we are still waiting for Bret to reply so that I can remove my credit card from that account.
17:30:02 <tardyp> I need to go back to him
17:30:10 <tardyp> about gsoc, yesterday was the deadline.
17:30:24 <tardyp> I think we have a quite good idea page
17:30:37 <tardyp> https://github.com/buildbot/buildbot/wiki/Buildbot-GSoC-Projects-2019
17:30:50 <tardyp> djmitche: if you have some idea to enhance it from your experience.
17:31:03 <djmitche> #topic GSoC
17:31:16 <tardyp> I hope we will be selected, and that being a sub-org of python will get us more chance as a small org
17:31:17 <djmitche> #info ideas page established; still looking for mentors
17:31:35 <djmitche> that's a great page - nice work!
17:32:18 <djmitche> ok, we're over time so I'll wrap up
17:32:20 <djmitche> #endmeeting