Process-one
ejabberd
ejabberd@conference.process-one.net
Tuesday, July 3, 2018< ^ >
zinid has set the subject to: ejabberd discussions: https://docs.ejabberd.im
Room Configuration
Room Occupants

GMT+0
[00:02:31] lorddavidiii leaves the room: Connection failed: connection closed
[00:04:13] deavmi leaves the room: Disconnected: Replaced by new connection
[00:08:16] ThUnD3r|Gr33n® leaves the room
[00:10:37] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[00:11:50] ludo leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[00:11:51] tester leaves the room
[00:18:04] lorddavidiii leaves the room: Connection failed: connection timed out
[00:30:41] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[00:35:14] prezident leaves the room: Connection failed: connection timed out
[00:47:11] alexis leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[00:52:08] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[01:17:07] jere leaves the room: Disconnected: Replaced by new connection
[01:21:45] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[01:24:32] mrDoctorWho leaves the room
[01:34:25] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[01:38:10] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[01:53:11] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[02:08:08] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[02:22:52] ThUnD3r|Gr33n® leaves the room
[02:25:23] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[02:40:07] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[02:46:42] newbie leaves the room: Stream closed by us: Replaced by new connection (conflict)
[03:22:52] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[03:25:05] pztrn leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[03:33:30] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[03:45:37] Licaon_Kter leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[03:58:46] tester leaves the room
[04:03:46] spicewiesel leaves the room: Stream closed by us: Replaced by new connection (conflict)
[04:16:19] mimi89999 leaves the room
[04:16:34] tester leaves the room
[04:19:19] Marzanna leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[04:46:15] prezident leaves the room: I'm happy Miranda IM user. Get it at http://miranda-im.org/.
[04:47:27] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[04:49:02] robert_mobil leaves the room: Stream closed by us: Replaced by new connection (conflict)
[04:52:30] vogt leaves the room: Stream closed by us: Replaced by new connection (conflict)
[04:53:42] jere leaves the room
[04:55:43] Alacer leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:05:03] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[05:09:51] xinit leaves the room
[05:14:01] deavmi leaves the room
[05:20:53] rom1dep leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:21:03] Alacer leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:22:37] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[05:31:13] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[05:35:06] spicewiesel leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:39:13] spicewiesel leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:44:06] lorddavidiii leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[05:46:02] bammes leaves the room: Stream closed by us: Replaced by new connection (conflict)
[05:48:48] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[05:53:42] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[06:08:00] lorddavidiii leaves the room: Connection failed: host is unreachable
[06:10:23] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[06:10:27] adrien leaves the room: Stream closed by us: Replaced by new connection (conflict)
[06:20:37] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[06:30:56] ta leaves the room: Disconnected: Replaced by new connection
[06:31:10] robert_mobil leaves the room: Stream closed by us: Replaced by new connection (conflict)
[06:32:08] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[06:33:43] srgcdev leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[06:42:07] pod leaves the room
[06:43:11] Marzanna leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[06:46:07] bammes leaves the room
[06:52:24] ingolf leaves the room
[06:55:46] Alacer leaves the room
[07:02:15] fp leaves the room: Stream closed by us: Replaced by new connection (conflict)
[07:03:09] rom1dep leaves the room
[07:08:22] alexis leaves the room: Stream closed by us: Replaced by new connection (conflict)
[07:15:09] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[07:15:18] marc leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[07:16:22] rom1dep leaves the room
[07:17:22] Alacer leaves the room
[07:18:14] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[07:32:52] joekokker leaves the room: Stream closed by us: Replaced by new connection (conflict)
[07:40:05] sindrake leaves the room
[07:53:08] bammes leaves the room: Rechner geht in den Ruhezustand
[07:59:34] <404> hello. How to check config file ejabberd.yml ?
[07:59:39] joekokker leaves the room
[07:59:56] joekokker leaves the room: Stream closed by us: Replaced by new connection (conflict)
[08:00:19] 404 is now known as 404.city
[08:00:31] 404.city leaves the room
[08:01:30] <Licaon_Kter> 404: check what? Correctness?
[08:02:46] <404> Licaon_Kter, Yes, without rebooting
[08:03:35] <Licaon_Kter> Reboot? You means restarting the program I hope.
[08:05:16] <404> Licaon_Kter,  Check and no Restart
[08:05:41] <Licaon_Kter> It has been said before, while the new setup is that it stops on config errors, it will NOT stop on reload.
Hence, open a console with `tail -f ejabberd.log`
And another one with `ejabberdctl reload_config`
[08:07:32] vogt leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:07:33] <404> Licaon_Kter,  I do so now, but I thought there is another method
[08:08:09] <404> Licaon_Kter,  Thanks
[08:08:57] <Licaon_Kter> Fix anything you see in the log, since the server will not restart if it stops in the future
[08:08:58] <Holger> I agree a "check_config" command would be nice.
[08:09:36] carlos leaves the room
[08:10:17] bammes leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:11:24] ata2001 leaves the room: Disconnected: closed
[08:12:07] <cromain> for a command to run, you need ejabberd started
[08:13:58] <zuglufttier> https://social.tchncs.de/@trashserver/100309482266518821
5GB RAM for 1200 concurrent users, can't we do a crowdfunding for an ejabberd installation? :D
[08:14:50] <404> Thanks cromain, Holger Licaon_Kter  
[08:15:54] sindrake leaves the room: Disconnected: Replaced by new connection
[08:16:17] ta leaves the room
[08:16:26] <Holger> cromain: Yes this would have to be special-cased.  I think it would be nice to have both a proper ejabberd 'command' and an "ejabberctl check_config" call that works without a running instance.
[08:17:16] <Holger> cromain: Might not be worth the hassle, dunno.  I'm just making the highly useful comment that it would be nice to have :-)
[08:17:29] deavmi leaves the room: Disconnected: Received SIGTERM
[08:20:43] <404> zuglufttier, Because of this shit, I had to move the server from Prosody to Ejabberd
[08:22:06] <cromain> Holger: i want to rewrite ejabberdctl to get it cleaner/simpler. we need to think about how to make it check_config, maybe just firering erlang node and call a validator function
[08:22:20] <404> zuglufttier, The server crashed every day
[08:22:43] <Licaon_Kter> zuglufttier: not really
He says: https://social.tchncs.de/@trashserver/99804958194701061 (you responded already)
He wants: https://social.tchncs.de/@trashserver/99804992984926781
[08:22:49] Licaon_Kter leaves the room
[08:23:46] <zuglufttier> Licaon_Kter, yeah, I know. He wants to live on the edge!
[08:23:48] <Holger> Yes he's a Go fanboy and wants to wait until https://github.com/ortuman/jackal is usable.  If you look at the commit history this is probably going to take a while.
[08:24:30] <Licaon_Kter> zuglufttier: not really
He says: https://social.tchncs.de/@trashserver/99804958194701061 (you responded already)
He wants: https://social.tchncs.de/@trashserver/99804992984926781
[08:24:44] <Holger> Duplicate alarm!!
[08:25:39] Licaon_Kter leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:26:21] Licaon_Kter leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:26:47] <zuglufttier> Holger, did you get that as well? Bug in 18.06? :D
[08:27:03] <404> >An XMPP server written in Go.
[08:27:05] <404> fuck
[08:27:19] <Licaon_Kter> Holger: what duplicate?🤷
[08:28:01] <Holger> Licaon_Kter: We received your "zuglufttier: not really" message twice.  My crystal ball would assume your client sent it twice, but who knows.
[08:28:23] <404> Erland is good at the task
[08:28:35] <Licaon_Kter> No, C crashes on a emoji+) combo....
[08:28:45] <Holger> Licaon_Kter: You could check your logs whether you reconnected around 08:22 UTC.
[08:28:57] <Holger> Licaon_Kter: I would assume you did.
[08:29:04] <Licaon_Kter> I crashed like 5 times....sorry for the noise
[08:29:12] <Holger> Oh heh.
[08:29:19] <Holger> !version Licaon_Kter
[08:29:21] <Servant> Holger: Licaon_Kter is running Conversations version 2.2.4-fcr on Android
[08:29:44] <Licaon_Kter> It's not a new crash, btw
[08:30:06] <debalance> I did not receive that duplicate
[08:30:14] <Holger> Licaon_Kter: The noise is no problem.  If it's not a server-side issue, I'm happy ;-)
[08:30:15] <debalance> not really
[08:30:17] <Holger> debalance: Uhm.
[08:31:15] <Holger> debalance: The duplicate is stored in this room's MAM archive though (I see it when ditching/re-fetching the history with Conversations).
[08:31:47] <zuglufttier> I'm using gajim.
[08:32:00] Holger <- MCabber.
[08:32:27] <Holger> I was super-intelligent (as usual) and didn't look at the history in Conversations before re-fetching it.
[08:32:37] <Holger> Maybe it does some magic dedup.
[08:32:55] <debalance> Funny, my two gajim instances don't show the duplicate, but my conversation instance does
[08:33:12] <Holger> Probably just an unrelated Gajim bug eating messages :-)
[08:33:21] ata2001 leaves the room: Disconnected: closed
[08:33:28] <zuglufttier> The second message came in one second later.
[08:34:25] <debalance> or maybe it's a feature ;)
[08:34:33] <Holger> You never know.
[08:36:44] Licaon_Kter leaves the room: Stream closed by us: Replaced by new connection (conflict)
[08:37:48] ChaosKid42 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:39:37] frainz leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[08:49:18] fp leaves the room: Stream closed by us: Replaced by new connection (conflict)
[08:53:05] sindrake leaves the room
[09:16:48] fp leaves the room: Stream closed by us: Replaced by new connection (conflict)
[09:22:48] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[09:27:50] Marzanna leaves the room
[09:27:51] Marzanna leaves the room
[09:31:36] spicewiesel leaves the room: Stream closed by us: Replaced by new connection (conflict)
[09:35:00] alexis leaves the room: Stream closed by us: Replaced by new connection (conflict)
[09:40:50] ata2001 leaves the room: Disconnected: Replaced by new connection
[09:42:43] alexis leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[09:43:28] bammes leaves the room: Rechner geht in den Ruhezustand
[09:48:24] xinit leaves the room
[10:02:06] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[10:07:53] rom1dep leaves the room: Stream closed by us: Replaced by new connection (conflict)
[10:19:03] 4223 leaves the room
[10:24:36] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[10:31:22] Alacer leaves the room
[10:31:29] nop leaves the room
[10:31:29] mrDoctorWho leaves the room
[10:31:29] mrDoctorWho leaves the room
[10:36:32] Licaon_Kter leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[10:37:02] robert_mobil leaves the room: Stream closed by us: Replaced by new connection (conflict)
[10:38:32] Andrew Nenakhov leaves the room
[10:39:48] ta leaves the room: Disconnected: Replaced by new connection
[10:40:26] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[10:40:58] joekokker leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[10:45:56] jonasw leaves the room
[10:47:45] ta leaves the room: Disconnected: Replaced by new connection
[10:53:15] ta leaves the room: Disconnected: Replaced by new connection
[10:55:36] ta leaves the room: Disconnected: Replaced by new connection
[11:01:00] pod leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:02:30] pod leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:04:24] ta leaves the room: Disconnected: Replaced by new connection
[11:04:50] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[11:05:36] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:07:37] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:09:35] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:12:52] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:20:29] ThUnD3r|Gr33n® leaves the room
[11:23:08] Alacer leaves the room
[11:27:25] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:27:39] ThUnD3r|Gr33n® leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:31:55] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:31:58] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:32:37] ChaosKid42 leaves the room
[11:34:54] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:35:49] ChaosKid42 leaves the room
[11:36:09] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:36:28] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:37:25] ata2001 leaves the room: Disconnected: closed
[11:38:12] ingolf leaves the room
[11:38:54] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:41:35] cippaciong leaves the room
[11:45:14] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[11:47:48] alexis leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:50:06] vanitasvitae leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[11:56:41] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[11:58:28] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:01:37] jeremy leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:03:27] mimi89999 leaves the room
[12:07:13] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:07:21] vanitasvitae leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:07:37] mimi89999 leaves the room
[12:08:38] mimi89999 leaves the room
[12:11:53] ChaosKid42 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:13:49] fp leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:14:14] nekit leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:18:05] xinit leaves the room
[12:27:00] Licaon_Kter leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:27:55] cromain leaves the room
[12:29:04] mimi89999 leaves the room
[12:29:11] mimi89999 leaves the room: Disconnected: Replaced by new connection
[12:30:41] mightyBroccoli leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:31:51] jere leaves the room: Disconnected: Replaced by new connection
[12:35:54] mimi89999 leaves the room: Disconnected: Replaced by new connection
[12:36:11] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:38:13] <zinid> Sounds like some hand crafted deduplication, that's Gajim after all 😁
[12:39:57] <zinid> > Yes he's a Go fanboy and wants to wait until https://github.com/ortuman/jackal is usable.  If you look at the commit history this is probably going to take a while.
The funny thing is that with Go there will be the same result: due to lack of runtime introspection bugs will start accumulating and nobody knows what's the reason
[12:41:05] <MattJ> For the record there is no lack of runtime introspection in Prosody/Lua. I'm not a Go developer, but I'd be surprised if they don't have similar tools
[12:41:15] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:42:41] <zinid> MattJ: why does his server crashes every day then?
[12:42:43] <MattJ> I have a complete browsable dump of the Lua memory environment from one of the affected servers, and I can execute commands and introspect objects on a running server
[12:42:58] <zinid> Can't someone help him to investigate?
[12:43:00] <MattJ> I was due to have a debugging session with one of the admins (of a different server) today, but they couldn't make it
[12:43:27] <MattJ> It's a problem that was introduced in a recent nightly build, or in one of our community modules and it is affecting multiple servers
[12:43:38] <MattJ> Working on debugging it currently
[12:43:46] <zinid> Okay
[12:44:01] fp leaves the room: Stream closed by us: Replaced by new connection (conflict)
[12:45:31] <MattJ> I'm not an erlang developer either, but assuming you follow the actor model then I agree debugging memory issues there is going to be much easier than in a language that allows all-too-easy state sharing across module boundaries
[12:45:50] <Holger> MattJ: So you can query and modify the state of individual c2s/s2s connections, kill them (if they're in a blocking call), figure out which one eats most resources?
[12:46:50] <MattJ> Holger, filtering the erlang-specific aspects of your statement, yes
[12:47:16] <zinid> MattJ: erlang had internal shell which looks like Unix shell but with erlang syntax, so you can list processes, kill them, restart them, modify there code and so on
[12:47:25] <Holger> I tried to filter them myself, seems I failed 🙂
[12:48:04] <MattJ> Lua doesn't have "processes", though Prosody trunk is moving towards that model
[12:48:19] sindrake leaves the room
[12:48:55] <zinid> MattJ: does Lua support multithreading without GIL?
[12:49:23] <Holger> MattJ: Sounds good, though I do assume you need to bend Lua quite a bit to get these things.
[12:49:26] <MattJ> zinid, depends on the model you choose. Lua doesn't have any threading by default
[12:49:28] vogt leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[12:50:13] <MattJ> Holger, Lua has a debug API which allows introspection (and we have tools to dump a memory state graph from this, to find large objects and what references them), and it has a coroutine API
[12:50:44] <zinid> Not sure why all the hassle...
[12:51:14] <MattJ> Hassle?
[12:51:42] <zinid> I'll probably never understand that, when instead of using already existing bullet proof solution people want to reinvent a wheel
[12:52:15] <MattJ> You're talking to the person who restarted ejabberd multiple times per day on jabber.org for quite some time
[12:52:20] <MattJ> So don't tell me about "bullet-proof"
[12:52:41] <MattJ> I believe there are different programming models, and they all have their strengths and weaknesses
[12:52:42] <zinid> MattJ: try it now 😁
[12:52:49] fp leaves the room: Stream closed by us: system-shutdown
[12:52:55] <MattJ> There is no magic model that will just solve all problems forever, not even erlang's :)
[12:53:23] <zinid> MattJ: still for our task erlang is better than Lua
[12:53:57] <MattJ> I have no doubt ejabberd (and erlang) has improved since 10 years ago, but I know it well enough to know that it's not what I want to spend my day on
[12:54:16] <MattJ> Holger was experiencing performance issues he mentioned just the other day
[12:54:27] newbie leaves the room
[12:54:53] <zinid> Holger: what performance problem?
[12:54:55] <Holger> MattJ: But the Lua strength is "easy to use/extend", no? It wasn't designed nor used for internet service kind of things so you do have to work around the limitations, which I agree with zinid sounds like the wrong approach to me. But the discussion is moot, you won't rewrite Prosody obviously.
[12:55:02] <Holger> MattJ: I didn't.
[12:55:58] <zinid> MattJ: also, according to your logic, after 10 years of development Lua crashes every day on trashserver
[12:56:12] <Link Mauve> Just two years ago, when I inherited JabberFR, migrating to Prosody was a much better option than upgrading Ejabberd, in big part because the community was very helpful.
[12:56:27] <MattJ> Holger, zinid, then apologies, I guess I misinterpreted
[12:56:29] <Holger> MattJ: I mentioned that I see a CPU usage greater than zero and was interested in testing whether compiling ejabberd to native code makes a difference that's all.
[12:57:29] <Holger> MattJ: And I know ejabberd was buggy in the past but AFAICS this was in no way talked to the language choice, so it's kinda besides the point.
[12:57:33] fp leaves the room
[12:57:41] <Holger> *related
[12:58:04] <MattJ> 10-50% usage on 4 cores on what I assume is a service nowhere near the size of jabber.org seemed like an issue to me
[12:58:33] <zinid> MattJ: let's do benchmarking?
[12:58:43] <MattJ> But I don't really care, believe me that I'm glad that ejabberd exists, is better than it was before, and it suits some people great
[12:58:44] <Holger> Link Mauve: Community is also not really related 🙂 And people seem to be happy with our support these days.
[12:58:44] <zinid> I have a benchmark tool
[12:58:54] <Link Mauve> Holger, looks like it.
[12:59:12] <Link Mauve> Me knowing Prosody’s internals more than Ejabberd’s also played a big role.
[13:00:05] <zinid> MattJ: if you assist me to install prosody and tune it I can benchmark both on the same hardware
[13:00:06] <MattJ> zinid, I'm up for something like that, just out of curiosity
[13:00:35] <zinid> Ok, I'm starting the benchmarking machine
[13:01:35] <zinid> MattJ, do you have some time or should we do that later
[13:01:41] <Holger> MattJ: Sigh I was wondering whether I need to look closer before making public statements on the CPU usage 🙂 It's more like 1-5% overall, with a load of 0.1-0.3 or something. In Irix mode (or non-Irix? dammit) top comes up with those larger numbers.
[13:01:47] <zinid> or maybe someone else can help me at least to configure Prosody?
[13:03:01] <MattJ> zinid, start with the steps here: https://prosody.im/download/package_repository#terminal
[13:03:03] <Link Mauve> zinid, I probably can, which distribution are you running?
[13:03:22] <zinid> Link Mauve, some AWS crap, looks like fedora, there is yum
[13:03:22] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:03:40] <MattJ> 0.10.2 should be available there
[13:03:46] <MattJ> which is the latest
[13:05:02] <SouL> Quite an interesting chat you've got there...
[13:05:15] <SouL> ^^
[13:06:07] <Holger> 🍿
[13:07:10] cippaciong leaves the room: Stream closed by us: Replaced by new connection (conflict)
[13:09:24] <SouL> Exactly 😸
[13:12:41] ta leaves the room
[13:13:20] <Holger> MattJ: BTW for *me* the #1 Erlang feature isn't introspection (that's #2) but the preemptive scheduler which makes programming concurrent workloads extremely convenient.  You wont get anyway close with coroutines.
[13:13:41] <MattJ> Holger, why not?
[13:14:10] <Holger> It's not like you can't get the same result with other languages/VMs, but it requires lots of thought on the developers side.  I much prefer having that abstracted away by the VM.
[13:14:53] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:15:16] <Holger> MattJ: Because you need to think about when to 'yield' yourself.
[13:15:46] <MattJ> Not necessarily, the scheduler can do that, and it can yield in blocking calls, etc.
[13:16:57] <zinid> > Holger, why not?
the same reason because modern OSes have preemptive scheduler
[13:17:06] <Holger> MattJ: Yes, my point is that the BEAM scheduler does way better job on that :-)
[13:17:19] <MattJ> I've honestly no doubt that it does :)
[13:17:53] <MattJ> If my #1 goal was developing a system with that model as a requirement, it would be stupid not to use it
[13:18:36] <Holger> If you do this manually (or by yielding at blocking syscalls only), you get it wrong in a single place and any non-trivial workload falls apart.
[13:18:42] <MattJ> But as I said earlier, I don't think it's a magic bullet. We did benchmarks in the early days of Prosody. In "SMP" mode ejabberd performed far worse
[13:19:20] mimi89999 leaves the room
[13:19:31] <Holger> Isn't good concurrency is a primary requirement of any Internet daemon potentially serving many clients?
[13:20:00] <MattJ> The old "events vs threads" debate that will never end? :)
[13:21:07] <zinid> # yum install prosody
^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C^C
[13:21:10] <zinid> fedora...
[13:21:35] <zinid>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                          
4431 root      20   0  365096  29564  14616 S   0.0  0.0   0:00.08 yum                                              
4604 root      20   0  362556  29556  14604 S   0.0  0.0   0:00.07 yum            
[13:21:44] <zinid> what a piece of shit
[13:22:13] <MattJ> Not a fedora user, so no idea why it would behave like that
[13:22:22] <Holger> MattJ: I think it's hard to argue against my assumptions that #1 implementing a mix of both is most efficient (i.e. choose a number of threads depending on the number of cores) and that #2 exposing a thread model to the *developer* (like with coroutines) is most convenient.
[13:22:38] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[13:23:27] <zinid> openat(AT_FDCWD, "/var/lib/rpm/.dbenv.lock", O_RDWR|O_CREAT, 0644) = 3
umask(022)                              = 022
fcntl(3, F_SETLKW, {l_type=F_WRLCK, l_whence=SEEK_SET, l_start=0, l_len=0}
[13:23:53] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:23:54] <MattJ> Holger, yet all this assumes you're severely CPU or I/O bound, which generally isn't reality in XMPP (outside of benchmarks)
[13:25:55] <Johan> Hi all, I have 3 connected machines. All of them run an ejabberd daemon. They all try to join the same cluster but sometimes one of the ejabberdctl join_cluster commands fails. Any idea how I could activate some debug information to get an idea why such a command can fail?
[13:26:47] <Johan> When such a command, I end up with one cluster of size 2 and one cluster of size 1 ;-/
[13:26:57] <Johan> fails*
[13:27:35] <Johan> the all run ejabberd 17.11
[13:27:40] <zinid> Johan, are you trying to join them everytime on start? The join procedure should be only performed once
[13:28:13] <Johan> Yes I do try th join them everytime on start as I currently put the spool_dir on /tmp
[13:28:38] <Johan> the rootfs is in read-only, so I don't have any persistent partition I can write to.
[13:29:25] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:31:55] mightyBroccoli leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:33:50] <zinid> yeah, great
[13:34:10] <zinid> then you should full detach them an rejoin
[13:34:38] <Holger> MattJ: AFAIK distribution accross cores can help with I/O waiting as well, but my point about preemtive scheduling wasn't related to multi-core usage, I was just commenting on "events vs. threads" now.  Quite the opposite, my point was that the scheduler makes it most convenient and robust to program N processes/threads/whatever-you-call-them on M cores, where N > M.
[13:35:18] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:35:26] <MattJ> Sure, I agree it's a handy model to have
[13:36:50] <Johan> What do you mean by full detach? If the join_cluster fails, I should:
ejabberdctl stop; rm -f /path/to/ejabberd/*.{DAT,DCD}; ejabberdctl start; ejabberdctl join_cluster
No matter what the reason of the failure on the join_cluster was?
[13:37:17] <zinid> MattJ,
# yum install prosody
Loaded plugins: langpacks, priorities, update-motd
No package prosody available.
Error: Nothing to do
[13:37:23] <Holger> MattJ: Oh you're saying XMPP is neither CPU *nor* I/O bound?  What's the bottleneck then? :-)
[13:37:23] <zinid> MattJ, should probably install from sources
[13:38:29] <Link Mauve> Holger, often it’s memory.
[13:38:41] <zinid> Link Mauve, depends on usage
[13:38:51] <Link Mauve> Sure.
[13:39:23] <nosnilmot> Amazon Linux?
yum install epel-release
yum-config-manager --enable epel
yum install prosody
[13:39:48] <zinid> yum install epel-release
Loaded plugins: langpacks, priorities, update-motd
No package epel-release available.
Error: Nothing to do
[13:40:55] <nosnilmot> cat /etc/system-release ?
[13:41:09] sindrake leaves the room
[13:41:28] <zinid> # cat /etc/system-release
Amazon Linux release 2 (2017.12) LTS Release Candidate
[13:41:45] <nosnilmot> oh
[13:43:32] <ChaosKid42> https://aws.amazon.com/de/premiumsupport/knowledge-center/ec2-enable-epel/
[13:43:32] <zinid> https://forums.aws.amazon.com/thread.jspa?threadID=271156
[13:45:29] <zinid> Installing:
prosody                         x86_64                  0.10.2-1.el7
[13:45:35] <zinid> I guess that's the recent one?
[13:46:23] <zinid> installed, started
[13:46:47] <Johan> oh, maybe I need to add a --no-timeout parameter to my ejabberdctl calls?
[13:46:56] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:48:03] <MattJ> zinid, yes, that's the latest stable
[13:48:55] <zinid> Benchmark failure: Authentication failed: Peer responded with error: Unable to authorize you with the authentication credentials you've sent. (not-authorized)
[13:49:09] <MattJ> zinid, prosodyctl register username hostname password
[13:49:10] <zinid> ok, need to generate users and rosters
[13:49:18] <zinid> also, need to configure postgresql
[13:50:20] <zinid> or will we test on flat files?
[13:51:20] <MattJ> zinid, I'm fine with files
[13:51:31] <MattJ> Would probably prefer it to be honest :)
[13:51:35] <zinid> do you have any scripts to populate the database?
[13:52:23] <MattJ> You mean rosters and such? If so, nothing handy afaik
[13:52:42] <zinid> ok, we can try without rosters, but this will not be fair
[13:53:26] <Holger> Link Mauve: I think it's actually often CPU or I/O bound, e.g. when all clients re-log in in parallel after server restart or something (what's the effect called where the load increase is worse than O(N) due to bad interactions?).  Though large parts of that load will often be on the auth/storage backends of course.
[13:53:29] <zinid> it will consume much less memory and will not route presences, which consumes a lot of cpu
[13:54:25] <Link Mauve> zinid, you could use ejabberd(sql?)2prosody.lua on a pre-populated Ejabberd database.
[13:54:29] <Link Mauve> zinid, you could use ejabberd(sql?)2prosody.lua on a pre-populated Ejabberd database dump.
[13:55:04] <zinid> Link Mauve, I have 20M rows in roster table, this will probably take eternity
[13:55:14] <zinid> also, I doubt prosody can handle 1M of users
[13:55:48] <Link Mauve> Probably, if so you can use my mod_storage_ejabberd_sql_readonly, it implements the part of the Ejabberd MySQL layout as a readonly Prosody storage.
[13:55:56] <zinid> argh
[13:56:04] <zinid> like I have nothing more to do 🙂
[13:56:05] <Link Mauve> Using prosodyctl migrate it can export it to plain files.
[13:56:39] <Link Mauve> I implemented it against the 2.13 (?) table layout, if that has changed it may be out of date.
[13:56:43] <Holger> Hehe will mod_storage_ejabberd_sql_readonly perform better than the standard SQL storage backend? :-)
[13:56:56] <Holger> I've heard complaints about the schema (but I'm clueless).
[13:57:01] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:57:03] <zinid> Holger, they are trying to cheat, I will enable p1db then 😉
[13:57:07] <Link Mauve> Holger, no idea, I’d expect it to perform way worse than flat files though.
[13:57:16] <Holger> zinid: :-)
[13:57:26] <Link Mauve> Hence the migration suggestion.
[13:57:30] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[13:57:51] <Holger> If flat files perform best this *does* sound like a problem with a schema.
[13:58:03] <zinid> Link Mauve, can't we just use SQL for prosody? I think that's what is used on trashserver for example
[13:58:03] <Link Mauve> ejabberdsql2prosody.lua didn’t work for my 220k users, it was using the 8 GiB of RAM I had to give it just to parse the dump in memory.
[13:58:32] <Link Mauve> zinid, that’s most likely part of their performances issues.
[13:58:46] <zinid> and if you just give me an example row of roster in CSV format I will generate rows using my script and load it into pgsql using \copy
[13:59:01] <zinid> Link Mauve, then what should we test?
[13:59:24] <Link Mauve> zinid, if possible plain files.
[13:59:36] <zinid> Holger, you see 😀
[13:59:40] <Link Mauve> You could try the mod_storage_ejabberd_sql_readonly first, if that’s easier for you.
[13:59:58] <Link Mauve> Anyway, time to eat, I didn’t endure this thesis for nothing!
[14:00:04] <zinid> Link Mauve, I don't even know what to do with this mod_storage_something_something
[14:00:20] <Holger> Isn't it trivial to create a script that creates the flat-file users?
[14:00:40] <zinid> ok, let's start without rosters, fuck that
[14:00:43] <zinid> how to generate users?
[14:01:02] <zinid> is it that hard to write
for i to 10000 do write("file data") into file?
[14:01:27] <zinid> what is the format of a flat user file?
[14:01:33] <zinid> can I have an example?
[14:01:41] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:02:29] <Holger> My Prosody server is on my b0rked disks :-/
[14:03:53] <zinid> # cat /var/lib/prosody/localhost/accounts/tsung1.dat
return {
        ["iteration_count"] = 4096;
        ["stored_key"] = "d67604bb64e460661ef19d27ac9dde19c51d5e33";
        ["salt"] = "a179c59f-882a-4f2a-a0db-0720cb254a44";
        ["server_key"] = "b45ff7e06f58abfa2a1c4a212a6f96b64860cd2a";
};
[14:03:56] <zinid> lol, SCRAM
[14:04:20] <Holger> "prosodyctl adduser test@example.com" and then look at the result?
[14:04:22] <Holger> Ah.
[14:04:34] <zinid> how to disable scram?
[14:05:25] <Holger> mod_auth_internal_plain instead of mod_auth_internal_hashed, as I understand it.
[14:05:50] Holger <-- Prosody support guy.
[14:06:28] <zinid> ah, found
[14:06:34] <zinid> yeah, thanks
[14:07:04] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:07:24] <zinid> # cat /var/lib/prosody/localhost/accounts/tsung1.dat
return {
        ["password"] = "tsung1";
};
[14:07:26] ingolf leaves the room
[14:07:37] <zinid> the populating script will be brutal 🙂
[14:09:20] cippaciong leaves the room
[14:10:55] <zinid> Benchmark failure: localhost doesn't support required features: urn:xmpp:csi:0 and urn:xmpp:sm:3
[14:11:10] <Holger> mod_smacks for the latter.
[14:11:19] <zinid> Holger, their default configs sucks even more than ours 😀
[14:11:37] Akasch leaves the room: Rechner geht in den Ruhezustand
[14:11:41] <Holger> Yes those things are community modules.
[14:12:08] <Holger> I think there's more than one module for CSI.
[14:12:41] <Holger> The original one was mod_csi which does nothing but announcing the feature and adding hooks to handle <active/> and <inactive/>.
[14:12:58] <Holger> mod_throttle_presence and mod_filter_chatstates will handle those hooks.
[14:13:09] pod leaves the room
[14:13:49] <zinid> Jul 03 14:12:23 modulemanager   error   Unable to load module 'smacks': /usr/lib64/prosody/modules/mod_smacks.lua: No such file or directory
[14:14:01] <Holger> Those things are community modules.
[14:14:07] <Holger> Prosody is easier to set up :-P
[14:14:07] <zinid> sigh
[14:14:22] <zinid> always remember!
[14:14:41] <Holger> https://prosody.im/doc/installing_modules
[14:15:02] <4223> > Those things are community modules.
> Prosody is easier to set up :-P
😥😥😥
[14:15:06] <Holger> plugin_paths = { "/path/to/modules", "/path/to/more/modules" }
[14:15:26] <Holger> And then "hg clone https://hg.prosody.im/prosody-modules/ /path/to/more/modules".
[14:15:28] <Holger> Or something.
[14:16:29] <MattJ> Indeed, hg clone https://hg.prosody.im/prosody-modules/ /etc/prosody/modules
[14:16:41] <MattJ> and plugin_paths = { "/etc/prosody/modules" } near the top of the config
[14:16:44] <zinid> argh, that fucking commans and brackets
[14:16:49] <zinid> *commas
[14:17:22] <MattJ> Ok
[14:18:23] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:19:00] <Holger> zinid: Do you enable MAM in your benchmarks BTW?
[14:19:07] <zinid> Holger, sure
[14:19:14] <MattJ> Heh
[14:19:44] <zinid> now I have: Authentication failed: Peer responded with error: Unable to authorize you with the authentication credentials you've sent. (not-authorized)
[14:19:55] <zinid> why? nothing in the log
[14:20:00] <zinid> I had it working before restart
[14:20:02] <MattJ> "nothing"?
[14:20:16] <zinid> Jul 03 14:19:16 c2s2eec500      info    Client disconnected: closed
Jul 03 14:19:16 c2s2eeeb10      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f27f00      info    Client connected
Jul 03 14:19:16 c2s2eda130      info    Client disconnected: closed
Jul 03 14:19:16 c2s302d2c0      info    Client connected
Jul 03 14:19:16 c2s2f66f20      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f3da10      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f81490      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f0fae0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f56040      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2ece4d0      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f446a0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s301da50      info    Client connected
Jul 03 14:19:16 c2s2fd8d50      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f200b0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2ec0c70      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f36b80      info    Client disconnected: closed
Jul 03 14:19:16 c2s3075c80      info    Client connected
Jul 03 14:19:16 c2s307a540      info    Client connected
Jul 03 14:19:16 c2s2f81490      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f0fae0      info    Client disconnected: closed
Jul 03 14:19:16 c2s2eceaa0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f56040      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f446a0      info    Client disconnected: closed
Jul 03 14:19:16 c2s2e51d20      info    Client connected
Jul 03 14:19:16 c2s2f66f20      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f3da10      info    Client disconnected: closed
Jul 03 14:19:16 c2s2e2d0f0      info    Client connected
Jul 03 14:19:16 c2s2fd8d50      info    Client disconnected: closed
Jul 03 14:19:16 c2s2f200b0      info    Client disconnected: closed
Jul 03 14:19:16 c2s2fec9e0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s2f27f00      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s3089660      info    Client connected
Jul 03 14:19:16 c2s303e350      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s30700c0      info    Client connected
Jul 03 14:19:16 c2s2eceaa0      info    Client disconnected: closed
Jul 03 14:19:16 c2s3012e80      info    Client connected
Jul 03 14:19:16 c2s302d2c0      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s30c4600      info    Client connected
Jul 03 14:19:16 c2s2f27f00      info    Client disconnected: closed
Jul 03 14:19:16 c2s30c8650      info    Client connected
Jul 03 14:19:16 c2s2fec9e0      info    Client disconnected: closed
Jul 03 14:19:16 c2s303e350      info    Client disconnected: closed
Jul 03 14:19:16 c2s301da50      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:19:16 c2s30fc140      info    Client connected
Jul 03 14:19:16 c2s2e51d20      info    Client disconnected: closed
Jul 03 14:19:16 c2s3012e80      info    Client disconnected: ssl handshake error: closed
Jul 03 14:19:16 c2s3089660      info    Client disconnected: ssl handshake error: closed
Jul 03 14:19:16 c2s30700c0      info    Client disconnected: ssl handshake error: closed
Jul 03 14:19:16 c2s2e2d0f0      info    Client disconnected: ssl handshake error: closed
Jul 03 14:19:16 c2s30fc140      info    Client disconnected: closed
Jul 03 14:19:16 c2s30c8650      info    Client disconnected: ssl handshake error: closed
Jul 03 14:19:16 c2s302d2c0      info    Client disconnected: closed
Jul 03 14:19:16 c2s301da50      info    Client disconnected: closed
Jul 03 14:19:16 c2s3075c80      info    Client disconnected: closed
Jul 03 14:19:16 c2s307a540      info    Client disconnected: closed
Jul 03 14:19:16 c2s30c4600      info    Client disconnected: ssl handshake error: closed
[14:20:19] <zinid> here you go!!!!
[14:20:35] <Holger> zinid: Totally makes sense to get it closer to real-world usage, but I guess you can't send too many messages without making the DB backend the bottleneck then?
[14:20:51] <zinid> # cat /var/log/prosody/prosody.err
Jul 03 14:12:23 modulemanager   error   Unable to load module 'smacks': /usr/lib64/prosody/modules/mod_smacks.lua: No such file or directory
[14:21:04] <MattJ> zinid, I'm guessing that's from before
[14:21:17] <zinid> yeah, now there is nothing in error log
[14:21:32] <MattJ> Enable debug logging then, it's probably not doing authentication where you expect
[14:21:43] <Holger> zinid: Hashed vs. plain passwords?
[14:21:47] <zinid> # ls -la localhost/accounts/ | wc -l
100003
[14:21:49] <zinid> Holger, plain
[14:22:04] <Holger> zinid: Did it already work after switching to plain?
[14:22:06] <zinid> MattJ, "enable logging"
[14:22:11] <zinid> Holger, yes!
[14:22:20] <MattJ> zinid, find the line that says "log = ...stuff..."
[14:22:20] <zinid> hum, or maybe not...
[14:22:27] <MattJ> and change info to debug
[14:22:40] sindrake leaves the room
[14:22:42] spicewiesel leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:22:52] <zinid> MattJ, found, thanks
[14:23:01] cippaciong leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:24:11] <zinid> Jul 03 14:23:57 socket  debug   server.lua: accepted new client connection from 10.0.1.243:23338 to 5222
Jul 03 14:23:57 c2s2ed1890      info    Client connected
Jul 03 14:23:57 c2s2ed1890      debug   Client sent opening <stream:stream> to localhost
Jul 03 14:23:57 c2s2ed1890      debug   Sent reply <stream:stream> to client
Jul 03 14:23:57 c2s2ed1890      debug   Not offering authentication on insecure connection
Jul 03 14:23:57 c2s2ed1890      debug   Received[c2s_unauthed]: <starttls xmlns='urn:ietf:params:xml:ns:xmpp-tls'>
Jul 03 14:23:57 socket  debug   server.lua: we need to do tls, but delaying until send buffer empty
Jul 03 14:23:57 c2s2ed1890      debug   TLS negotiation started for c2s_unauthed...
Jul 03 14:23:57 socket  debug   server.lua: attempting to start tls on tcp{client}: 0x2f302a8
Jul 03 14:23:57 socket  debug   server.lua: ssl handshake done
Jul 03 14:23:57 c2s2ed1890      debug   Client sent opening <stream:stream> to localhost
Jul 03 14:23:57 c2s2ed1890      debug   Sent reply <stream:stream> to client
Jul 03 14:23:57 c2s2ed1890      info    Stream encrypted (TLSv1.2 with ECDHE-RSA-AES256-GCM-SHA384)
Jul 03 14:23:57 c2s2ed1890      debug   Not offering disabled mechanism DIGEST-MD5
Jul 03 14:23:57 c2s2ed1890      debug   Offering mechanism PLAIN
Jul 03 14:23:57 c2s2ed1890      debug   Offering mechanism SCRAM-SHA-1
Jul 03 14:23:57 c2s2ed1890      debug   Offering mechanism SCRAM-SHA-1-PLUS
Jul 03 14:23:57 c2s2ed1890      debug   Received[c2s_unauthed]: <auth mechanism='PLAIN' xmlns='urn:ietf:params:xml:ns:xmpp-sasl'>
Jul 03 14:23:57 localhost:auth_internal_plain   debug   get_password for username 'tsung1'
Jul 03 14:23:57 localhost:saslauth      debug   sasl reply: <failure xmlns='urn:ietf:params:xml:ns:xmpp-sasl'><not-authorized/><text>Unable to authorize you with the authentication credentials you&apos;ve sent.</text></failure>
Jul 03 14:23:57 c2s2ed1890      debug   Received </stream:stream>
Jul 03 14:23:57 c2s2ed1890      debug   c2s stream for 10.0.1.243 closed: session closed
Jul 03 14:23:57 c2s2ed1890      debug   Destroying session for (unknown) ((unknown)@localhost)
Jul 03 14:23:57 c2s2ed1890      info    Client disconnected: connection closed
Jul 03 14:23:57 c2s2ed1890      debug   Destroying session for (unknown) ((unknown)@(unknown))
Jul 03 14:23:57 socket  debug   server.lua: closed client handler and removed socket from list
Jul 03 14:24:00 localhost:mam   debug   Removing old messages for user "tsung1"
[14:25:08] <zinid> ah
[14:25:09] <MattJ> Looks like a simple password mismatch, or account doesn't exist, or something
[14:25:11] <zinid> found the issue
[14:26:54] <zinid> Benchmark failure: localhost doesn't support required feature: urn:xmpp:csi:0. Set option csi to false in order to ignore this.
[14:27:03] <zinid> mod_csi?
[14:27:47] <MattJ> Yes
[14:27:58] <zinid> 16:27:47.592 [critical] Benchmark failure: Resource binding failure: service-unavailable
[14:27:59] <zinid> wow
[14:28:13] <Holger> zinid: As I said above: mod_csi + mod_throttle_presence + mod_filter_chatstates to get an approximation of our default behavior.
[14:28:30] <zinid> Jul 03 14:27:47 c2s226a700      debug   Received[c2s_unbound]: <iq id='10688932059631306753' type='set' to='tsung1@localhost' from='tsung1@localhost/xmppb'>
Jul 03 14:27:47 stanzarouter    debug   Unhandled c2s_unbound stanza: iq; xmlns=urn:ietf:params:xml:ns:xmpp-bind
[14:28:43] <zinid> Holger, ok
[14:29:30] <zinid> Holger, sorry, I'm not always reading, I'm creating!!!
[14:29:45] <Holger> Sure sure, you're the artist!
[14:30:07] zuglufttier leaves the room
[14:31:08] <zinid> you're damn right
[14:31:13] <zinid> so, what to do with binding failure?
[14:31:46] <MattJ> zinid, there should be no 'to' attribute on the stanza
[14:32:34] <MattJ> https://xmpp.org/rfcs/rfc6120.html#bind-servergen-success
[14:33:13] <zinid> really?
[14:33:45] <MattJ> Strange but true
[14:34:22] <zinid> where is it said there should no be "to"?
[14:34:24] <zinid> I don't see
[14:34:52] <zinid> okay, whatever, I'll fix it
[14:34:56] <zinid> but I disagree!!!
[14:35:31] <MattJ> It also doesn't say that there should be no foobar="i am a rubber ducky", but you don't add that either :)
[14:36:26] <MattJ> The text doesn't say one way or the other, but the example omits it and that's enough for me
[14:36:56] <zinid> 16:36:34.053 [critical] Benchmark failure: localhost doesn't support required feature: http://jabber.org/protocol/bytestreams
[14:37:08] <zinid> MattJ, not a problem, I will not go into bikeshedding
[14:37:22] <MattJ> "proxy65" module
[14:37:33] <MattJ> The benchmark uses that too?
[14:37:38] <Holger> I'm the man for bikeshedding.  Don't the generic stanza rules apply?  I.e. <https://xmpp.org/rfcs/rfc6120.html#stanzas-attributes-to-c2s>?
[14:37:39] <MattJ> This is quite a complex benchmark...
[14:37:43] <zinid> mod_proxy65
[14:37:46] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:37:58] <zinid> what ports and addresses does it use by default?
[14:38:15] mrDoctorWho leaves the room
[14:38:41] <MattJ> Holger, you mean the bit there that says "A stanza sent from a client to a server for direct processing by the server (e.g., roster processing as described in [XMPP‑IM] or presence sent to the server for broadcasting to other entities) MUST NOT possess a 'to' attribute."? :)
[14:38:53] <MattJ> zinid, port 5000 listens on all interfaces
[14:39:59] <Holger> MattJ: Yes that's what I had in mind.
[14:40:10] <Holger> I wasn't saying you're wrong :-)
[14:40:19] <Holger> But neither of you mentioned that section.
[14:40:42] <MattJ> I didn't want to distract zinid with reading while he's creating
[14:41:18] <Holger> :-)
[14:42:12] vogt leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:43:06] <Holger> Though the #1 rule seems to also match the stanzas matched by the #2 rule, no?  This always confused me.
[14:43:41] <Holger> I mean a "stanza sent from a client to a server for direct processing by the server" is also a "stanza with a specific intended recipient (e.g., [...] the server itself)" ...
[14:43:49] <zinid> finally, started
[14:44:04] <Holger> So the idea is #2 overrules #1 or something.  Or it's just my bad English.
[14:44:13] <zinid> I will not test http-upload
[14:44:41] ta leaves the room
[14:44:46] <MattJ> Holger, I think by "server itself" it means example.com
[14:45:02] <MattJ> It's confusing because "the server" can mean a number of things
[14:45:02] <zinid> {verify_cert, false},
     {sasl_mechanisms, ["PLAIN"]},
     {negotiation_timeout, 100},
     {connect_timeout, 100},
     {message_body_size, 100},
     {reconnect_interval, 100},
     {message_interval, 600},
     {presence_interval, 600},
     {disconnect_interval, 600},
     {proxy65_interval, 600},
     {proxy65_size, 10485760},
     {starttls, true},
     {csi, true},
     {sm, true},
     {mam, true},
     {carbons, true},
     {blocklist, true},
     {roster, true},
     {rosterver, true},
     {private, true}
[14:45:10] <zinid> those are the parameters
[14:45:12] <Holger> MattJ: Hm kay.
[14:45:15] <zinid> everyone agrees?
[14:45:22] <zinid> numbers are in seconds
[14:45:31] <zinid> or in bytes
[14:45:33] cippaciong leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:45:39] <MattJ> zinid, lgtm, whatever :)
[14:45:48] <zinid> lgbt
[14:45:51] zuglufttier leaves the room
[14:46:04] alexis leaves the room
[14:46:08] <zinid> 100k users
[14:46:16] <Holger> zinid: But this is without contacts/presence for now?
[14:46:27] <zinid> arrival rate = 1 connection per 1 *milli*second
[14:46:32] <zinid> Holger, right
[14:46:34] <zinid> Holger, so far
[14:47:01] <Holger> k
[14:48:03] <zinid> argh, that resource binding fix has borked something else
[14:48:11] <zinid> I assume every my stanza has from/to 🙁
[14:48:17] <zinid> stupid RFC
[14:49:32] <Holger> Maybe it's a trivial change in Prosody? :-)
[14:49:49] <MattJ> Hmm
[14:49:56] <zinid> damn, prosody doesn't set 'from'
[14:50:02] <zinid> why?
[14:50:03] <MattJ> Not trivial, no
[14:50:12] <zinid> this is fucking inconvenient
[14:50:20] <Holger> XMPP is totally awesome because it makes interop so easy.
[14:50:54] <MattJ> "when a client receives a stanza that does not include a 'from' attribute, it MUST assume that the stanza is from the user's account on the server. "
[14:51:02] <MattJ> If you follow the RFCs, sure
[14:51:37] <zinid> <iq id='13312783680088918017'
    type='error'
    to='tsung1561@localhost/xmppb'>
  <error type='wait'>
    <internal-server-error xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'/>
    <text xmlns='urn:ietf:params:xml:ns:xmpp-stanzas'>Error reading storage</text>
  </error>
</iq>
[14:51:41] <zinid> yeah
[14:51:59] <zinid> well that's good I test my library on prosody after all 🙂
[14:52:53] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:53:13] <zinid> and if I don't have to?
[14:53:50] <MattJ> Don't have to what?
[14:55:35] <Johan> I guess zinid meant if the stanza doesn't include a 'to' attribute ;-)
[14:56:12] <MattJ> Oh right
[14:56:28] <MattJ> I don't see that clients should ever look at the 'to' attribute
[14:56:35] <MattJ> Seems totally useless to me
[14:57:07] <MattJ> unless you think the server is so broken that it misdelivers messages
[14:57:42] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:59:14] ChaosKid42 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[14:59:17] jere leaves the room
[15:00:56] <prefiks> does it set to when it's bare jid or jid with resources or it always strip that?
[15:02:12] <MattJ> If it's a stanza sent by someone else, the 'to' is preserved. If it's a stanza from the client to the server/it's own account, it gets normalized to no-'to'
[15:02:15] newbie leaves the room
[15:02:18] <MattJ> which means the reply will also have no 'to'
[15:02:30] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:04:31] avb leaves the room
[15:05:58] secret_agent leaves the room: Stream closed by us: Replaced by new connection (conflict)
[15:08:10] bammes leaves the room: Rechner geht in den Ruhezustand
[15:08:18] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:09:01] marc leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:11:02] <zinid> something wrong with ulimit
[15:11:02] jonasw leaves the room
[15:11:28] <zinid> configuring /etc/security/limits.conf on AWS linux is apparently not enough
[15:12:50] <zinid> $ su --shell=/bin/sh prosody
sh-4.2$ ulimit -n
2048576
[15:12:52] <zinid> odd
[15:14:00] <Holger> zinid: How do you start Prosody?
[15:14:11] <zinid> Holger, systemd I guess
[15:14:16] <zinid> service prosody start
[15:14:30] <Holger> Then you'd set the limit in prosody.service.
[15:14:44] <Holger> /etc/security/limits.conf only affects PAM (i.e. shell logins).
[15:15:23] <Holger> LimitNOFILE=65536
[15:15:28] <Holger> ... or whatever.
[15:15:36] <Holger> Within the [Service] section.
[15:18:11] <Holger> zinid: If this is some Debian package, the location is probably /lib/systemd/system/prosody.service ...
[15:18:41] <zinid> the fuck, it doesn't help
[15:18:46] <zinid> should I restart systemd?
[15:18:55] <Holger> systemctl daemon-reload
[15:19:17] <Holger> Then restart Prosody.
[15:19:58] <zinid> doesn't help
[15:20:25] <zinid> # cat /usr/lib/systemd/system/prosody.service
[Unit]
Description=Prosody XMPP (Jabber) server
After=network-online.target
[Service]
Type=forking
LimitNOFILE=4248576
PIDFile=/run/prosody/prosody.pid
ExecStart=/usr/bin/prosodyctl start
ExecReload=/usr/bin/prosodyctl reload
ExecStop=/usr/bin/prosodyctl stop
[Install]
WantedBy=multi-user.target
[15:20:27] <Holger> You get resource limit errors somewhere?
[15:20:31] <zinid> no
[15:20:46] <newbie> mod_vcard option search_all_hosts got removed?
[15:21:01] newbie leaves the room: Stream closed by us: system-shutdown
[15:21:07] <Holger> zinid: So why do you assume it's an "uname -n" issue?
[15:21:28] <zinid> Holger, then what? prosody just got stuck without errors on 1024 connections?
[15:21:34] <Holger> Ah.
[15:21:38] erik leaves the room: Connection failed: timeout
[15:22:20] <zinid> Jul 03 15:10:09 datamanager     error   Failed to load accounts storage ('/var/lib/prosody/localhost/accounts/tsung2963.dat: Too many open files') for user: tsung2963@localhost
[15:22:20] <Holger> # ulimit -n 4248576
[15:22:21] <Holger> ulimit: setrlimit failed: operation not permitted
[15:22:24] <MattJ> Hmm, I don't know what the Fedora package pulls in, but it may be that it's not using libevent
[15:22:25] <Holger> zinid: The number is too large.
[15:22:26] <zinid> but that was before
[15:22:32] <MattJ> Oh, that's ulimit
[15:22:41] <zinid> Holger, yeah, logging is overrated 😉
[15:22:57] <Holger> zinid: I'm not sure what the threshold is.  Try "ulimit -n ..." as root.
[15:23:05] <zinid> # ulimit -n
4248576
[15:23:16] <Holger> "Oh."
[15:23:37] <Holger> Then I don't know.
[15:23:52] <zinid> just a second
[15:24:22] <Holger> systemctl show ejabberd | grep LimitNOFILE
[15:24:25] <Holger> Er.
[15:24:30] <Holger> systemctl show prosody | grep LimitNOFILE
[15:24:35] <zinid> fixed
[15:24:38] <Holger> Ah.
[15:24:43] <zinid> ah, no 🙂
[15:24:54] <Holger> What's the output of my command?
[15:25:13] <newbie> 2018-07-03 19:22:14.247 [error] <0.318.0>@gen_mod:module_error:571 Unknown option 'search_all_hosts' of module 'mod_vcard', available options are: matches, hosts, cache_missed, host, cache_life_time, cache_size, use_cache, db_type, name, search, allow_return_all
[15:25:24] <zinid> # systemctl show ejabberd | grep LimitNOFILE
LimitNOFILE=4096
[15:25:32] <newbie> if this got removed its still on the documentation
[15:25:33] <zinid> newbie, this is only for mnesia
[15:25:51] <newbie> zinid, ok thank you 🙂
[15:25:52] <Holger> zinid: 4096 is what you epected?
[15:25:59] <zinid> # systemctl show prosody | grep LimitNOFILE
LimitNOFILE=2048576
[15:26:05] <Holger> Ah ejabberd :-)
[15:26:11] <zinid> Holger, no, of course not, ejabberd is able to handle 2M at least
[15:26:11] <Holger> Hum.
[15:26:32] <zinid> but ejabberd doesn't use systemd
[15:26:41] <zinid> it's hand-crated retarded script
[15:26:43] <zinid> 🙂
[15:27:06] <Holger> systemctl stop prosody && systemctl start prosody
[15:27:10] <Holger> And it's still broken?
[15:27:10] <zinid> Holger, the strange thing is that prosody doesn't report that file error anymore
[15:27:48] <zinid> yes: http://52.22.32.97:8091/
[15:29:25] <Holger> No idea, sorry.
[15:29:59] <Holger> I mean short of properly debugging this.  Let a wrapper script log "uname -n" or whatever, then the same thing from within Prosody, get it to log proper errors :-/
[15:30:02] <MattJ> If you're still reaching a limit on the connections around 1024, it's probably because Prosody is using select()
[15:30:12] <Holger> Ah!
[15:30:13] <MattJ> if you see "too many open files" though, it's ulimit
[15:30:37] <Holger> So how to tell it to use something else?
[15:30:47] <Holger> Why not always use something else? :-)
[15:30:49] <zinid> Jul  3 15:30:01 ip-10-0-1-90 systemd: Created slice User Slice of root.
Jul  3 15:30:01 ip-10-0-1-90 systemd: Starting User Slice of root.
Jul  3 15:30:01 ip-10-0-1-90 systemd: Started Session 23 of user root.
Jul  3 15:30:01 ip-10-0-1-90 systemd: Starting Session 23 of user root.
Jul  3 15:30:01 ip-10-0-1-90 systemd: Removed slice User Slice of root.
Jul  3 15:30:01 ip-10-0-1-90 systemd: Stopping User Slice of root.
Jul  3 15:30:13 ip-10-0-1-90 systemd: Stopping Prosody XMPP (Jabber) server...
Jul  3 15:30:13 ip-10-0-1-90 prosodyctl: Stopped
Jul  3 15:30:13 ip-10-0-1-90 systemd: Stopped Prosody XMPP (Jabber) server.
Jul  3 15:30:13 ip-10-0-1-90 systemd: Starting Prosody XMPP (Jabber) server...
Jul  3 15:30:14 ip-10-0-1-90 prosodyctl: Started
Jul  3 15:30:14 ip-10-0-1-90 systemd: Started Prosody XMPP (Jabber) server.
[15:30:54] <zinid> nothing in the log
[15:31:04] <zinid> MattJ, there is absolutely nothing in the log
[15:31:13] mightyBroccoli leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:31:22] <MattJ> Holger, extra dependencies, unnecessary on 90% of deployments
[15:31:57] <zinid> select()?
[15:31:59] <Holger> https://prosody.im/doc/libevent
[15:32:03] <zinid> so how will we benchmark?
[15:32:08] <Holger> ^ ... is the solution?
[15:32:29] <zinid> Holger, ok
[15:32:39] <Holger> "apt install lua-event" and set "use_libevent = true"?
[15:32:50] <Holger> zinid: No idea, this was meant as a question for MattJ :-)
[15:32:59] <Holger> This was just the first Google hit.
[15:33:21] <zinid> Jul 03 15:33:00 general error   libevent not found, falling back to select()
[15:33:25] <zinid> yeah, need to install that
[15:34:09] <zinid> No package lua-libevent available.
No package lua-event available.
[15:34:14] <zinid> easy to install, my ass
[15:34:19] <Holger> What distro?
[15:34:24] <zinid> I spent already how much? 2 hours?
[15:34:33] <zinid> Holger, AWS Linux Crap Edition
[15:34:43] <Holger> "Oh."
[15:36:04] <zinid> # yum search lua | grep event
nothing as well
[15:36:07] <Holger> MattJ: Is it easy to install manually?
[15:36:59] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[15:37:19] <zinid> http://rpms.famillecollet.com/rpmphp/zoom.php?rpm=lua-event
[15:37:30] <zinid> I bet we can use some magic to install from there
[15:38:06] <MattJ> Easiest is to use luarocks if that is packaged
[15:38:32] <zinid> ok, installed
[15:39:00] <zinid> next what? it still doesn't see libevent
[15:39:23] <MattJ> Near the top of the config is a use_libevent option, change to true
[15:39:28] <zinid> I did
[15:39:36] <zinid> Jul 03 15:38:46 general error   libevent not found, falling back to select()
[15:39:40] <MattJ> You installed the lua-event package or luarocks?
[15:39:42] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:40:15] <zinid> luarocks
[15:40:52] <MattJ> as root: luarocks install luaevent
[15:40:59] <MattJ> and restart Prosody
[15:41:12] <zinid> Using https://luarocks.org/luaevent-0.4.4-1.src.rock... switching to 'build' mode
Error: Could not find header file for EVENT
  No file event.h in /usr/local/include
  No file event.h in /usr/include
You may have to install EVENT in your system and/or pass EVENT_DIR or EVENT_INCDIR to the luarocks command.
Example: luarocks install luaevent EVENT_DIR=/usr/local
[15:41:18] <zinid> ah
[15:41:22] <zinid> no devel files
[15:42:01] <zinid> just a second, gcc takes some time 😉
[15:42:58] <zinid> finally
[15:43:02] <zinid> no error in the log
[15:43:35] <Holger> Yay.
[15:44:43] <Holger> zinid: Did you enable MAM in the Prosody config?  Or is it enabled by default?
[15:44:48] <zinid> Holger, yes
[15:45:16] <zinid> http://52.22.32.97:8091/
[15:45:36] <Holger> Hooray.
[15:46:03] ChaosKid42 leaves the room: Stream closed by us: Replaced by new connection (conflict)
[15:46:20] Alacer leaves the room
[15:48:58] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:49:04] <zinid> stopped
[15:49:12] <zinid> will restart in a second
[15:49:16] <zinid> I forgot to disable debug
[15:49:36] mightyBroccoli leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:51:28] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:52:41] badlop leaves the room: Stream reset by peer
[15:52:56] <zinid> 20692 prosody   20   0 2166980   2.0g   4516 R 100.0  3.2   2:22.71 lua
[15:53:23] jere leaves the room
[15:55:21] <zinid> errors so far:
Stream closed by us: Stream negotiation timeout (connection-timeout): 2
Connection failed: timeout: 250
[15:56:37] <zinid> Resumption failed: item-not-found: 26
Connection failed: connection closed: 18
Stream closed by us: Stream negotiation timeout (connection-timeout): 11
Connection failed: timeout: 1445
[15:56:53] deavmi leaves the room: Disconnected: Replaced by new connection
[15:58:56] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[15:58:59] <zinid> should I wait more?
[15:59:22] <zinid> Resumption failed: item-not-found: 318
Connection failed: connection closed: 630
Stream closed by us: Stream negotiation timeout (connection-timeout): 92
Connection failed: timeout: 11048
[15:59:35] <MattJ> It's up to you, whatever is standard practice for you
[15:59:48] <zinid> # telnet 10.0.1.90 5222
Trying 10.0.1.90...
[15:59:53] <zinid> finita la comedia 🙂
[16:00:05] marc leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:00:27] <zinid> so that's it, 50k?
[16:00:39] <zinid> with thousands of errors/
[16:00:39] <zinid> ?
[16:00:45] <zinid> let's bench ejabberd?
[16:01:02] <MattJ> Go for it
[16:01:30] <zinid> 6Gb
[16:04:59] <Holger> zinid: 6Gb was Prosody's memory consumption for 50k users?
[16:05:06] <zinid> Holger, yes
[16:05:12] <zinid> a bit more in fact
[16:05:23] <zinid> 60k maybe, just some sessions are resumed
[16:05:44] <zinid> well ejabberd already consumes more
[16:05:58] <zinid> 7.5Gb
[16:06:42] <zinid> 9Gb on where prosody stale
[16:06:50] <zinid> 45k that is
[16:09:09] ata2001 leaves the room: Disconnected: closed
[16:10:26] ata2001 leaves the room: Disconnected: closed
[16:10:31] mightyBroccoli leaves the room
[16:10:55] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:11:08] <zinid> okay, everything is connected
[16:11:08] ata2001 leaves the room: Disconnected: closed
[16:11:15] <zinid> one error:
Connection failed: timeout: 1
[16:11:26] <zinid> 16.3gb
[16:12:04] jonasw leaves the room
[16:12:04] <MattJ> Memory usage you quote is RES in top?
[16:12:28] <zinid> RSS, yes
[16:12:34] <zinid> *RES
[16:13:47] <zinid> 21928 ejabberd  20   0   14.7g   8.8g   9648 S 412.0 14.4  67:44.92 beam.smp
[16:13:51] <zinid> shrinked
[16:13:54] <zinid> seems like GC
[16:13:58] <zinid> 9Gb
[16:14:24] ChaosKid42 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:15:00] <zinid> well 🙂
[16:15:21] <zinid> errors so far:
Resumption failed: Previous session timed out (item-not-found): 4
Connection failed: timeout: 1
[16:15:51] <Holger> Is it obvious where the bottleneck was for Prosody to be unable to handle more connections?
[16:16:04] <zinid> sure, single core
[16:16:29] <Holger> So CPU at 100%?
[16:16:31] <zinid> it should be trivial to understand, no? 🙂
[16:16:34] <MattJ> How does erlang/ejabberd handle that? Is each connection automatically a process?
[16:16:41] <zinid> Holger, yes, from the very beginning actually
[16:16:49] <zinid> MattJ, yes
[16:17:09] <zinid> an "Erlang process"
[16:17:10] <MattJ> So the accept() loop is in beam itself?
[16:17:17] <zinid> yes
[16:17:29] <zinid> accept() -> start process() -> repeat()
[16:17:58] <Holger> Well it's in another Erlang process.
[16:18:06] <MattJ> Ok, so it's in "user" code?
[16:18:16] <Holger> Yes.
[16:18:53] 4223 leaves the room
[16:19:39] <Holger> One Erlang process per port (i.e. per accept() loop).
[16:19:47] 404 leaves the room
[16:19:59] <zinid> ah, and btw, ejabberd's users and mam is in SQL
[16:20:10] <zinid> and SQL is quite expensive
[16:21:24] newbie leaves the room: Stream closed by us: system-shutdown
[16:21:24] newbie leaves the room: Stream closed by us: system-shutdown
[16:22:05] <zinid> so, is it legit to say that 60k is the maximum for prosody?
[16:22:09] <zinid> without rosters 🙂
[16:22:31] <zinid> processor       : 15
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Platinum 8175M CPU @ 2.50GHz
stepping        : 4
microcode       : 0x2000043
cpu MHz         : 3099.975
cache size      : 33792 KB
[16:22:42] Alacer leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:23:14] <zinid> I can try to decrease arrival rate
[16:24:28] <zinid> Link Mauve, you see, the limiting factor is not memory
[16:24:48] <zinid> single thread sucks
[16:24:52] lukas leaves the room
[16:27:16] <Holger> zinid: If the CPU was at 100% right from the beginning this does indeed suggest Prosody was overloaded with the arrival rate I guess?
[16:27:37] <zinid> Holger, well, I set 10ms
[16:27:48] <zinid> I mean it used to be 1ms 🙂
[16:27:56] <zinid> let's check
[16:28:13] <Holger> Then again the real world arrival rate might not be much better on server restart. Precisely the situation I mentioned above 🙂
[16:28:18] <zinid> 37% now
[16:28:22] <zinid> at the beginning
[16:29:17] 404 leaves the room
[16:30:17] <zinid> Holger, yeah, I rant about this all the time: xmpp should tell a client to reconnect in X seconds when overloaded
[16:30:42] <zinid> but nobody cares, seems like highload is not interested in the community
[16:31:27] <Holger> I would hope sane clients do that anyway.
[16:31:44] <Holger> Without spitting out frightening messages.
[16:31:52] <zinid> I know only tkabber is using exponential backoff
[16:32:08] <zuglufttier> Well, if you have five users, you're just happy if some of them come back online some time.
[16:32:09] <nosnilmot> pidgin does too
[16:32:33] <Holger> zuglufttier: 😂
[16:32:55] <zinid> zuglufttier, and if you have none...
[16:33:19] <zinid> Holger, 54%
[16:34:06] 4223 leaves the room
[16:34:07] <Holger> Did I get it right the last run was with 1ms and now 10ms?
[16:34:15] <zinid> Holger, yes
[16:34:19] Andrew Nenakhov leaves the room
[16:34:30] nekit leaves the room
[16:34:54] <Holger> k, how many sessions now (with 54%)?
[16:35:05] <zinid> but memory is good on prosody, I admin, around 1Gb per 10k
[16:35:09] <zinid> (without rosters, hehe)
[16:35:29] <Holger> Yeah well that's always been better.
[16:35:32] <zinid> Holger, http://52.22.32.97:8091/
[16:35:47] <Holger> Unless you run jabber.fr 🙂
[16:35:53] <zinid> 60%
[16:36:21] <zinid> reaching 70%
[16:36:29] <zinid> damn, I need to draw graphs for this
[16:36:54] <zinid> it's on my todo to improve output of the benchmarking tool stats
[16:38:10] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:38:10] <zinid> 80%
[16:38:37] <zinid> but listener is quite fast, no lags
[16:38:46] <zinid> well, it will start lagging at 100% cpu
[16:38:46] Andrew Nenakhov leaves the room
[16:39:56] <zinid> seems like noone is interested 🙂
[16:40:03] <zinid> when it comes to real benchmark
[16:40:07] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:40:14] <zinid> eveyone just want to rant in theory
[16:40:34] <MattJ> Evidence?
[16:40:39] <404> I'm confirming the problem with server maintenance on Prosody. I have a server Prosody crashing regularly
[16:40:39] <zinid> Holger, 100%
[16:40:54] <MattJ> 404, reported?
[16:41:17] <ThUnD3r|Gr33n®> Thought this would be ejabberd muc 🤔
[16:41:36] <zinid> > 404, reported?
when it comes to prosody: "reported?" when it comes to ejabberd: I just changed it!
[16:42:06] <MattJ> zinid, you fix things that you don't know are broken? Well done :)
[16:42:54] <zinid> prosody again is dying at the same point
[16:43:03] <zinid> so yeah, 60k is a theoretical limit
[16:43:12] <MattJ> 100% CPU?
[16:43:13] <404> MattJ,  I changed the server to Ejabberd.
[16:43:44] <404> MattJ,   I changed the server to Ejabberd.  
[16:43:46] <zinid> MattJ, yes
[16:44:06] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:44:14] lukas leaves the room
[16:44:37] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:44:37] <Holger> The graph stopped at about 42k no?
[16:44:39] <MattJ> 404, ok, never mind :)
[16:44:47] <zinid> (xmppb@localhost)3> xmppb_sm:size().
41219
(xmppb@localhost)4> xmppb_sm:size().
41180
(xmppb@localhost)5> xmppb_sm:size().
41218
(xmppb@localhost)6> xmppb_sm:size().
41134
(xmppb@localhost)7> xmppb_sm:size().
41747
[16:45:03] <zinid> Holger, well, it sucks a bit, it only counts active session, not those in resume
[16:45:13] <zinid> MattJ, how to get the stats about sessions?
[16:45:15] <Holger> Ah.
[16:45:44] <MattJ> zinid, c2s:count() will include non-resumed sessions I think
[16:45:49] vanitasvitae leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:45:51] <MattJ> erm, requires telnet console though
[16:46:13] <zinid> MattJ, I need all sessions, including in "detached" state (waiting for resume)
[16:46:15] <404> I've heard of the Prosody server with 80,000 users online, but they pay a lot of money for the cloud
[16:46:34] <404> CryptoCat
[16:46:55] <zinid> 404, how did they do that?
[16:47:10] <zinid> 404, we barely can hit 50k
[16:47:27] <MattJ> zinid, in this specific benchmark
[16:47:29] <zinid> maybe I'm sending stanzas too aggressively
[16:47:40] <zinid> MattJ, we can tune the parameters 🙂
[16:47:42] <MattJ> Benchmarks are only benchmarks
[16:47:46] <404> zinid, In any case, this does not compare with the millions online in Ejabberd
[16:47:54] <zinid> we need Holger to collect the statistics from conversations.im 😉
[16:48:26] <Licaon_Kter> 404: howmany users/ online do you have? Why did you switch?
[16:49:16] vanitasvitae leaves the room
[16:49:18] <zinid> MattJ, for the record, this specific benchmark doesn't even have rosters and rosterversions, and massive presence broadcasts (due to rosters) 🙂
[16:51:05] <Holger> zinid: You mean usage statistics such as the (re)connection rate, number of messages/presence per second and so on?
[16:51:11] zuglufttier leaves the room
[16:51:24] <zinid> Holger, yes, it's just a bit pain to collect 😕
[16:51:33] <404> EVE Online ~ 1 000 000 online user  Ejabberd
[16:51:59] <zinid> 24534 prosody   20   0 8185132   7.6g   4568 R 100.0 12.4  19:12.59 lua
[16:52:05] <zinid> the memory is growing
[16:52:13] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[16:52:19] vanitasvitae leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:53:16] <zinid> 7.7Gb
[16:53:23] <Licaon_Kter> zinid: you need a memusage and/or cpuusage line on every graph
[16:53:25] <zinid> funny, I will let it running so far
[16:53:32] <zinid> Licaon_Kter, I know
[16:53:51] <zinid> Licaon_Kter, I have all this stats, I'm just lazy to read gnuplot docs
[16:54:08] <zinid> I have connection/stanza/etc delays
[16:54:09] Andrew Nenakhov leaves the room
[16:54:18] <zinid> built in a histogram
[16:54:40] vanitasvitae leaves the room
[16:54:41] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:55:53] <404> Licaon_Kter,  before it was 250 online user. It simply could not become more, because prosody were falling and dispersing users. Current 700 online
[16:56:02] <MattJ> what
[16:56:13] nekit leaves the room
[16:56:15] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:56:26] <MattJ> Never mind, if you switched I don't really want to know :)
[16:56:45] <zinid> 7.8g
[16:56:47] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[16:57:20] <zinid> Resumption failed: item-not-found: 5316
Connection failed: connection closed: 3455
Stream closed by us: Stream negotiation timeout (connection-timeout): 5692
Connection failed: timeout: 95459
[16:57:39] <zinid> yeah
[16:57:50] <zinid> 36k connected
[16:57:51] <Licaon_Kter> 404: not running on AWS with Intel(R) Xeon(R) Platinum 8175M CPUs? You could have scaled up to 45000 users...🙅
[16:57:55] Andrew Nenakhov leaves the room
[16:58:15] <zinid> http://52.22.32.97:8091/c2s-sessions.png
[16:58:29] Andrew Nenakhov leaves the room: Connection failed: connection closed
[16:58:42] <MattJ> zinid, nice
[16:58:51] <zinid> http://upload.zinid.ru/99bb0647172ef83f67aa892f2cd0f2fa9d04f206/irrpwNqJX5GSq1f5qGIGm3Ox9z8zbmiFqKzdKaWB/c2s-sessions.png
[16:59:21] <404> Licaon_Kter,  45000 online user It is not enough  😀
[16:59:27] <Licaon_Kter> zinid: you're connecting locally?
[16:59:33] <zinid> Licaon_Kter, yes
[16:59:51] <zinid> Licaon_Kter, the problem is:
# telnet 10.0.1.90 5222
Trying 10.0.1.90...
[16:59:53] Andrew Nenakhov leaves the room
[17:00:00] <MattJ> zinid, would you care to share an AMI or something of this setup?
[17:00:02] <Licaon_Kter> You're bypass some stuff...bandwidth, latency, tcpstack...
[17:00:23] Andrew Nenakhov leaves the room: Connection failed: connection closed
[17:00:28] <zinid> MattJ, I don't know what's that, it's configured by our sysadmin 🙂
[17:01:14] <zinid> Licaon_Kter, well, that's the point, I don't want network to be the bottle-neck
[17:01:20] <MattJ> zinid, machine image. If it's not a clean server then probably not :)
[17:01:24] <zinid> Licaon_Kter, otherwise it's hard to understand where the problem is
[17:02:05] Holger leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:02:13] <Licaon_Kter> zinid: true
[17:02:27] <Licaon_Kter> MattJ: aren't Prosody devs doing these benches too?
[17:03:02] <MattJ> Licaon_Kter, yes, we have our own tools, we don't use tsung
[17:03:15] <MattJ> and we don't bench ejabberd, because I never manage to get it running properly
[17:03:16] <zinid> MattJ, I don't use tsung neither
[17:03:18] <zinid> not anymore
[17:04:01] <MattJ> Licaon_Kter, and for our purposes we're more interested in catching regressions than comparing to other servers in specific benchmarks
[17:04:29] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:04:36] <MattJ> We just had a server donated, on which we'll be setting up some more permanent stuff soon
[17:05:06] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[17:05:09] <zinid> MattJ, actually my idea is to use "golden" benchmark, so we can compare servers
[17:05:30] <zinid> if we collect enough real-data statistics it's not a problem to tune it
[17:05:32] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:05:32] <MattJ> zinid, the problem is that benchmarks are always going to be artificial
[17:05:55] <zinid> sure, but we still can create some "golden" one
[17:06:04] <zinid> otherwise we cannot compare stuff
[17:06:08] <MattJ> and run it on what hardware?
[17:06:21] <Licaon_Kter> MattJ: that was not my question, I meant, are these numbers lower than Prosodys testing?
[17:06:28] <zinid> m4.x4large 🙂
[17:06:58] <MattJ> Licaon_Kter, we've achieved higher... but that's not the point (and is why we don't publish what testing we do)
[17:07:10] <MattJ> It depends so heavily on config and traffic patterns
[17:07:32] Andrew Nenakhov leaves the room
[17:07:39] <zinid> sure, if I disable evertything, even TLS then I can connect 2M to ejabberd on the same node I'm testing now 😀
[17:07:51] <Licaon_Kter> MattJ: ok
[17:08:09] <MattJ> zinid, without TLS is indeed a valid benchmark, it removes more of the layers that create noise
[17:08:18] vanitasvitae leaves the room
[17:08:20] <MattJ> Does ejabberd still use OpenSSL?
[17:08:32] <zinid> MattJ, but it's very synthetic without TLS, no real usage
[17:08:42] <zinid> MattJ, yes
[17:09:02] <MattJ> Well that's a memory and a CPU hog for sure
[17:09:13] Andrew Nenakhov leaves the room
[17:09:34] <zinid> meanwhile, seems like Prosody stopped growing memory at 7.9g, so no leak seems like
[17:09:37] <MattJ> I'm not saying it shouldn't be tested, I'm just saying that testing without is still a useful test
[17:10:02] <MattJ> zinid, now install 500 unreviewed community modules, see a leak and report it to our bug tracker ;)
[17:10:03] <zinid> MattJ, well, yes, because you can use TLS offload for example
[17:10:14] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:10:17] <404> If you use a home server or few people, there is no difference between Prosody and Ejabberd
[17:10:35] <zinid> 404, prosody consumes less memory on start
[17:10:50] <zinid> right now it consumes 25Mb on start on the machine
[17:11:01] <zinid> ejabberd 60Mb
[17:11:19] <zinid> not sure if it's valid anymore, since even the low-priced VPs is now 1GB 🙂
[17:11:30] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:11:47] vanitasvitae leaves the room
[17:12:34] <zinid> and the majority of memory in ejabberd at start is consumed by the bytecode, this is fucking terrible
[17:12:54] <MattJ> anyway, thanks for all this zinid... hope you found it as interesting as I did :)
[17:12:58] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:13:09] <zinid> I need rosters!
[17:13:11] <zinid> 😀
[17:13:25] <MattJ> Well, that's why I wondered if you could share the setup somehow, so it can be easily replicated
[17:13:33] <MattJ> and I guess this becomes your golden benchmark :)
[17:14:48] Andrew Nenakhov leaves the room
[17:15:01] <zinid> okay, I probably need to test Openfire now
[17:15:05] <zinid> where is Guus? 😀
[17:15:07] <MattJ> I'd be willing to work on an ansible role or something that would automate setup of Prosody, if you could add setup of ejabberd and the benchmark tool
[17:15:29] <zinid> MattJ, what exactly you need?
[17:15:37] <zinid> I will probably publish my tool soon
[17:15:45] <MattJ> Let me know when you do, then
[17:15:50] <zinid> okay
[17:15:54] <MattJ> and maybe we can set something up
[17:16:19] <404> MattJ, Prosody It's interesting, but I'd like to have more people than sit on one core
[17:16:48] <MattJ> 404, if you dream of millions of users only, I would have told you to use ejabberd from the start
[17:17:17] <MattJ> But you had... what did you say? 250? and Prosody crashing? That's just wrong and broken, and I would have helped you solve the problem if you wanted that
[17:17:20] <zinid> ok, I stopped the machines
[17:20:47] <MattJ> Thanks! I look forward to you publishing your benchmark tool, especially if it gains CPU and RAM graphs
[17:21:00] <MattJ> and then we can work on automating the testing
[17:21:28] <zinid> 👍
[17:21:47] <404> MattJ, I did not know what to report, because there were no errors in the logs. I studied this dialogue. I think my prosody fell because the hoster set a limit on the use of the kernel.
[17:22:47] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:22:48] adrien leaves the room
[17:23:10] <MattJ> 404, and ejabberd works fine with the same number of users on the same host?
[17:23:19] rom1dep leaves the room: Machine going to sleep
[17:26:26] <404> MattJ, ejabberd faling does not leave work ejabberd
[17:27:01] <MattJ> Hmm?
[17:28:38] vanitasvitae leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:29:06] vanitasvitae leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:30:27] <404> MattJ, I've seen similar to Ejabberd, it did not stoped, it just restarted
[17:31:14] <MattJ> Ok, so your issue was not a Prosody issue
[17:32:00] <MattJ> and you could have probably got the same behaviour as ejabberd (restart) by adding/changing one line in the systemd file
[17:33:01] <MattJ> I am happy if you are happy with ejabberd, I just wanted to be clear about the problem you had
[17:33:09] <zinid> That's how users make their opinion on software 😁
[17:34:05] <404> MattJ, Yes, this is not a problem for prosody, since it was probably a baseness from the hoster VPS
[17:36:15] <404> limit on the processor time, the CPU load was small
[17:36:25] Holger leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:44:12] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:47:26] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:48:58] rom1dep leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[17:50:28] tester leaves the room
[17:50:59] Neustradamus leaves the room
[17:55:39] prefiks leaves the room
[18:00:37] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:03:41] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:06:52] ChaosKid42 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[18:10:07] Alacer leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:18:05] 4223 leaves the room
[18:18:48] Andrew Nenakhov leaves the room: Connection failed: connection closed
[18:18:55] Alacer leaves the room
[18:19:22] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:19:54] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:20:52] mightyBroccoli leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:20:52] lukas leaves the room
[18:23:17] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:25:03] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:25:24] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:36:06] 4223 leaves the room
[18:41:17] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[18:41:27] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:43:47] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:44:38] ata2001 leaves the room: Disconnected: Replaced by new connection
[18:48:41] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[18:53:16] tester leaves the room
[18:53:19] <syever> Holger: hi, do you know a guide for migrate database MySQL ejabberd 17.07 to 18.06 , sorry may be this is not the group ...
[18:54:12] <mimi89999> Oh, I didn't expect to see MattJ here 😀
[18:54:25] mightyBroccoli leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[18:54:55] <MattJ> mimi89999, I used to be a regular here (well, I guess the room moved)
[18:55:20] ingolf leaves the room
[18:55:21] <mimi89999> Oh
[18:57:31] <MattJ> You know, I was an ejabberd user at one point too :) (and was one of the admins of jabber.org when it ran ejabberd)
[18:57:46] <MattJ> and have worked with multiple other places that used it too
[18:59:19] tester leaves the room
[19:00:01] tester leaves the room
[19:06:20] syever leaves the room: Stream closed by us: Replaced by new connection (conflict)
[19:09:52] <Licaon_Kter> syever: see https://docs.ejabberd.im/admin/upgrade/from_18.01_to_18.03/ maybe
[19:09:54] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:13:02] secret_agent leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:14:27] <zinid> Maybe just diff between mysql.sql revisions?
[19:15:57] sindrake leaves the room
[19:17:16] <syever> Licaon_Kter: thanks
[19:19:18] lukas leaves the room
[19:19:46] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:23:16] frainz leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:26:23] rom1dep leaves the room
[19:32:19] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:36:48] Licaon_Kter leaves the room: Stream closed by us: Replaced by new connection (conflict)
[19:39:49] Akasch leaves the room: Disconnected: Replaced by new connection
[19:44:53] erik leaves the room: Connection failed: timeout
[19:46:21] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:49:50] Alacer leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:54:30] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:56:21] <zinid> syever, https://gist.github.com/zinid/41109c2fde7eb21ebf24242af5b70124
[19:56:33] <zinid> the diff is not that complex
[19:56:52] mimi89999 leaves the room
[19:57:24] <zinid> most of the diff is add table with indexes and delete irc_custom
[19:57:29] <zinid> then alter on two tables
[19:57:51] VC leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[19:58:18] Man_Life leaves the room: Logged out
[19:58:42] mightyBroccoli leaves the room: Rechner geht in den Ruhezustand
[20:01:38] pod leaves the room: Stream closed by us: Replaced by new connection (conflict)
[20:01:52] xinit leaves the room
[20:02:15] pod leaves the room
[20:02:20] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[20:03:31] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:04:18] Akasch leaves the room
[20:05:15] <zinid> CertStorageError: expected /etc/letsencrypt/live/zinid.ru-0001/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/zinid.ru-0001.conf is broken. Skipping.
[20:05:21] <zinid> 3 months is good they said
[20:05:29] <zinid> you just need to automate they said
[20:05:42] <zinid> gosh 🙁
[20:06:22] <zinid> I need to finally fix RSA in ejabberd ACME
[20:06:35] <zinid> fuck this python shit
[20:06:46] debalance leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:07:38] pod leaves the room
[20:10:38] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[20:10:48] <syever> zinid: thanks
[20:12:19] VC leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:14:45] xinit leaves the room
[20:19:58] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:22:41] marc leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:23:02] VC leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:30:54] lukas leaves the room: Stream closed by us: Replaced by new connection (conflict)
[20:31:29] lukas leaves the room
[20:32:20] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:34:34] <Link Mauve> “18:31:53 zinid> I know only tkabber is using exponential backoff”, poezio too.
[20:35:00] <Link Mauve> “18:24:48 zinid> single thread sucks”, I know, that’s partly why I’m interested in that new haproxy thing for legacy SSL.
[20:35:05] <zinid> Licaon_Kter, okay
[20:35:20] <zinid> damn
[20:35:24] <zinid> Licaon_Kter, sorry dude!
[20:35:43] <Link Mauve> Are your benchmark graphs still available somewhere?
[20:35:50] <zinid> no
[20:36:38] newbie leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:36:43] <Link Mauve> “18:35:47 Holger> Unless you run jabber.fr 🙂”, we have more than enough memory, with 8 GiB available I don’t care much if we use 1.2 GiB to serve 4k connections.
[20:37:33] <zinid> Link Mauve, you're too late to the party, seems like nobody cares already 😉
[20:40:49] syever leaves the room: Stream closed by us: Replaced by new connection (conflict)
[20:41:17] ata2001 leaves the room: Disconnected: closed
[20:41:30] <zinid> http://upload.zinid.ru/99bb0647172ef83f67aa892f2cd0f2fa9d04f206/dhfVC1kQqDnFYKeIvFi5LNTxhuPA9HFrWC78fY5R/c2s-sessions.png
[20:41:40] <zinid> Link Mauve, here is what left
[20:44:11] mightyBroccoli leaves the room: Rechner geht in den Ruhezustand
[20:44:54] sindrake leaves the room: Disconnected: Replaced by new connection
[20:45:27] <Link Mauve> Damn, I shouldn’t have watched this anime with my comrades then!
[20:45:37] <zinid> > I’m interested in that new haproxy thing for legacy SSL
That's nice scenario to benchmark, I have haproxy already, I can offload TLS to haproxy and benchark haproxy+prosody
[20:45:40] <Link Mauve> And instead I should spend my entire life on Jabber, as usual!
[20:46:26] <Link Mauve> zinid, I have done exactly no setup for that yet, so I can’t assist on it, but from my understanding it allows to use multiple cores for TLS handshakes, which AIUI is your benchmark’s bottleneck.
[20:47:26] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[20:47:31] <zinid> Link Mauve, yes, haprxoy is able to do that, it has two modes: multiple processes or multiple threads, the latter is experimental and haproxy crashed in our benchmarks 🙂
[20:47:39] <Link Mauve> :D
[20:47:52] <zinid> but multiple processes worked flawlessly
[20:47:59] <Link Mauve> https://modules.prosody.im/mod_net_proxy.html seems to be the module.
[20:49:08] <Link Mauve> Uh, the Limitations part seems to say that won’t work for the usecase I have in mind?
[20:49:46] <zinid> not sure why you even need that
[20:50:03] <zinid> just configure on plain TCP port and let haproxy do the TLS stuff
[20:50:33] <zinid> however, say goodbye to SASL EXTERNAL (read: s2s won't work this way)
[20:50:39] <Link Mauve> Ah yes, maybe.
[20:50:49] <Link Mauve> That benchmark is only about c2s anyway.
[20:51:24] sindrake leaves the room
[20:51:29] <zinid> I can benchmark s2s, but what's the point? There are like 15k servers *globally*, it's nothing
[20:52:01] <zinid> it's unlikely you will ever connect to more than 5k
[20:54:41] <Link Mauve> We have 3k currently, and we’re not a big server.
[20:54:46] rom1dep leaves the room
[20:54:53] <zinid> jabber.ru has around 5k
[20:55:08] <zinid> used to have, dunno about now
[20:56:19] jeremy leaves the room: Machine going to sleep
[20:56:20] pod leaves the room
[20:57:29] deavmi leaves the room: Disconnected: Replaced by new connection
[20:59:20] <zinid> Link Mauve, or maybe just use ejabberd 😛
[21:00:16] <Link Mauve> I wouldn’t be opposed to migrating, but it would take a lot of time which I’d rather pour in other things.
[21:00:36] <zinid> I can help
[21:00:41] <Link Mauve> Especially rewriting most custom modules.
[21:00:49] <zinid> what modules?
[21:00:58] <zinid> are they public/private?
[21:01:14] <Link Mauve> First one to come to mind is the one generating this table: https://chat.jabberfr.org/
[21:01:32] <Link Mauve> Also mod_firewall.
[21:01:46] <zinid> ah, no mod_firewall
[21:02:21] <zinid> but that table is of course trivial to generate
[21:03:13] <Link Mauve> Also a DNSBL module which mathieui wrote to flag Tor accounts as very likely spammers, and the rest is trivial or already pushed to prosody-modules.
[21:03:30] srgcdev leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:03:39] <Link Mauve> Maybe the statistics modules, the stuff we have here: https://stats.jabberfr.org/d/000000002/node-exporter-stats?orgId=1&from=now-7d&to=now&refresh=30s
[21:03:59] <zinid> I have mod_dnsbl running on jabber.ru
[21:04:10] fp leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:04:36] <zinid> hum, that stats is generated by prosody, lol? 🙂
[21:05:07] Alacer leaves the room
[21:05:20] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:05:49] <Link Mauve> Why lol?
[21:06:31] <zinid> ah, there are presences and stuff like that
[21:06:36] <zinid> I just saw CPU/memory
[21:06:59] <Link Mauve> Even memory should have given Prosody away, with the lua field. :p
[21:07:15] <zinid> why so much?
[21:07:19] <zinid> what consumes it?
[21:07:32] <zinid> MattJ, told you have runtime introspection tools 😉
[21:07:53] <Link Mauve> Not too sure, probably fragmentation due to the gc and OpenSSL stuff.
[21:08:23] <Link Mauve> There is another module which gives way more information from malloc, but I don’t have it enabled.
[21:08:44] <zinid> ah, and Lua only sees around 477Mb, hehe
[21:09:08] <zinid> well, another part can be openssl for sure
[21:09:21] <zinid> I recall at some customer openssl fragmented 10Gb
[21:09:40] <zinid> terrible library in fact, we have so many hacks for it in our wrapper
[21:09:51] tester leaves the room
[21:11:10] <Link Mauve> I’ve been looking at the Rust TLS ecosystem, but haven’t played with plugging them in Lua yet.
[21:11:35] <Link Mauve> They are way more modular than OpenSSL, so it may be a good match for a server.
[21:12:08] <zinid> oh, I looked at a lot of them, but probably I will eventually migrate to the Erlang's native one: the network protocol is written in Erlang, and only stuff like ciphers computation is a wrapper in C
[21:12:37] <zinid> but currently it has drawbacks and doesn't scale 😕
[21:13:31] <Link Mauve> :(
[21:13:48] <Link Mauve> zinid, have you written anything about your survey?
[21:14:03] <zinid> my survey?
[21:14:23] lorddavidiii leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:14:57] <Licaon_Kter> Server perf bench one...🤣
[21:15:04] <Link Mauve> “23:12:08 zinid> oh, I looked at a lot of them, […]”
[21:15:15] <zinid> Link Mauve, ah, no
[21:15:22] <Link Mauve> Ok.
[21:15:26] <zinid> I don't write any articles 😉
[21:15:34] tester leaves the room
[21:15:40] <Link Mauve> Even just notes would be fine.
[21:15:45] edhelas leaves the room
[21:17:00] <zinid> it was a long time ago, I barely can remember why I rejected that one or another one, and also this might not hold true anymore today
[21:17:05] <zinid> so...
[21:17:12] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:17:38] <Link Mauve> Ok.
[21:19:11] <zinid> the problem with C wrappers in Erlang is that they may lock the VM, thus confusing the scheduler and degrading preemptive performance
[21:19:33] <zinid> even several microseconds "lock" may confuse the scheduler
[21:19:48] <zinid> and openssl sometimes computes something for a millisecond!
[21:20:23] <zinid> in practice you see it like you still have several cores idle, but everything got stuck
[21:20:38] <Link Mauve> Ow.
[21:20:48] robert_mobil leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:24:19] erik leaves the room: Stream closed by us: Replaced by new connection (conflict)
[21:27:53] VC leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:29:19] jeremy leaves the room
[21:34:38] zinid leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:35:25] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:38:20] joekokker leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:38:36] daniel leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:38:58] syever leaves the room
[21:41:32] erik leaves the room: Connection failed: connection closed
[21:47:23] Licaon_Kter leaves the room
[21:47:52] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[21:49:01] sindrake leaves the room
[21:50:53] Alacer leaves the room
[21:52:05] sindrake leaves the room
[21:52:14] Alacer leaves the room: Stream closed by us: Replaced by new connection (conflict)
[21:53:01] <Link Mauve> Oh, another good thing in Prosody which AFAIK Ejabberd still lacks is MUC participants staying in the MUC despite a service restart.
[21:53:35] ludo leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:00:04] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:00:24] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:00:45] Licaon_Kter leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:02:04] beri leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:04:01] Akasch leaves the room: Rechner geht in den Ruhezustand
[22:04:39] adrien leaves the room: Stream closed by us: Replaced by new connection (conflict)
[22:07:55] <zinid> That's a critical feature when you restart twice a day 👍
[22:13:19] <Holger> zinid: The erl_nif man page says up to 1 millisecond is acceptable.
[22:13:45] <Link Mauve> zinid, uh, we restart only once every kernel vulnerability, not much more.
[22:14:09] <Link Mauve> Or when I stupidly do stupid things, but that’s me being stupid. ^^
[22:14:28] <Link Mauve> Like running `reboot` on the wrong machine… ¬_¬
[22:16:30] <Holger> Link Mauve: Isn't this MUC thing that ejabberd *still* lacks a very new Prosody trunk feature? :-)
[22:17:19] <Link Mauve> Holger, very new from some years ago already.
[22:17:32] <Holger> Ah misunderstood that then.
[22:17:38] <Link Mauve> It was decided quite early to ship the future 0.10 with the old MUC code.
[22:17:48] <Link Mauve> And keep the new one in trunk.
[22:19:21] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:19:23] beri leaves the room
[22:27:04] debalance leaves the room: Stream closed by us: Replaced by new connection (conflict)
[22:31:15] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:39:15] Andrew Nenakhov leaves the room: Stream closed by us: Replaced by new connection (conflict)
[22:39:31] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:46:22] 404 leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[22:48:34] Odin leaves the room: Stream closed by us: Replaced by new connection (conflict)
[22:52:36] <Holger> I didn't like the approach of storing the MUC state on shutdown. But might be okay if we do it on-the-fly.
[23:02:06] ta leaves the room: Machine going to sleep
[23:03:37] marc leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:05:21] debalance leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:06:03] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:06:08] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:09:50] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:15:52] alexis leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:21:26] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:26:10] deavmi leaves the room: Disconnected: Replaced by new connection
[23:26:59] joekokker leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:28:22] spicewiesel leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:32:10] tyler leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:32:53] ingolf leaves the room
[23:36:29] Akasch leaves the room
[23:44:20] jeremy leaves the room: Machine going to sleep
[23:44:45] cippaciong leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
[23:46:46] Odin leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:50:20] Zash leaves the room
[23:52:59] ata2001 leaves the room: Disconnected: closed
[23:56:36] jannic leaves the room: Stream closed by us: Replaced by new connection (conflict)
[23:59:09] jeremy leaves the room: Stream closed by us: Timed out waiting for stream resumption (connection-timeout)
Powered by ejabberd - robust, scalable and extensible XMPP server Powered by Erlang Valid XHTML 1.0 Transitional Valid CSS!