Process-one
ejabberd
ejabberd@conference.process-one.net
Wednesday, July 7, 2021< ^ >
zinid has set the subject to: ejabberd discussions: https://docs.ejabberd.im
Room Configuration
Room Occupants

GMT+0
[00:00:00] thrrgilag joins the room
[00:00:35] carlos joins the room
[00:00:41] thrrgilag leaves the room
[00:06:04] writer77 leaves the room
[00:07:52] JonNJ leaves the room: Disconnected: Replaced by new connection
[00:07:53] JonNJ joins the room
[00:10:21] stp leaves the room
[00:10:57] [Alien] leaves the room
[00:10:57] norkki leaves the room
[00:10:58] Lemm1ng leaves the room
[00:10:58] dennis leaves the room
[00:10:58] ff leaves the room
[00:11:33] jerry joins the room
[00:14:56] debacle leaves the room
[00:16:57] deepest leaves the room
[00:16:57] pixel5 leaves the room
[00:16:58] basheer leaves the room
[00:17:48] mimi89999 leaves the room
[00:17:53] mimi89999 joins the room
[00:21:56] bowlofeggs leaves the room
[00:21:58] bowlofeggs joins the room
[00:23:56] raver leaves the room
[00:25:52] deepest joins the room
[00:28:50] mimi89999 leaves the room
[00:28:54] mimi89999 joins the room
[00:33:59] [Alien] leaves the room
[00:34:37] [Alien] joins the room
[00:36:53] kryptos joins the room
[00:37:28] gooya leaves the room
[00:43:11] kryptos leaves the room
[00:45:41] kinetik joins the room
[00:47:33] multixmp222 leaves the room
[00:54:23] ij joins the room
[00:57:53] Demo leaves the room
[01:00:23] ij leaves the room
[01:05:23] ij joins the room
[01:06:25] hlad joins the room
[01:09:16] norkki joins the room
[01:11:39] presprout leaves the room
[01:12:05] ij leaves the room
[01:15:51] isotelo leaves the room
[01:16:16] ThFree leaves the room
[01:19:43] ThFree joins the room
[01:35:46] bowlofeggs leaves the room
[01:35:46] bowlofeggs leaves the room
[01:40:48] bowlofeggs joins the room
[01:40:51] bowlofeggs joins the room
[01:48:12] isotelo leaves the room
[01:49:05] paul leaves the room
[01:54:54] isotelo joins the room
[01:56:50] [Alien] leaves the room
[01:57:39] twl leaves the room
[01:57:55] [Alien] joins the room
[01:59:29] deepest leaves the room
[01:59:59] isotelo leaves the room
[02:01:40] deepest joins the room
[02:01:54] JuniorJPDJ leaves the room
[02:07:14] doctor leaves the room
[02:07:18] doctor joins the room
[02:14:46] isotelo joins the room
[02:17:19] deepest leaves the room
[02:17:30] raver joins the room
[02:23:35] moparisthebest joins the room
[02:24:35] deepest joins the room
[02:26:39] pixel5 joins the room
[02:32:10] deepest leaves the room
[02:33:14] kaonashi696 leaves the room
[02:33:14] deepest joins the room
[02:42:35] moparisthebest leaves the room
[02:43:37] ThFree leaves the room
[02:44:13] [Alien] leaves the room
[02:45:39] ThFree joins the room
[02:46:44] carlos joins the room
[02:51:06] thrrgilag leaves the room
[02:56:39] [Alien] joins the room
[02:57:14] dos leaves the room
[03:05:35] mss_cyclist leaves the room
[03:05:46] marc0s leaves the room: Disconnected: Replaced by new connection
[03:05:47] marc0s joins the room
[03:05:53] mss_cyclist joins the room
[03:07:55] mss_cyclist leaves the room
[03:08:06] mss_cyclist joins the room
[03:08:44] carlos leaves the room
[03:08:48] mss_cyclist leaves the room
[03:08:59] mss_cyclist joins the room
[03:11:45] ff joins the room
[03:12:30] [Alien] leaves the room
[03:16:28] Mike leaves the room
[03:16:28] Mike joins the room
[03:16:29] [Alien] joins the room
[03:17:03] mss_cyclist leaves the room
[03:17:10] mss_cyclist joins the room
[03:21:46] thrrgilag joins the room
[03:25:20] pprrks leaves the room: Disconnected: Replaced by new connection
[03:25:23] pprrks joins the room
[03:29:27] ThFree leaves the room
[03:29:45] ThFree joins the room
[03:31:54] riau.pro leaves the room: Disconnected: Received SIGTERM
[03:32:21] riau.pro joins the room
[03:33:44] [Alien] leaves the room
[03:37:47] thrrgilag joins the room
[03:39:12] carlos joins the room
[03:46:47] St0ne leaves the room
[03:50:57] sumpfralle_ leaves the room
[03:50:57] sumpfralle leaves the room
[03:51:28] St0ne joins the room
[03:56:02] nekit leaves the room
[04:00:58] lorddavidiii joins the room
[04:03:08] JonNJ leaves the room: Disconnected: Replaced by new connection
[04:03:08] JonNJ joins the room
[04:05:55] kaonashi696 leaves the room
[04:13:31] robert_mobil joins the room
[04:13:52] Ellenor Bjornsdottir joins the room
[04:17:07] pod joins the room
[04:25:41] St0ne leaves the room
[04:27:15] ThFree leaves the room
[04:28:09] JuniorJPDJ joins the room
[04:28:47] St0ne joins the room
[04:30:04] ThFree joins the room
[04:34:39] daniel leaves the room
[04:37:10] DebXWoody joins the room
[04:39:45] Licaon_Kter leaves the room
[04:40:25] Licaon_Kter joins the room
[04:43:50] abba joins the room
[04:44:30] paul joins the room
[04:49:14] Lemm1ng joins the room
[04:51:02] rauhfasertapete joins the room
[04:52:21] daniel joins the room
[04:53:35] schäfchen726 joins the room
[04:55:43] ben leaves the room
[04:59:05] mss_cyclist leaves the room
[04:59:36] nand leaves the room
[04:59:45] mss_cyclist joins the room
[05:00:21] ij joins the room
[05:02:37] pod joins the room
[05:05:51] pprrks leaves the room
[05:06:14] Dom leaves the room
[05:11:36] Jeroen joins the room
[05:11:56] robert_mobil leaves the room
[05:11:59] stp joins the room
[05:12:10] Jeroen leaves the room
[05:12:11] robert_mobil joins the room
[05:13:25] robert_mobil leaves the room
[05:13:40] robert_mobil joins the room
[05:13:44] Dom joins the room
[05:14:07] Jeroen joins the room
[05:14:36] Jeroen leaves the room
[05:14:58] thrrgilag leaves the room
[05:15:18] thrrgilag leaves the room
[05:19:58] karme joins the room
[05:20:07] sander_ leaves the room
[05:24:23] carlos leaves the room
[05:26:55] bowlofeggs leaves the room
[05:26:58] Licaon_Kter leaves the room
[05:27:01] Licaon_Kter joins the room
[05:32:32] sander_ joins the room
[05:33:54] schäfchen726 leaves the room
[05:35:11] abba leaves the room
[05:37:24] norkki leaves the room
[05:41:09] norkki joins the room
[05:41:47] mss_cyclist leaves the room
[05:42:08] mss_cyclist joins the room
[05:42:17] sander leaves the room
[05:43:16] basheer joins the room
[05:44:28] sander joins the room
[05:45:18] pprrks joins the room
[05:50:13] schmillin leaves the room
[05:50:19] kaonashi696 joins the room
[05:53:17] ij leaves the room
[05:55:00] schmillin joins the room
[05:56:52] ij joins the room
[05:57:17] Jeroen joins the room
[05:57:46] Jeroen leaves the room
[06:04:05] danielepiccoli leaves the room
[06:05:34] JonNJ leaves the room: Disconnected: Replaced by new connection
[06:05:34] JonNJ joins the room
[06:06:12] stp leaves the room
[06:10:11] nekit joins the room
[06:10:50] lorddavidiii leaves the room
[06:11:41] marc joins the room
[06:12:01] Jeroen joins the room
[06:12:17] Holger leaves the room
[06:12:32] Jeroen leaves the room
[06:14:15] kaonashi696 leaves the room: offline
[06:14:34] Holger joins the room
[06:14:55] stp joins the room
[06:17:28] sachaobado joins the room
[06:17:57] Ellenor Bjornsdottir joins the room
[06:18:03] Jeroen joins the room
[06:18:32] Jeroen leaves the room
[06:24:39] Marc joins the room
[06:25:23] JonNJ leaves the room: Disconnected: Replaced by new connection
[06:25:23] JonNJ joins the room
[06:27:12] Ellenor Bjornsdottir leaves the room
[06:30:06] danielepiccoli joins the room
[06:31:21] SERGE90 joins the room
[06:33:42] lorddavidiii joins the room
[06:36:08] ij leaves the room
[06:37:02] schmillin joins the room
[06:40:18] ij joins the room
[06:41:01] Mike leaves the room
[06:43:04] JonNJ leaves the room: Disconnected: Replaced by new connection
[06:43:04] JonNJ joins the room
[06:44:23] schmillin leaves the room
[06:45:57] writer77 joins the room
[06:48:17] Jeroen joins the room
[06:48:19] writer77 leaves the room
[06:48:21] writer77 joins the room
[06:48:30] Jeroen leaves the room
[06:48:31] Jeroen joins the room
[06:48:47] Jeroen leaves the room
[06:48:47] Jeroen joins the room
[06:48:53] Jeroen leaves the room
[06:50:16] Lemm1ng leaves the room
[06:50:16] Tim leaves the room
[06:50:17] kinetik leaves the room
[06:50:25] Jeroen joins the room
[06:50:42] ThUnD3r|Gr33n leaves the room
[06:50:45] Lemm1ng joins the room
[06:50:47] ThUnD3r|Gr33n joins the room
[06:50:53] Jeroen leaves the room
[06:51:06] Tim joins the room
[06:53:53] sander leaves the room
[06:54:55] مسعود leaves the room
[06:54:57] مسعود joins the room
[06:55:46] danielepiccoli leaves the room
[06:56:20] Tshering Lhamo leaves the room
[06:57:09] sander joins the room
[06:57:15] LNJ joins the room
[06:57:53] JonNJ leaves the room: Disconnected: Replaced by new connection
[06:57:54] JonNJ joins the room
[06:58:18] bammes joins the room
[06:58:36] Tshering Lhamo joins the room
[06:58:57] madmalkav joins the room
[06:59:28] danielepiccoli joins the room
[07:02:16] sander leaves the room
[07:03:55] ij leaves the room
[07:04:10] ThUnD3r|Gr33n leaves the room
[07:07:03] schäfchen726 joins the room
[07:07:09] sander joins the room
[07:07:42] sonny leaves the room: Disconnected: No route to host
[07:13:43] JonNJ leaves the room: Disconnected: Replaced by new connection
[07:13:43] JonNJ joins the room
[07:15:33] quentin leaves the room
[07:16:59] SERGE90 leaves the room
[07:19:04] prefiks joins the room
[07:21:54] ij joins the room
[07:22:21] chrys leaves the room
[07:22:22] chrys joins the room
[07:23:51] x0n joins the room
[07:24:12] schmillin leaves the room
[07:24:14] schmillin joins the room
[07:24:22] Marc leaves the room
[07:24:50] ThUnD3r|Gr33n joins the room
[07:25:27] sonny joins the room
[07:27:56] ij leaves the room
[07:30:14] ave leaves the room
[07:31:08] ave joins the room
[07:31:44] Dom joins the room
[07:31:44] JonNJ leaves the room: Disconnected: Replaced by new connection
[07:31:45] JonNJ joins the room
[07:32:19] marevalo joins the room
[07:35:01] 86ul joins the room
[07:37:46] ChaosKid42 leaves the room
[07:43:50] neox joins the room
[07:46:19] writer77 leaves the room
[07:46:20] writer77 joins the room
[07:46:44] JonNJ leaves the room: Disconnected: Replaced by new connection
[07:46:44] JonNJ joins the room
[07:50:47] debacle joins the room
[07:51:25] deepest leaves the room
[07:52:08] deepest joins the room
[07:52:57] ChaosKid42 leaves the room
[07:53:23] stp joins the room
[07:54:45] Tshering Lhamo leaves the room
[07:55:36] Tshering Lhamo joins the room
[07:57:40] mss_cyclist leaves the room
[07:59:17] mss_cyclist joins the room
[08:01:08] nekit leaves the room
[08:02:11] [Alien] joins the room
[08:03:37] JonNJ leaves the room
[08:03:38] JonNJ joins the room
[08:04:55] JonNJ joins the room
[08:05:27] danielepiccoli leaves the room
[08:06:33] ij joins the room
[08:07:35] norkki joins the room
[08:26:18] egnun leaves the room
[08:27:10] JonNJ leaves the room: Disconnected: Replaced by new connection
[08:27:10] JonNJ joins the room
[08:29:39] jerry leaves the room
[08:29:47] deepest leaves the room
[08:35:22] mss_cyclist leaves the room
[08:37:27] norkki leaves the room
[08:38:16] mss_cyclist joins the room
[08:38:32] jerry joins the room
[08:39:59] norkki leaves the room
[08:42:36] deepest joins the room
[08:44:06] jerry leaves the room
[08:45:05] jerry joins the room
[08:47:14] norkki joins the room
[08:48:04] Marc joins the room
[08:50:15] deepest leaves the room
[08:51:14] jerry leaves the room
[08:51:30] deepest joins the room
[08:53:42] mimi89999 leaves the room
[08:54:52] quentin joins the room
[08:57:30] multixmp222 joins the room
[09:00:10] ij leaves the room
[09:01:16] presprout joins the room
[09:02:06] ChaosKid42 joins the room
[09:04:07] mimi89999 joins the room
[09:04:18] Alacer leaves the room
[09:04:33] neox leaves the room
[09:04:37] neox joins the room
[09:05:30] quentin leaves the room
[09:05:35] quentin joins the room
[09:06:26] ChaosKid42 joins the room
[09:13:10] ij joins the room
[09:15:22] kikuchiyo joins the room
[09:15:47] JonNJ leaves the room: Disconnected: Replaced by new connection
[09:15:48] JonNJ joins the room
[09:17:49] Marc leaves the room
[09:19:12] debacle leaves the room
[09:19:56] Holger joins the room
[09:20:15] jerry joins the room
[09:26:25] jerry leaves the room
[09:27:03] kikuchiyo leaves the room
[09:29:08] gluecksmoehre leaves the room
[09:29:08] gluecksmoehre joins the room
[09:30:30] kinky_nekoboi joins the room
[09:30:51] Alacer joins the room
[09:34:01] nekit joins the room
[09:37:53] kikuchiyo joins the room
[09:38:58] Tim leaves the room
[09:41:45] Tim joins the room
[09:42:39] ij leaves the room
[09:44:02] kikuchiyo leaves the room
[09:44:37] kikuchiyo joins the room
[09:44:52] gooya joins the room
[09:45:22] Jeroen joins the room
[09:45:50] 86ul leaves the room
[09:46:01] Jeroen leaves the room
[09:46:11] 86ul joins the room
[09:48:16] Jeroen joins the room
[09:48:25] writer77 leaves the room
[09:48:34] Holger leaves the room
[09:48:37] <Giovanni> Brace yourself, another benchmarking question is coming 😃
[09:49:28] Holger joins the room
[09:49:32] <Licaon_Kter> :)
[09:50:16] Jeroen leaves the room
[09:51:59] Jeroen joins the room
[09:52:26] Jeroen leaves the room
[09:53:34] ThUnD3r|Gr33n leaves the room
[09:53:37] bammes leaves the room
[09:53:40] doctor leaves the room
[09:53:57] <Holger> 😱️
[09:54:59] writer77 joins the room
[09:55:03] <Holger> Giovanni, now that you're into RTB, please compare ejabberd on Erlang 24.x vs. ejabberd on any older Erlang version!
[09:55:23] jerry joins the room
[09:55:30] <Holger> (I promised Licaon_Kter to do that for the past 10 or 20 years.)
[09:56:16] <Holger> (On AMD64, that is.  Because JIT.)
[09:56:43] <Holger> Once you did that, we'll continue answering questions.
[09:56:48] JonNJ leaves the room
[09:56:49] <Holger> 🙂
[09:56:49] JonNJ joins the room
[09:57:03] <Giovanni> ok, the fact that i'm into RTB it's... questionable 😃 I'm still struggling to understand 😃
[09:57:21] <Holger> Excuses.
[09:57:51] <Giovanni> but of course in the next couple of years i'm sure i can be useful for your tests 😃
[09:58:09] <Holger> \o/
[09:58:20] <Holger> So what's your question 🙂
[09:59:05] <Giovanni> So, as I already said many times (sorry for being so boring), my aim for now is to reach 20k concurrent users on ejabberd
[10:00:51] <Giovanni> This time I tried this rtb configuration:
interval: 300
capacity: 20000
negotiation_timeout: 100
connect_timeout: 100
reconnect_interval: 60
message_interval: 300
presence_interval: 600
disconnect_interval: false
message_body_size: 100
no http_upload or proxy65
[10:01:33] <Giovanni> but I cannot reach more than 4000 users online (more or less)
[10:01:46] Jeroen joins the room
[10:01:58] ThUnD3r|Gr33n joins the room
[10:02:14] nekit leaves the room
[10:02:16] Jeroen leaves the room
[10:02:19] <Giovanni> the ejabberd server is:
cpu 16 virtual core Intel(R) Xeon(R) CPU E5-2665 0 @ 2.40GHz
ram 32 gb
[10:03:01] <Giovanni> it seems to be pretty busy but with no iowait or memory or cpu problems
[10:03:12] Jeroen joins the room
[10:03:18] <Holger> (FWIW I do have something in between 4 and 20k on very similar hardware and that server is bored.)
[10:04:08] <prefiks> Could you start debug shell on ejabberd when you run that test (ejabberdctl debug) and see what ‚p1_prof:q().’ and ‚p1_prof:r()’ returns?
[10:04:24] <Holger> Well I see two options.  (1) Radically adjust all RTP settings except the capacity towards reducing load for tracking things down.  (2) ... what prefiks said 🙂
[10:05:14] <Holger> Well I see two options.  (1) Radically adjust all RTB settings except the capacity towards reducing load, for tracking things down.  (2) ... what prefiks said 🙂
[10:05:51] Jeroen leaves the room
[10:06:03] <Giovanni> Holger "bored" in the sense that is overloaded or the opposite?
[10:06:04] deepest leaves the room
[10:06:31] <Holger> Giovanni, opposite.  `Load average: 0.26, 0.33, 0.41`.
[10:06:40] Jeroen joins the room
[10:06:46] <Giovanni> ok
[10:06:51] <Holger> (With MariaDB running on the same system, and having tons of RAM allocated for SQL caching.)
[10:07:08] Jeroen leaves the room
[10:07:11] <Giovanni> I have also mariadb running on same node
[10:07:24] <Giovanni> perhaps i can play with sql caching, i don't know
[10:07:41] <Holger> ejabberd itself eats about 400k/device of RAM and no CPU.
[10:08:35] <Giovanni> I'll forward you the htop of the ejabberd server
[10:08:52] Jeroen joins the room
[10:09:06] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/RkYJp25VmjwGIXjyTBCprs5JAEwNxc7iaEeDyw9X/image.png
[10:09:08] <Holger> Giovanni, see prefiks' suggestion.  It's easy and may well show the bottleneck.
[10:09:23] Jeroen leaves the room
[10:09:45] Holger leaves the room
[10:09:49] <Giovanni> yes, I'll do it right now, one sec because the rtb shell is running on my colleague connection 😃
[10:10:11] <Giovanni> the magic of smart working 😃
[10:16:59] ij joins the room
[10:18:50] kinky_nekoboi leaves the room
[10:21:07] <Giovanni> Here it is the pastebin with the output of the two prefiks commands on rtb
https://pastebin.com/evPb7ewD
[10:24:01] Tshering Lhamo leaves the room
[10:24:31] Marc joins the room
[10:24:38] jadmanthrat joins the room
[10:24:42] <prefiks> So looks like no queues, doesn’t looks like overloaded
[10:25:39] Tshering Lhamo joins the room
[10:25:47] debacle joins the room
[10:26:13] <Giovanni> if I do rtb_stats:show_errors(). I see this
[10:26:15] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/BpClb8Cuq3XrxGhJc5YgBDFHe5YeiFiPeaX2snNu/image.png
[10:26:25] <Giovanni> but I don't know if this is normal
[10:32:59] yann-kaelig joins the room
[10:33:06] <Giovanni> Il'l forward you the rtb graphs screenshots for the sake of completeness
[10:33:28] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/kR5rraEqgiEyYE8Ky0gUmapOoQW1KUCLWnIjRVr1/image.png
[10:34:09] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/xvlVc76h4jU1KWfliobIG8hjLTyKASOPSFevsHEw/image.png
[10:35:04] spla joins the room
[10:35:40] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/jyePtMEGQu4EOLhMAbHxRxZpuxS0Wd8DVOEAwmIw/image.png
[10:38:39] sumpfralle_ joins the room
[10:38:50] sumpfralle joins the room
[10:39:05] Holger joins the room
[10:42:19] spla leaves the room
[10:42:56] <Giovanni> sorry prefiks I totally misunderstood your suggestions
[10:43:34] <Giovanni> I run the commands in rtb, not in ejabberd. Now I'm running in ejabberd and let you know
[10:44:40] <Giovanni> I just launched p1_prof:q(). and I'm waiting the output, it seems it needs time
[10:46:21] deepest joins the room
[10:46:29] <Giovanni> I presume that this long time before the output is not a good news
[10:47:12] <Holger> You didn't forget the terminating `.`?
[10:47:24] <Giovanni> no, no there is the .
[10:48:01] <Holger> And now it's hanging, no prompt?
[10:48:07] <Giovanni> yes
[10:48:24] Tshering Lhamo leaves the room
[10:48:28] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/UNynxYlsgRMu8ZuDXEeLxkMz2xfg6X982X8BGdho/image.png
[10:49:39] Tshering Lhamo joins the room
[10:49:52] Marc leaves the room
[10:49:59] <Holger> 🤔
[10:52:44] <Giovanni> ok here it is the output
[10:53:15] <Holger> So it did return after a long time?
[10:53:18] <Giovanni> https://pastebin.com/ZQ1hQRfa
[10:53:28] <Giovanni> Holger yes it did
[10:53:38] <Giovanni> now i'm trying the second command
[10:53:48] <Holger> p1_prof:r().
[10:54:20] <Giovanni> yes just launched
[10:54:56] <Holger> First one says ejabberd is queueing up MySQL queries.
[10:55:03] <Holger> Is MySQL/MariaDB running on the same system?
[10:55:43] ij leaves the room
[10:56:11] <Giovanni> yes, is running on the same system
[10:57:18] <Giovanni> so it may be that mysql is bad tuned
[10:58:00] <Holger> Let's see the `r()` output first (basically shows which ejabberd tasks eat most of your CPU time).
[10:58:03] emilio.bustos leaves the room
[10:58:36] <Holger> You do have ejabberd-internal caching enabled?  Also for authentication?
[10:58:38] sander leaves the room
[11:00:03] <Giovanni> I have cache_size defined globally
[11:00:03] sander joins the room
[11:00:11] <Giovanni> but use_cache defined only on mod_roster
[11:02:03] <Giovanni> this means the cache is used only from mod_roster?
[11:02:04] sumpfralle leaves the room
[11:02:04] sumpfralle_ leaves the room
[11:02:08] <Holger> Should be okay then.  Caching is enabled by default.
[11:02:12] <Giovanni> ok
[11:02:14] <Giovanni> fine
[11:02:23] <Holger> https://docs.ejabberd.im/admin/configuration/toplevel/#use-cache
[11:02:44] <Holger> Size is 1000 by default, https://docs.ejabberd.im/admin/configuration/toplevel/#cache-size ...
[11:03:08] <Holger> And auth_use_cache falls back to use_cache by default: https://docs.ejabberd.im/admin/configuration/toplevel/#auth-use-cache ...
[11:03:33] <Giovanni> I have the output for r()
[11:03:36] <Holger> Yay.
[11:03:53] emilio.bustos joins the room
[11:04:00] <Giovanni> https://pastebin.com/ey0ZESdS
here it is
[11:04:12] Jeroen joins the room
[11:04:20] ij joins the room
[11:04:41] Jeroen leaves the room
[11:05:16] nuron joins the room
[11:05:35] sander leaves the room
[11:05:47] yann-kaelig leaves the room
[11:06:33] writer77 leaves the room
[11:06:38] conv leaves the room
[11:06:38] conv joins the room
[11:07:05] Holger leaves the room
[11:07:10] sander joins the room
[11:09:54] <Licaon_Kter> > Giovanni, now that you're into RTB, please compare ejabberd on Erlang 24.x vs. ejabberd on any older Erlang version!
Pfft any fool can JIT on x86
We need RISC-V results now!!!
[11:10:09] writer77 joins the room
[11:10:20] mightyBroccoli joins the room
[11:10:24] <Giovanni> LOL
[11:11:37] <Holger> So nothing obvious (to me), ejabberd is just fighting with handling the clients.  (prefiks?)
[11:11:51] <Holger> Giovanni, I would indeed start with trying to tune MySQL, just to get that out of the way.
[11:11:58] <Holger> Are you basically using the defaults?
[11:12:15] <prefiks> Yes, it doesn’t look like bottleneck here is ejabberd processing
[11:12:21] <prefiks> Could be db access indeed
[11:12:34] <prefiks> You could try increasing sql workers count
[11:12:40] Holger joins the room
[11:12:41] <Giovanni> I added these:
[mysqld]
max_connections = 29538
max_user_connections = 0
wait_timeout = 50000
interactive_timeout = 50000
thread_cache_size = 400
max_connect_errors = 200
max_allowed_packet = 500M
[11:13:22] <Holger> You have 32G RAM and don't need if for anything noteworthy besides ejabberd/MySQL?
[11:13:31] <Holger> And: Do you have SSD storage?
[11:14:08] <Giovanni> Holger nothing more, it's a dedicated virtual machine
[11:14:35] <Holger> I asked before sorry: MySQL or MariaDB (which version)?
[11:14:37] <Giovanni> I don't know if the storage in the test environment is ssd I'll ask it to my sysadm
[11:14:58] Marzanna leaves the room
[11:15:01] <Licaon_Kter> Giovanni: sql_pool is what?
[11:15:38] <Giovanni> Licaon_Kter sql_pool_size for ejabberd is 1000
[11:15:52] <Holger> Ohh that's a lot 🙂
[11:15:54] deepest leaves the room
[11:16:03] <Holger> Maybe even reduce that.
[11:16:16] <Giovanni> Server version: 10.3.29-MariaDB-0ubuntu0.20.04.1 Ubuntu 20.04
[11:16:26] Wojtek joins the room
[11:16:33] <Holger> Dunno but can imagine performance being degraded by creating too many MariaDB threads.
[11:16:35] deepest joins the room
[11:16:49] <Giovanni> Holger ok so maybe reducing to default (10) for a single vhost is fine?
[11:16:50] <Holger> The default is 10 connections per virtual ejabberd host.
[11:16:56] <Holger> I'd try.
[11:16:58] schäfchen726 leaves the room
[11:16:59] schäfchen726 joins the room
[11:17:00] <Giovanni> ok
[11:17:09] <Holger> And I'd adjust the MariaDB config.  But depends on storage.
[11:18:14] <Giovanni> what attribute in particular?
[11:18:31] <Holger> For 32G RAM and SSD storage (i.e., constant `seek()` time) I'd start with something like this (in addition to your settings above, which are fine but won't help with I/O performance):
```
max_heap_table_size      = 64M
tmp_table_size           = 64M
innodb_buffer_pool_size             = 16G
innodb_buffer_pool_instances        = 8
innodb_buffer_pool_dump_at_shutdown = 1
innodb_buffer_pool_load_at_startup  = 1
innodb_flush_log_at_trx_commit      = 0
innodb_flush_neighbors              = 0
innodb_checksum_algorithm           = crc32
```
[11:18:39] ij leaves the room
[11:18:54] <Giovanni> ok
[11:19:22] <Giovanni> the weird thing is iostats doesn't seem to suggest somebody is going wrong with i/o
[11:19:39] <Giovanni> but i'm not such expert
[11:20:01] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/yj8Tpcwo6UxETEciL6frQSr5XApUSTCP7ROb7u8X/image.png
[11:20:07] <Holger> I.e., half the RAM for MariaDB caching.  Might be too much in the end, the relative amount RAM for MariaDB vs. ejabberd caching is debatable.  But I'd kinda start with the opposite of the current relation.
[11:20:15] <Holger> And then play around.
[11:20:51] <Giovanni> ok
[11:21:07] <Giovanni> I'll try it just after lunch, thank you very much
[11:21:15] bowlofeggs joins the room
[11:22:17] Jeroen joins the room
[11:24:06] Jeroen leaves the room
[11:24:15] [Alien] joins the room
[11:26:39] danielepiccoli joins the room
[11:30:25] Jeroen joins the room
[11:30:30] nuron leaves the room
[11:31:00] Jeroen leaves the room
[11:31:14] ij joins the room
[11:32:06] ij leaves the room
[11:32:26] poni joins the room
[11:32:45] sander leaves the room
[11:33:10] sander joins the room
[11:34:43] ij joins the room
[11:35:34] sander leaves the room
[11:36:02] sander joins the room
[11:39:31] robert_mobil leaves the room
[11:43:28] [Alien] leaves the room
[11:46:20] neox leaves the room
[11:49:21] Marc joins the room
[11:50:55] Jeroen joins the room
[11:51:28] Jeroen leaves the room
[11:51:32] Jeroen joins the room
[11:53:25] Jeroen leaves the room
[11:53:26] Jeroen joins the room
[11:55:17] anuj9122 leaves the room
[11:56:00] Holger leaves the room
[11:56:22] Holger joins the room
[11:57:16] spla joins the room
[11:57:29] Jeroen leaves the room
[11:58:54] kryptos joins the room
[12:02:17] egnun joins the room
[12:03:41] ij leaves the room
[12:05:01] Dom leaves the room
[12:06:21] Demo joins the room
[12:06:53] batonga leaves the room
[12:06:53] batonga joins the room
[12:07:29] Walter leaves the room
[12:07:44] yann-kaelig joins the room
[12:09:25] Holger leaves the room
[12:09:40] robert_mobil joins the room
[12:11:11] spla leaves the room
[12:11:38] sander leaves the room
[12:15:09] Barny leaves the room
[12:16:06] Barny joins the room
[12:16:24] Dom joins the room
[12:17:09] sander joins the room
[12:17:10] basheer leaves the room
[12:20:31] nico leaves the room
[12:21:40] nico joins the room
[12:22:10] vanitasvitae joins the room
[12:23:36] anuj9122 joins the room
[12:26:49] Jeroen joins the room
[12:27:16] Jeroen leaves the room
[12:27:25] mss_cyclist leaves the room
[12:29:34] sander leaves the room
[12:30:11] ij joins the room
[12:31:46] ij leaves the room
[12:32:08] sander joins the room
[12:35:02] mss_cyclist joins the room
[12:35:52] neox joins the room
[12:37:37] mightyBroccoli leaves the room
[12:40:00] kryptos leaves the room
[12:48:39] lorddavidiii leaves the room
[12:52:02] x51 joins the room
[12:52:42] flow joins the room
[12:58:22] deepest leaves the room
[12:58:56] deepest joins the room
[12:59:20] lorddavidiii joins the room
[13:00:00] lorddavidiii leaves the room
[13:01:35] lorddavidiii joins the room
[13:02:32] thrrgilag joins the room
[13:06:35] Mike joins the room
[13:07:52] moparisthebest joins the room
[13:09:38] sachaobado leaves the room
[13:09:38] Tim leaves the room
[13:11:59] danielepiccoli leaves the room
[13:12:01] danielepiccoli joins the room
[13:16:38] dos joins the room
[13:18:40] sachaobado joins the room
[13:19:25] badlop joins the room
[13:20:56] kryptos joins the room
[13:21:10] dos leaves the room
[13:21:13] dos joins the room
[13:21:28] stp leaves the room
[13:21:43] kinky_nekoboi joins the room
[13:28:33] [Alien] joins the room
[13:32:10] sachaobado joins the room
[13:33:01] moparisthebest leaves the room
[13:33:07] moparisthebest joins the room
[13:33:08] ave leaves the room
[13:34:24] ij joins the room
[13:34:41] ave joins the room
[13:38:17] debacle leaves the room
[13:38:28] Holger joins the room
[13:38:35] debacle joins the room
[13:39:27] Marc leaves the room
[13:40:27] ij leaves the room
[13:41:05] dos leaves the room
[13:43:43] dos joins the room
[13:45:22] nico leaves the room
[13:45:22] sachaobado leaves the room
[13:45:44] stp joins the room
[13:48:58] Tim leaves the room
[13:51:01] sumpfralle_ joins the room
[13:51:12] sumpfralle joins the room
[13:51:40] nico joins the room
[13:54:50] thrrgilag joins the room
[13:58:15] twl leaves the room
[14:02:17] lorddavidiii leaves the room
[14:05:45] ij joins the room
[14:08:17] gooya leaves the room
[14:14:14] JuniorJPDJ leaves the room
[14:14:30] JuniorJPDJ joins the room
[14:15:42] mss_cyclist leaves the room
[14:15:42] ben joins the room
[14:15:56] mike leaves the room
[14:16:14] mss_cyclist joins the room
[14:16:25] mike joins the room
[14:16:47] isotelo leaves the room
[14:16:48] isotelo joins the room
[14:17:51] Wojtek leaves the room
[14:21:29] isotelo joins the room
[14:22:58] mike leaves the room
[14:23:03] mike joins the room
[14:25:54] nico leaves the room
[14:26:41] nico joins the room
[14:29:42] Wojtek joins the room
[14:32:54] <Giovanni> Ok, after tuning mysql with Holger suggestions, the performance got better and I achieved 5000 connections before the system start to struggle. It's a big step for humanity but i'm still far from the desired outcome. I'll forward some of the rtb graphs.
[14:33:00] <Giovanni> https://xmpp.jix.im/upload/99e944b81a9df91592c53fda34890c47c0498178/YQYLxgNIrpTzPWjQ6tcTO0oh9kJMCZ4B49zfl4UC/image.png
[14:34:06] Dom leaves the room
[14:34:52] twl joins the room
[14:36:44] <Giovanni> Odd messages on rtb:
rtb@localhost)5> rtb_stats:show_errors().
Stream closed by peer: Too many (30) failed authentications from this IP address (::ffff:XX.XX.XXX.XXX). The address will be unblocked at 15:13:42 07.07.2021 UTC (policy-violation): 1
it could be an error related to mysql not responding to ejabberd authentication?
[14:37:51] mike leaves the room
[14:38:25] mike joins the room
[14:41:21] JonNJ leaves the room: Machine is going to sleep
[14:42:34] <Giovanni> I add that now I haven't a SSD storage
[14:43:26] JonNJ leaves the room
[14:43:27] JonNJ joins the room
[14:43:44] neox joins the room
[14:43:59] debalance joins the room
[14:45:17] gooya joins the room
[14:46:28] <Holger> Giovannni, remove the `innodb_flush_neighbors` setting then.
[14:46:38] <Giovanni> ok
[14:46:41] <Holger> If you're using a normal HDD, that is.
[14:46:50] <Giovanni> ok
[14:46:59] egnun leaves the room
[14:47:51] emilio.bustos leaves the room
[14:48:21] mightyBroccoli leaves the room
[14:48:54] JonNJ leaves the room
[14:49:06] nekit joins the room
[14:51:15] JonNJ joins the room
[14:51:49] mightyBroccoli joins the room
[14:52:19] <badlop> hi debalance !
[14:53:28] multixmp222 leaves the room
[14:58:21] jerry leaves the room
[14:59:50] JonNJ leaves the room
[15:00:06] jerry joins the room
[15:01:24] deepest leaves the room
[15:06:16] jerry leaves the room
[15:11:28] neox leaves the room
[15:13:51] emilio.bustos joins the room
[15:15:52] <debalance> hello badlop
[15:17:36] Jeroen joins the room
[15:17:46] robert_mobil joins the room
[15:18:02] Jeroen leaves the room
[15:18:31] <badlop> when preparing the debian packages of ejabberd and libraries, it uses rebar2 (the default tool), right?
[15:19:33] kryptos leaves the room
[15:20:55] Stephen leaves the room
[15:21:52] Stephen joins the room
[15:23:29] sachaobado joins the room
[15:23:41] Holger leaves the room
[15:24:00] neox joins the room
[15:24:17] marevalo leaves the room
[15:24:52] JonNJ joins the room
[15:25:37] nand joins the room
[15:27:29] mss_cyclist leaves the room
[15:27:47] Holger joins the room
[15:28:01] mss_cyclist joins the room
[15:29:54] basheer joins the room
[15:31:30] danielepiccoli leaves the room
[15:32:43] sachaobado leaves the room
[15:33:47] ij leaves the room
[15:34:14] jannic leaves the room
[15:37:31] basheer leaves the room
[15:38:29] JonNJ leaves the room
[15:38:30] JonNJ joins the room
[15:40:39] sachaobado joins the room
[15:42:05] deepest joins the room
[15:42:20] danielepiccoli joins the room
[15:43:15] lorddavidiii joins the room
[15:43:32] lorddavidiii leaves the room
[15:44:24] jannic joins the room
[15:45:25] lorddavidiii joins the room
[15:46:02] lorddavidiii leaves the room
[15:46:19] JonNJ leaves the room
[15:47:18] lorddavidiii joins the room
[15:47:54] lorddavidiii leaves the room
[15:48:31] danielepiccoli leaves the room
[15:49:15] lorddavidiii joins the room
[15:49:55] lorddavidiii leaves the room
[15:51:20] lorddavidiii joins the room
[15:51:34] Lemm1ng leaves the room
[15:51:50] lorddavidiii leaves the room
[15:53:13] lorddavidiii joins the room
[15:53:49] lorddavidiii leaves the room
[15:54:51] mss_cyclist leaves the room
[15:55:13] mss_cyclist joins the room
[15:55:18] lorddavidiii joins the room
[15:55:49] JonNJ joins the room
[15:56:04] lorddavidiii leaves the room
[15:57:20] lorddavidiii joins the room
[15:58:05] lorddavidiii leaves the room
[15:59:21] lorddavidiii joins the room
[16:00:08] lorddavidiii leaves the room
[16:01:13] sezuan joins the room
[16:01:23] lorddavidiii joins the room
[16:02:11] lorddavidiii leaves the room
[16:02:24] stp leaves the room
[16:02:48] basheer joins the room
[16:03:37] lorddavidiii joins the room
[16:03:57] stp joins the room
[16:04:15] lorddavidiii leaves the room
[16:04:24] sezuan joins the room
[16:04:43] Holger leaves the room
[16:05:11] Jeroen joins the room
[16:05:32] lorddavidiii joins the room
[16:05:45] Jeroen leaves the room
[16:06:17] debalance leaves the room
[16:06:19] sezuan leaves the room
[16:06:21] lorddavidiii leaves the room
[16:07:44] lorddavidiii joins the room
[16:08:33] lorddavidiii leaves the room
[16:09:04] JonNJ joins the room
[16:09:28] ij joins the room
[16:09:52] lorddavidiii joins the room
[16:10:33] lorddavidiii leaves the room
[16:11:03] prefiks joins the room
[16:11:10] neox leaves the room
[16:11:54] lorddavidiii joins the room
[16:12:00] sezuan leaves the room
[16:12:17] Tshering Lhamo leaves the room
[16:12:31] Tshering Lhamo joins the room
[16:12:37] lorddavidiii leaves the room
[16:13:57] lorddavidiii joins the room
[16:14:06] danielepiccoli joins the room
[16:14:36] nik leaves the room
[16:14:43] lorddavidiii leaves the room
[16:15:20] schäfchen726 joins the room
[16:15:21] schäfchen726 joins the room
[16:16:00] lorddavidiii joins the room
[16:16:04] prefiks leaves the room
[16:16:47] lorddavidiii leaves the room
[16:17:14] schäfchen726 leaves the room
[16:17:29] schäfchen726 joins the room
[16:17:56] schäfchen726 joins the room
[16:18:11] lorddavidiii joins the room
[16:18:34] isotelo leaves the room
[16:18:48] lorddavidiii leaves the room
[16:20:03] lorddavidiii joins the room
[16:20:50] lorddavidiii leaves the room
[16:21:28] pprrks leaves the room
[16:21:34] pprrks joins the room
[16:22:06] lorddavidiii joins the room
[16:22:50] lorddavidiii leaves the room
[16:23:32] nik joins the room
[16:23:34] isotelo joins the room
[16:24:08] lorddavidiii joins the room
[16:24:36] ChaosKid42 leaves the room
[16:24:36] ChaosKid42 leaves the room
[16:24:48] lorddavidiii leaves the room
[16:25:19] stp leaves the room
[16:26:21] inkey leaves the room
[16:26:46] lorddavidiii joins the room
[16:26:50] inkey joins the room
[16:27:16] Lemm1ng joins the room
[16:27:18] lorddavidiii leaves the room
[16:28:52] lorddavidiii joins the room
[16:29:32] lorddavidiii leaves the room
[16:30:49] lorddavidiii joins the room
[16:31:32] lorddavidiii leaves the room
[16:33:00] lorddavidiii joins the room
[16:33:14] robert_mobil leaves the room
[16:33:29] robert_mobil joins the room
[16:33:33] lorddavidiii leaves the room
[16:34:03] prefiks leaves the room
[16:34:52] prefiks joins the room
[16:34:56] lorddavidiii joins the room
[16:35:29] lorddavidiii leaves the room
[16:36:44] lorddavidiii joins the room
[16:36:49] danielepiccoli leaves the room
[16:37:27] lorddavidiii leaves the room
[16:38:41] Alacer leaves the room
[16:38:51] lorddavidiii joins the room
[16:39:14] Alacer joins the room
[16:39:26] marc leaves the room
[16:39:36] lorddavidiii leaves the room
[16:40:51] lorddavidiii joins the room
[16:41:00] robert_mobil joins the room
[16:41:33] lorddavidiii leaves the room
[16:41:37] stp joins the room
[16:42:57] lorddavidiii joins the room
[16:43:33] lorddavidiii leaves the room
[16:44:23] JonNJ leaves the room
[16:45:00] lorddavidiii joins the room
[16:45:35] lorddavidiii leaves the room
[16:47:11] lorddavidiii joins the room
[16:47:46] lorddavidiii leaves the room
[16:49:18] lorddavidiii joins the room
[16:50:02] lorddavidiii leaves the room
[16:50:32] JonNJ joins the room
[16:51:02] egnun joins the room
[16:51:25] lorddavidiii joins the room
[16:51:53] kryptos joins the room
[16:52:06] lorddavidiii leaves the room
[16:52:27] mike leaves the room
[16:52:52] danielepiccoli joins the room
[16:53:35] lorddavidiii joins the room
[16:54:18] lorddavidiii leaves the room
[16:55:34] lorddavidiii joins the room
[16:56:12] lorddavidiii leaves the room
[16:56:39] marc joins the room
[16:57:04] mike joins the room
[16:57:26] madmalkav leaves the room
[16:57:32] lorddavidiii joins the room
[16:57:45] Wojtek leaves the room
[16:58:04] Marc joins the room
[16:58:10] lorddavidiii leaves the room
[16:58:24] JonNJ leaves the room
[16:58:28] nekit leaves the room
[16:59:17] madmalkav joins the room
[16:59:28] lorddavidiii joins the room
[17:00:07] lorddavidiii leaves the room
[17:01:22] kryptos leaves the room
[17:01:24] lorddavidiii joins the room
[17:02:05] lorddavidiii leaves the room
[17:03:37] lorddavidiii joins the room
[17:04:10] lorddavidiii leaves the room
[17:05:31] lorddavidiii joins the room
[17:05:45] danielepiccoli leaves the room
[17:06:00] isotelo leaves the room
[17:06:05] isotelo joins the room
[17:06:08] stp leaves the room
[17:06:10] lorddavidiii leaves the room
[17:07:34] lorddavidiii joins the room
[17:08:07] lorddavidiii leaves the room
[17:08:56] Tim joins the room
[17:09:32] stp joins the room
[17:09:44] lorddavidiii joins the room
[17:11:12] danielepiccoli joins the room
[17:12:07] poni leaves the room
[17:15:36] stp leaves the room
[17:22:14] mimi89999 leaves the room: Disconnected: Replaced by new connection
[17:22:20] mimi89999 joins the room
[17:23:51] JonNJ joins the room
[17:25:13] [Alien] leaves the room
[17:25:50] [Alien] joins the room
[17:27:43] ChaosKid42 joins the room
[17:28:47] twl joins the room
[17:28:50] ChaosKid42 joins the room
[17:28:52] JonNJ leaves the room
[17:29:04] JonNJ joins the room
[17:29:55] debacle leaves the room
[17:29:56] debalance joins the room
[17:31:35] writer77 leaves the room
[17:31:44] vanitasvitae leaves the room
[17:32:16] <debalance> badlop: right, rebar3 has not been packaged yet
[17:37:46] ff leaves the room
[17:39:40] JonNJ leaves the room
[17:39:44] prefiks leaves the room
[17:41:53] robert_mobil leaves the room
[17:41:57] robert_mobil joins the room
[17:42:36] [Alien] leaves the room
[17:42:39] [Alien] joins the room
[17:44:03] JonNJ joins the room
[17:44:21] ff joins the room
[17:45:32] writer77 joins the room
[17:46:30] <badlop> as rebar2 is deprecated, sooner or later we'll have to move on to rebar3
[17:47:14] robert_mobil leaves the room
[17:47:38] <badlop> i'm considering including rebar3 in the next ejabberd release, and suggest trying it with
  ./configure --with-rebar=./rebar3
  make
[17:48:11] <badlop> later, in a future release, use rebar3 by default (maintaining rebar2 just in case)
[17:48:24] <badlop> and someday, when nobody needs rebar2 at all, remove it
[17:48:25] Tshering Lhamo leaves the room
[17:48:31] Tshering Lhamo joins the room
[17:48:48] kryptos joins the room
[17:49:05] JonNJ leaves the room
[17:49:29] stp joins the room
[17:50:56] <badlop> so... when you build the next ejabberd&libraries packages, would it be too costly to try using rebar3?
[17:51:30] sander leaves the room
[17:52:31] sachaobado leaves the room
[17:54:07] vanitasvitae joins the room
[17:54:07] sachaobado joins the room
[17:54:39] kinky_nekoboi leaves the room
[17:55:16] robert_mobil leaves the room
[17:55:50] stp leaves the room
[17:56:56] jannic leaves the room
[17:57:59] sander joins the room
[18:00:29] jannic joins the room
[18:03:18] x0n leaves the room
[18:05:12] stp joins the room
[18:11:25] mss_cyclist leaves the room
[18:13:00] mss_cyclist joins the room
[18:19:31] hlad leaves the room
[18:19:32] hlad joins the room
[18:19:33] hlad leaves the room
[18:20:37] dennis joins the room
[18:21:07] schmillin leaves the room
[18:21:42] schmillin joins the room
[18:22:18] inkey leaves the room
[18:24:18] smalltimebloke joins the room
[18:26:07] sander leaves the room
[18:26:43] sander joins the room
[18:27:20] sander leaves the room
[18:28:53] inkey joins the room
[18:29:14] hlad joins the room
[18:29:14] hlad leaves the room
[18:30:06] sander leaves the room
[18:30:28] sander joins the room
[18:31:08] marc leaves the room
[18:31:22] spla joins the room
[18:31:28] hlad joins the room
[18:31:29] hlad leaves the room
[18:32:06] spla leaves the room
[18:32:07] spla joins the room
[18:32:08] sander joins the room
[18:40:01] rozzin leaves the room
[18:40:26] nik leaves the room
[18:42:09] JonNJ leaves the room: Machine is going to sleep
[18:42:53] nik joins the room
[18:44:18] JonNJ joins the room
[18:47:29] 86ul leaves the room
[18:47:35] sachaobado leaves the room
[18:47:51] Neustradamus_ joins the room
[18:49:21] Neustradamus_ leaves the room
[18:49:21] Neustradamus_ joins the room
[18:50:02] Neustradamus_ leaves the room
[18:50:03] Neustradamus_ joins the room
[18:52:45] sachaobado joins the room
[18:54:06] basheer leaves the room
[18:54:06] Tshering Lhamo leaves the room
[18:54:32] Tshering Lhamo joins the room
[18:54:53] mimi89999 leaves the room
[18:54:59] mimi89999 joins the room
[19:02:11] erik leaves the room
[19:02:11] nico leaves the room
[19:02:35] pino joins the room
[19:02:53] pino leaves the room
[19:03:10] pino joins the room
[19:03:18] JonNJ leaves the room
[19:05:30] pino leaves the room
[19:05:46] pino joins the room
[19:06:07] isotelo leaves the room
[19:06:39] lorddavidiii leaves the room
[19:07:13] isotelo joins the room
[19:08:02] marc joins the room
[19:08:05] nico joins the room
[19:11:55] hlad joins the room
[19:14:42] hlad leaves the room
[19:14:48] hlad joins the room
[19:14:48] hlad leaves the room
[19:16:51] erik joins the room
[19:17:44] hlad joins the room
[19:17:44] hlad leaves the room
[19:20:31] x51 leaves the room
[19:21:37] Camel leaves the room: Replaced by new connection
[19:21:38] Camel joins the room
[19:21:46] ave leaves the room
[19:22:30] ave joins the room
[19:26:54] Barny leaves the room
[19:30:33] andrey.g joins the room
[19:31:27] robert_mobil leaves the room
[19:31:33] robert_mobil joins the room
[19:35:08] ave leaves the room
[19:36:16] ave joins the room
[19:36:39] robert_mobil joins the room
[19:38:08] hlad joins the room
[19:38:09] hlad leaves the room
[19:40:36] kinetik joins the room
[19:42:37] rauhfasertapete leaves the room
[19:42:38] rauhfasertapete joins the room
[19:44:21] norkki leaves the room
[19:44:26] Neustradamus_ leaves the room
[19:44:26] Neustradamus_ joins the room
[19:46:40] neox joins the room
[19:46:46] ij joins the room
[19:48:58] norkki joins the room
[19:49:31] ij leaves the room
[19:49:32] Neustradamus_ leaves the room
[19:52:06] kinky_nekoboi joins the room
[19:53:22] schäfchen726 leaves the room
[19:53:54] hlad joins the room
[19:53:54] hlad leaves the room
[19:55:34] schäfchen726 leaves the room
[19:55:50] kinetik leaves the room
[19:57:31] kryptos leaves the room
[19:57:31] ff leaves the room
[19:57:44] kryptos joins the room
[19:58:25] marc leaves the room
[19:59:44] mss_cyclist leaves the room
[20:00:53] marc joins the room
[20:01:14] ff joins the room
[20:02:25] kinetik joins the room
[20:04:40] hlad joins the room
[20:04:49] mss_cyclist joins the room
[20:05:16] hlad leaves the room
[20:05:23] hlad joins the room
[20:05:23] hlad leaves the room
[20:06:43] isotelo leaves the room
[20:07:39] badlop leaves the room
[20:09:12] St0ne leaves the room
[20:10:17] St0ne joins the room
[20:12:14] mimi89999 leaves the room
[20:12:29] mimi89999 joins the room
[20:12:31] ij joins the room
[20:14:26] yann-kaelig leaves the room
[20:14:56] ij leaves the room
[20:15:28] hlad joins the room
[20:15:28] hlad leaves the room
[20:17:53] JonNJ joins the room
[20:17:55] mimi89999 leaves the room
[20:18:00] neox leaves the room
[20:18:04] mimi89999 joins the room
[20:18:30] hlad joins the room
[20:18:31] hlad leaves the room
[20:18:32] Lemm1ng leaves the room
[20:18:36] DebXWoody leaves the room
[20:23:56] JonNJ leaves the room
[20:25:07] JonNJ joins the room
[20:26:22] DebXWoody joins the room
[20:26:39] pod leaves the room
[20:27:03] kryptos leaves the room
[20:27:10] hlad joins the room
[20:27:10] hlad leaves the room
[20:27:43] hlad joins the room
[20:27:44] hlad leaves the room
[20:28:01] doctor joins the room
[20:29:57] norkki leaves the room
[20:30:49] neox joins the room
[20:31:10] norkki joins the room
[20:31:46] DebXWoody leaves the room
[20:34:53] daniel leaves the room
[20:34:55] ff leaves the room
[20:35:11] Marc leaves the room
[20:37:52] ij joins the room
[20:38:54] sumpfralle_ leaves the room
[20:38:54] sumpfralle leaves the room
[20:39:03] rozzin joins the room
[20:39:18] JonNJ joins the room
[20:39:23] ij leaves the room
[20:44:37] JonNJ leaves the room
[20:51:52] sachaobado leaves the room
[20:51:54] sachaobado joins the room
[20:52:36] mss_cyclist leaves the room
[20:52:53] ben leaves the room
[20:53:17] mss_cyclist joins the room
[20:55:49] JonNJ joins the room
[20:56:59] nuron joins the room
[20:57:17] rom1dep joins the room
[20:59:07] andrey.g leaves the room
[20:59:25] daniel joins the room
[20:59:35] Stephen leaves the room
[21:04:14] ru_maniac leaves the room
[21:04:32] JonNJ leaves the room
[21:04:51] mss_cyclist leaves the room
[21:05:01] pod leaves the room
[21:05:06] mss_cyclist joins the room
[21:08:48] pod joins the room
[21:11:32] basheer joins the room
[21:17:30] basheer leaves the room
[21:17:31] basheer joins the room
[21:19:18] JonNJ joins the room
[21:24:35] pod leaves the room
[21:24:41] JonNJ leaves the room
[21:26:14] JonNJ joins the room
[21:33:17] JonNJ leaves the room
[21:33:51] marc leaves the room
[21:37:56] [Alien] leaves the room
[21:38:32] [Alien] joins the room
[21:40:16] thrrgilag joins the room
[21:41:56] danielepiccoli leaves the room
[21:43:37] robert_mobil leaves the room
[21:45:40] marc0s leaves the room: Disconnected: Replaced by new connection
[21:45:40] marc0s joins the room
[21:48:55] mimi89999 leaves the room
[21:48:59] mimi89999 joins the room
[21:50:34] robert_mobil leaves the room
[21:54:05] JonNJ joins the room
[21:57:24] malloc leaves the room
[21:58:51] debacle joins the room
[21:59:06] JonNJ leaves the room
[21:59:54] VC leaves the room
[21:59:55] VC joins the room
[22:02:55] karme leaves the room
[22:03:02] norkki leaves the room
[22:03:26] norkki joins the room
[22:04:32] Akasch leaves the room
[22:05:43] isotelo joins the room
[22:06:02] vanitasvitae leaves the room
[22:06:37] vanitasvitae joins the room
[22:07:25] madmalkav leaves the room
[22:07:47] nik leaves the room
[22:08:39] daniel leaves the room
[22:19:49] JonNJ joins the room
[22:21:59] mss_cyclist leaves the room
[22:22:07] mss_cyclist joins the room
[22:23:56] writer77 leaves the room
[22:23:58] writer77 joins the room
[22:25:16] neox leaves the room
[22:26:18] JonNJ leaves the room
[22:29:58] writer77 leaves the room
[22:30:17] neox joins the room
[22:30:19] Camel leaves the room: Replaced by new connection
[22:30:20] Camel joins the room
[22:32:30] daniel joins the room
[22:37:39] doctor leaves the room
[22:37:42] doctor joins the room
[22:40:58] carlos joins the room
[22:44:07] jerry joins the room
[22:44:32] ij joins the room
[22:45:05] ij leaves the room
[22:47:26] nik joins the room
[22:50:00] JonNJ joins the room
[22:51:01] ij leaves the room
[22:53:59] ij joins the room
[22:56:12] multixmp222 joins the room
[22:56:55] nuron leaves the room
[22:57:04] nuron joins the room
[22:57:23] ij leaves the room
[22:59:54] marc0s leaves the room
[22:59:55] marc0s joins the room
[23:05:44] writer77 joins the room
[23:10:04] neox leaves the room
[23:11:45] LNJ leaves the room
[23:11:56] Robbilie leaves the room
[23:13:18] kryptos joins the room
[23:13:39] norkki leaves the room
[23:13:44] [Alien] leaves the room
[23:14:02] JonNJ leaves the room
[23:14:03] JonNJ joins the room
[23:14:16] [Alien] joins the room
[23:14:40] norkki joins the room
[23:15:41] writer77 leaves the room
[23:16:04] writer77 joins the room
[23:16:32] Stephen joins the room
[23:19:04] JonNJ leaves the room: Disconnected: Replaced by new connection
[23:19:05] JonNJ joins the room
[23:19:10] Demo leaves the room
[23:22:39] neox leaves the room
[23:24:31] JonNJ leaves the room
[23:29:49] norkki leaves the room
[23:30:56] norkki joins the room
[23:33:06] presprout leaves the room
[23:41:45] kryptos leaves the room
[23:43:05] ij joins the room
[23:44:15] schäfchen726 leaves the room
[23:46:11] norkki leaves the room
[23:47:15] norkki joins the room
[23:49:07] ij leaves the room
[23:51:58] JonNJ joins the room
[23:53:38] debacle leaves the room
[23:58:20] schäfchen726 leaves the room
[23:58:34] carlos leaves the room
[23:59:10] JonNJ leaves the room
Powered by ejabberd - robust, scalable and extensible XMPP server Powered by Erlang Valid XHTML 1.0 Transitional Valid CSS!