TZ: UTC +1/+2
User Details
User Details
- User Since
- Sep 1 2016, 6:48 AM (411 w, 3 d)
- Availability
- Busy Busy until Jul 29.
- IRC Nick
- marostegui
- LDAP User
- Marostegui
- MediaWiki User
- MArostegui (WMF) [ Global Accounts ]
Thu, Jul 18
Thu, Jul 18
• Marostegui renamed T369855: db1179 crashed - hardware issues from db1179 stopped answering ping, depooled to db1179 crashed - hardware issues.
• Marostegui edited projects for T370265: Create new translate_cache table on Wikimedia wikis with the Translate extension installed, added: Data-Persistence; removed DBA.
• Marostegui added a comment to T366959: 1.43.0-wmf.14 deployment blockers.
• Marostegui added a comment to T366959: 1.43.0-wmf.14 deployment blockers.
• Marostegui added a comment to T370304: Exception caught inside exception handler: Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded..
Wed, Jul 17
Wed, Jul 17
• Marostegui updated subscribers of T366959: 1.43.0-wmf.14 deployment blockers.
A few minutes after the deployment quite a number of s1 replicas started to show lag - the master started to have contention too.
https://grafana.wikimedia.org/goto/qcuiQWuSg?orgId=1
• Marostegui added a comment to T370304: Exception caught inside exception handler: Wikimedia\Rdbms\DBUnexpectedError: Database servers in extension1 are overloaded..
I believe this and T366574 are the same?
• Marostegui added a comment to T370122: dbstore1008:3317 (s7) crashed.
Thank you
• Marostegui added a comment to T370122: dbstore1008:3317 (s7) crashed.
@BTullis most likely, as I forgot this host has special grants yeah. If you can apply them, that'd be good. If not, I can do it first thing tomorrow morning.
• Marostegui added a comment to T370265: Create new translate_cache table on Wikimedia wikis with the Translate extension installed.
Is this table really needed or is it a code problem? There're more tickets about the reported error the mentioned T370219: DBQueryError marking page for translation: Table 'mediawikiwiki.translate_cache' doesn't exist and an older one T272428: Error 1146: Table 'mediawikiwiki.translate_cache' doesn't exist
• Marostegui added a comment to T365805: Test MariaDB 10.11.
After a week of capturing some traffic on API and main traffic - apart from the usual recentchanges queries, I've captured some slow queries which have the same query plans on 10.6 hosts. I'm going to increase a bit more db2136's weight to see how it performs under a bit more pressure
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui added a comment to T367781: Drop deprecated abuse filter fields on wmf wikis.
@ABran-WMF I've switched s7 master
Done
• Marostegui closed T370121: Switchover s7 master (db1181 -> db1236), a subtask of T367781: Drop deprecated abuse filter fields on wmf wikis, as Resolved.
• Marostegui closed T370121: Switchover s7 master (db1181 -> db1236), a subtask of T367856: Cleanup revision table schema, as Resolved.
• Marostegui updated the task description for T370121: Switchover s7 master (db1181 -> db1236).
• Marostegui updated the task description for T370121: Switchover s7 master (db1181 -> db1236).
• Marostegui added a comment to T304653: Release and package auto_schema 0.1.0.
@Ladsgroup is this ticket still valid? It's been a couple of years
• Marostegui updated the task description for T370121: Switchover s7 master (db1181 -> db1236).
• Marostegui added a comment to T370121: Switchover s7 master (db1181 -> db1236).
Old 400
Tue, Jul 16
Tue, Jul 16
This is done - host recloned from db1174
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui added a comment to T370122: dbstore1008:3317 (s7) crashed.
I've upgraded dbstore1008 to 10.6.18
• Marostegui added a comment to T370122: dbstore1008:3317 (s7) crashed.
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui updated the task description for T370121: Switchover s7 master (db1181 -> db1236).
• Marostegui updated the task description for T370121: Switchover s7 master (db1181 -> db1236).
• Marostegui added a parent task for T370121: Switchover s7 master (db1181 -> db1236): T367856: Cleanup revision table schema.
• Marostegui updated the task description for T367856: Cleanup revision table schema.
This is done
• Marostegui closed T370019: Switchover s3 master (db1157 -> db1223), a subtask of T367856: Cleanup revision table schema, as Resolved.
• Marostegui updated the task description for T370019: Switchover s3 master (db1157 -> db1223).
• Marostegui updated the task description for T370019: Switchover s3 master (db1157 -> db1223).
• Marostegui updated the task description for T370019: Switchover s3 master (db1157 -> db1223).
• Marostegui added a comment to T370019: Switchover s3 master (db1157 -> db1223).
Old 500
• Marostegui reopened T369445: VRTS errors arriving to root@ as "Open".
We got this to root:
Mon, Jul 15
Mon, Jul 15
• Marostegui updated the task description for T367856: Cleanup revision table schema.
Dzahn awarded T369445: VRTS errors arriving to root@ a Like token.
• Marostegui added a comment to T369445: VRTS errors arriving to root@.
Thank you!
• Marostegui added a comment to T369445: VRTS errors arriving to root@.
Probably not total silence as you could miss things, but silencing those "known things".
• Marostegui reopened T369445: VRTS errors arriving to root@ as "Open".
• Marostegui added a comment to T362824: Q#:rack/setup/install dbproxy200[5-8].
@Papaul dbproxy2005 looks good now - no ipv6 and I can reach it just fine. If you want to move it back to 10G that's great, and if you'd want to reimage the other hosts, that'd be great
• Marostegui added a comment to T369855: db1179 crashed - hardware issues.
• Marostegui updated the task description for T370019: Switchover s3 master (db1157 -> db1223).
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui added a comment to T369882: Switchover s7 master (db2121 -> db2218).
Thank you!
• Marostegui updated the task description for T365424: Upgrade clouddb* hosts to Bookworm.
• Marostegui added a comment to T365453: Bring an-redacteddb1001 into service to replace clouddb1021.
@BTullis we probably need to add an-redacteddb1001 to hieradata/regex.yaml somewhere.
• Marostegui added a comment to T368518: decommission clouddb1021.
@BTullis check this before proceeding https://phabricator.wikimedia.org/T365424#9980124
• Marostegui added a comment to T365424: Upgrade clouddb* hosts to Bookworm.
• Marostegui added a comment to T362824: Q#:rack/setup/install dbproxy200[5-8].
Just talked to @Papaul - the reimage was expected to fail since the iface was moved back to the 10G one.
• Marostegui added a comment to T362824: Q#:rack/setup/install dbproxy200[5-8].
@Papaul the interface wasn't in netbox anymore, but the DNS entry for that host is still gone.
I've tried to reimage the host but it gets stuck on the PXE boot and keeps rebooting upon timing out waiting for the boot:
• Marostegui moved T370019: Switchover s3 master (db1157 -> db1223) from Triage to In progress on the DBA board.
• Marostegui added a parent task for T370019: Switchover s3 master (db1157 -> db1223): T367856: Cleanup revision table schema.
Sun, Jul 14
Sun, Jul 14
• Marostegui updated the task description for T367856: Cleanup revision table schema.
Fri, Jul 12
Fri, Jul 12
• Marostegui moved T369855: db1179 crashed - hardware issues from Ready to In progress on the DBA board.
• Marostegui added a comment to T362824: Q#:rack/setup/install dbproxy200[5-8].
@Papaul I cannot access the host via ssh remotely, but the host is up and has network. I've connected via supermicro idrac and I think it is related to the DNS
root@dbproxy2005:~# host dbproxy2005.codfw.wmnet Host dbproxy2005.codfw.wmnet not found: 3(NXDOMAIN)
• Marostegui added a parent task for T369882: Switchover s7 master (db2121 -> db2218): T367856: Cleanup revision table schema.
• Marostegui removed a subtask for T369882: Switchover s7 master (db2121 -> db2218): T367856: Cleanup revision table schema.
• Marostegui removed a parent task for T367856: Cleanup revision table schema: T369882: Switchover s7 master (db2121 -> db2218).
@Volans let's do this together on Monday? There's no need to this early in the morning, as it is not a primary master, so no RO involved.
• Marostegui added a parent task for T367856: Cleanup revision table schema: T369882: Switchover s7 master (db2121 -> db2218).
• Marostegui updated the task description for T367856: Cleanup revision table schema.
Thu, Jul 11
Thu, Jul 11
• ABran-WMF awarded T369720: Clean pt-heartbeat from read only external store nodes a Party Time token.
• Marostegui updated the task description for T367856: Cleanup revision table schema.
I have removed all these rows
Done
• Marostegui closed T369691: Switchover s3 master (db2127 -> db2205), a subtask of T367856: Cleanup revision table schema, as Resolved.
• Marostegui updated the task description for T369691: Switchover s3 master (db2127 -> db2205).
• Marostegui updated the task description for T369691: Switchover s3 master (db2127 -> db2205).
• Marostegui updated the task description for T367856: Cleanup revision table schema.
• Marostegui updated the task description for T369691: Switchover s3 master (db2127 -> db2205).
• Marostegui added a comment to T369691: Switchover s3 master (db2127 -> db2205).
Old 500
• Marostegui updated the task description for T369691: Switchover s3 master (db2127 -> db2205).
• Marostegui moved T369691: Switchover s3 master (db2127 -> db2205) from Ready to In progress on the DBA board.
Done
• Marostegui updated the task description for T369514: Switchover s1 master (db1163 -> db1184).
• Marostegui updated the task description for T369514: Switchover s1 master (db1163 -> db1184).
• Marostegui updated the task description for T369514: Switchover s1 master (db1163 -> db1184).
• Marostegui added a comment to T369514: Switchover s1 master (db1163 -> db1184).
Old 200, API 100
Wed, Jul 10
Wed, Jul 10
• Marostegui moved T367781: Drop deprecated abuse filter fields on wmf wikis from Ready to In progress on the DBA board.
• Marostegui moved T369720: Clean pt-heartbeat from read only external store nodes from Triage to In progress on the DBA board.
• Marostegui added a comment to T360930: Section-wide circuit breaking.
Pulling it in mwdebug and reducing the threshold to four connection led to this:
Next is to build a more user-friendly error page.
This is what needs fixing:
# for i in `db-mysql db1215 -A zarcillo -e "select instance from section_instances where section between 'es1' and 'es5'" -BN`; do echo $i; db-mysql $i heartbeat -e "select * from heartbeat"; done es1027 +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | 2021-01-28T12:36:52.000970 | 171966484 | es1027-bin.000001 | 3775 | NULL | NULL | es1 | eqiad | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ es1029 +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | 2021-01-28T12:36:52.000970 | 171966484 | es1027-bin.000001 | 3775 | NULL | NULL | es1 | eqiad | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ es1032 +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ | 2021-01-28T12:36:52.000970 | 171966484 | es1027-bin.000001 | 3775 | NULL | NULL | es1 | eqiad | +----------------------------+-----------+-------------------+----------+-----------------------+---------------------+-------+------------+ es2028 ERROR 1049 (42000): Unknown database 'heartbeat' es2030 ERROR 1049 (42000): Unknown database 'heartbeat' es2032 ERROR 1049 (42000): Unknown database 'heartbeat' es1026 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2015-06-15T16:29:30.501080 | 1016153 | es1005-bin.001945 | 186397859 | es1006-bin.001982 | 1014808588 | NULL | NULL | | 2015-09-03T15:26:04.500960 | 1016154 | es1006-bin.002151 | 262959914 | es1011-bin.000017 | 286417364 | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2018-10-22T08:11:04.001020 | 180367401 | es2016-bin.002378 | 8940627 | es1015-bin.002761 | 823808274 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1030 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2014-09-14T10:32:34.500960 | 1016153 | es1005-bin.001399 | 139103315 | NULL | NULL | NULL | NULL | | 2015-06-06T17:00:45.001050 | 103217 | es1007-bin.001974 | 100665731 | es1006-bin.001966 | 856095036 | NULL | NULL | | 2015-09-03T15:26:04.500960 | 1016154 | es1006-bin.002151 | 262959914 | es1011-bin.000017 | 286417364 | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2018-10-22T08:11:04.001020 | 180367401 | es2016-bin.002378 | 8940627 | es1015-bin.002761 | 823808274 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1033 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2015-09-03T17:02:53.501140 | 1016154 | es1006-bin.002151 | 435646302 | es1011-bin.000017 | 453256724 | NULL | NULL | | 2014-09-14T10:32:34.500960 | 1016153 | es1005-bin.001399 | 139103315 | NULL | NULL | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2018-10-22T08:11:04.001020 | 180367401 | es2016-bin.002378 | 8940627 | es1015-bin.002761 | 823808274 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2026 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2014-09-14T10:32:34.500960 | 1016153 | es1005-bin.001399 | 139103315 | NULL | NULL | NULL | NULL | | 2014-10-17T10:30:59.501030 | 103217 | es1007-bin.001503 | 610785209 | es1006-bin.001511 | 772584249 | NULL | NULL | | 2015-09-03T15:26:04.500960 | 1016154 | es1006-bin.002151 | 262959914 | es1011-bin.000017 | 286417364 | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2020-03-04T09:50:28.001040 | 180367401 | es2016-bin.004034 | 406762616 | es1015-bin.004402 | 750373395 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2031 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2014-09-14T10:32:34.500960 | 1016153 | es1005-bin.001399 | 139103315 | NULL | NULL | NULL | NULL | | 2014-10-17T10:30:59.501030 | 103217 | es1007-bin.001503 | 610785209 | es1006-bin.001511 | 772584249 | NULL | NULL | | 2015-09-03T15:26:04.500960 | 1016154 | es1006-bin.002151 | 262959914 | es1011-bin.000017 | 286417364 | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2020-03-04T09:50:28.001040 | 180367401 | es2016-bin.004034 | 406762616 | es1015-bin.004402 | 750373395 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2033 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:05.062560 | 100234 | es5-bin.000283 | 980895084 | NULL | NULL | NULL | NULL | | 2014-09-14T10:32:34.500960 | 1016153 | es1005-bin.001399 | 139103315 | NULL | NULL | NULL | NULL | | 2014-10-17T10:30:59.501030 | 103217 | es1007-bin.001503 | 610785209 | es1006-bin.001511 | 772584249 | NULL | NULL | | 2015-09-03T15:26:04.500960 | 1016154 | es1006-bin.002151 | 262959914 | es1011-bin.000017 | 286417364 | NULL | NULL | | 2018-08-22T06:17:59.000790 | 171966470 | es1011-bin.002581 | 62175111 | NULL | NULL | es2 | eqiad | | 2020-03-04T09:45:04.000950 | 171974840 | es1015-bin.004402 | 750372971 | NULL | NULL | es2 | eqiad | | 2016-10-04T13:41:46.000780 | 180363394 | es2015-bin.000451 | 94383767 | es1015-bin.000849 | 285496084 | es2 | codfw | | 2020-03-04T09:50:28.001040 | 180367401 | es2016-bin.004034 | 406762616 | es1015-bin.004402 | 750373395 | es2 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1028 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:08.501010 | 100237 | es8-bin.000283 | 939979885 | NULL | NULL | NULL | NULL | | 2014-10-02T05:10:04.000980 | 103218 | es1008-bin.001429 | 462991923 | NULL | NULL | NULL | NULL | | 2015-09-03T16:35:06.500910 | 103219 | es1009-bin.002175 | 242760603 | es1014-bin.000016 | 176696514 | NULL | NULL | | 2015-06-24T10:29:28.500980 | 171974674 | es1008-bin.001945 | 218011198 | es1009-bin.002024 | 115233874 | NULL | NULL | | 2018-07-25T06:01:26.000580 | 171970747 | es1014-bin.002508 | 184374820 | NULL | NULL | es3 | eqiad | | 2016-12-01T14:50:16.001040 | 171978868 | es1019-bin.000996 | 604186859 | es1014-bin.001011 | 682669630 | es3 | eqiad | | 2017-11-24T11:27:07.000800 | 180359340 | es2018-bin.001481 | 3899609 | es1014-bin.001887 | 819577914 | es3 | codfw | | 2020-03-11T09:25:25.001050 | 171974721 | es1017-bin.004424 | 552430805 | NULL | NULL | es3 | eqiad | | 2018-10-22T08:13:11.000980 | 180355214 | es2017-bin.002374 | 271901865 | es1017-bin.002759 | 750868847 | es3 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1031 es1034 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:08.501010 | 100237 | es8-bin.000283 | 939979885 | NULL | NULL | NULL | NULL | | 2015-09-04T07:46:06.001140 | 103219 | es1009-bin.002176 | 430656852 | es1014-bin.000017 | 311069644 | NULL | NULL | | 2014-10-02T05:10:04.000980 | 103218 | es1008-bin.001429 | 462991923 | NULL | NULL | NULL | NULL | | 2015-06-24T10:29:28.500980 | 171974674 | es1008-bin.001945 | 218011198 | es1009-bin.002024 | 115233874 | NULL | NULL | | 2018-07-25T06:01:26.000580 | 171970747 | es1014-bin.002508 | 184374820 | NULL | NULL | es3 | eqiad | | 2016-12-01T14:50:16.001040 | 171978868 | es1019-bin.000996 | 604186859 | es1014-bin.001011 | 682669630 | es3 | eqiad | | 2017-11-24T11:27:07.000800 | 180359340 | es2018-bin.001481 | 3899609 | es1014-bin.001887 | 819577914 | es3 | codfw | | 2020-03-11T09:25:25.001050 | 171974721 | es1017-bin.004424 | 552430805 | NULL | NULL | es3 | eqiad | | 2018-10-22T08:13:11.000980 | 180355214 | es2017-bin.002374 | 271901865 | es1017-bin.002759 | 750868847 | es3 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2027 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:08.501010 | 100237 | es8-bin.000283 | 939979885 | NULL | NULL | NULL | NULL | | 2014-10-02T05:10:04.000980 | 103218 | es1008-bin.001429 | 462991923 | NULL | NULL | NULL | NULL | | 2014-10-02T06:43:53.501960 | 103220 | es1010-bin.001474 | 976384279 | es1009-bin.001486 | 223468842 | NULL | NULL | | 2015-09-03T16:35:06.500910 | 103219 | es1009-bin.002175 | 242760603 | es1014-bin.000016 | 176696514 | NULL | NULL | | 2015-06-24T10:29:28.500980 | 171974674 | es1008-bin.001945 | 218011198 | es1009-bin.002024 | 115233874 | NULL | NULL | | 2018-07-25T06:01:26.000580 | 171970747 | es1014-bin.002508 | 184374820 | NULL | NULL | es3 | eqiad | | 2016-12-01T14:50:16.001040 | 171978868 | es1019-bin.000996 | 604186859 | es1014-bin.001011 | 682669630 | es3 | eqiad | | 2018-07-24T16:46:07.001020 | 180359340 | es2018-bin.002107 | 515720489 | NULL | NULL | es3 | codfw | | 2020-03-11T09:25:25.001050 | 171974721 | es1017-bin.004424 | 552430805 | NULL | NULL | es3 | eqiad | | 2020-03-11T09:28:43.002270 | 180355214 | es2017-bin.004053 | 266817814 | es1017-bin.004424 | 552431229 | es3 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2029 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:08.501010 | 100237 | es8-bin.000283 | 939979885 | NULL | NULL | NULL | NULL | | 2014-10-02T05:10:04.000980 | 103218 | es1008-bin.001429 | 462991923 | NULL | NULL | NULL | NULL | | 2014-10-02T06:43:53.501960 | 103220 | es1010-bin.001474 | 976384279 | es1009-bin.001486 | 223468842 | NULL | NULL | | 2015-09-03T16:35:06.500910 | 103219 | es1009-bin.002175 | 242760603 | es1014-bin.000016 | 176696514 | NULL | NULL | | 2015-06-24T10:29:28.500980 | 171974674 | es1008-bin.001945 | 218011198 | es1009-bin.002024 | 115233874 | NULL | NULL | | 2018-07-25T06:01:26.000580 | 171970747 | es1014-bin.002508 | 184374820 | NULL | NULL | es3 | eqiad | | 2016-12-01T14:50:16.001040 | 171978868 | es1019-bin.000996 | 604186859 | es1014-bin.001011 | 682669630 | es3 | eqiad | | 2018-07-24T16:46:07.001020 | 180359340 | es2018-bin.002107 | 515720489 | NULL | NULL | es3 | codfw | | 2020-03-11T09:25:25.001050 | 171974721 | es1017-bin.004424 | 552430805 | NULL | NULL | es3 | eqiad | | 2020-03-11T09:28:43.002270 | 180355214 | es2017-bin.004053 | 266817814 | es1017-bin.004424 | 552431229 | es3 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2034 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2013-01-22T17:47:08.501010 | 100237 | es8-bin.000283 | 939979885 | NULL | NULL | NULL | NULL | | 2014-10-02T05:10:04.000980 | 103218 | es1008-bin.001429 | 462991923 | NULL | NULL | NULL | NULL | | 2015-09-03T16:35:06.500910 | 103219 | es1009-bin.002175 | 242760603 | es1014-bin.000016 | 176696514 | NULL | NULL | | 2014-10-02T06:43:53.501960 | 103220 | es1010-bin.001474 | 976384279 | es1009-bin.001486 | 223468842 | NULL | NULL | | 2018-07-25T06:01:26.000580 | 171970747 | es1014-bin.002508 | 184374820 | NULL | NULL | es3 | eqiad | | 2015-06-24T10:29:28.500980 | 171974674 | es1008-bin.001945 | 218011198 | es1009-bin.002024 | 115233874 | NULL | NULL | | 2020-03-11T09:25:25.001050 | 171974721 | es1017-bin.004424 | 552430805 | NULL | NULL | es3 | eqiad | | 2016-12-01T14:50:16.001040 | 171978868 | es1019-bin.000996 | 604186859 | es1014-bin.001011 | 682669630 | es3 | eqiad | | 2020-03-11T09:28:43.002270 | 180355214 | es2017-bin.004053 | 266817814 | es1017-bin.004424 | 552431229 | es3 | codfw | | 2018-07-24T16:46:07.001020 | 180359340 | es2018-bin.002107 | 515720489 | NULL | NULL | es3 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1020 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:43:45.001020 | 171970708 | es1021-bin.011827 | 108956172 | NULL | NULL | es4 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1021 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:44:18.001070 | 171970708 | es1021-bin.011827 | 108975444 | NULL | NULL | es4 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1022 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:43:46.001090 | 171970708 | es1021-bin.011827 | 108956756 | NULL | NULL | es4 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2020 +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:43:46.001090 | 171970708 | es1021-bin.011827 | 108956756 | NULL | NULL | es4 | eqiad | | 2024-05-16T05:43:46.001080 | 180359316 | es2021-bin.011871 | 1012496615 | es1021-bin.011827 | 108956756 | es4 | codfw | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ es2021 +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:43:46.001090 | 171970708 | es1021-bin.011827 | 108956756 | NULL | NULL | es4 | eqiad | | 2024-05-16T05:44:01.001040 | 180359316 | es2021-bin.011871 | 1012506148 | NULL | NULL | es4 | codfw | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ es2022 +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ | 2024-05-16T05:43:46.001090 | 171970708 | es1021-bin.011827 | 108956756 | NULL | NULL | es4 | eqiad | | 2024-05-16T05:43:47.001120 | 180359316 | es2021-bin.011871 | 1012497858 | es1021-bin.011827 | 108957340 | es4 | codfw | +----------------------------+-----------+-------------------+------------+-----------------------+---------------------+-------+------------+ es1023 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:17:37.000980 | 171966666 | es1024-bin.011766 | 481408637 | NULL | NULL | es5 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1024 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:19:18.000990 | 171966666 | es1024-bin.011766 | 481467621 | NULL | NULL | es5 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es1025 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:17:38.000960 | 171966666 | es1024-bin.011766 | 481409221 | NULL | NULL | es5 | eqiad | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2023 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:17:38.000960 | 171966666 | es1024-bin.011766 | 481409221 | NULL | NULL | es5 | eqiad | | 2024-05-16T06:17:39.001050 | 180355230 | es2024-bin.011814 | 337064446 | NULL | NULL | es5 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2024 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:17:38.000960 | 171966666 | es1024-bin.011766 | 481409221 | NULL | NULL | es5 | eqiad | | 2024-05-16T06:19:32.000800 | 180355230 | es2024-bin.011814 | 337130464 | NULL | NULL | es5 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ es2025 +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | ts | server_id | file | position | relay_master_log_file | exec_master_log_pos | shard | datacenter | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+ | 2024-05-16T06:17:38.000960 | 171966666 | es1024-bin.011766 | 481409221 | NULL | NULL | es5 | eqiad | | 2024-05-16T06:17:39.001050 | 180355230 | es2024-bin.011814 | 337064446 | NULL | NULL | es5 | codfw | +----------------------------+-----------+-------------------+-----------+-----------------------+---------------------+-------+------------+